Deconvolution super-resolution imaging method and device based on deep learning denoising and medium

Abstract

The invention relates to the technical field of biological microscopic imaging, and discloses a deconvolution super-resolution imaging method and device based on deep learning denoising and a medium. The method comprises the following steps: acquiring a to-be-denoised image of a to-be-imaged structure marked by fluorescent molecules; de-noising the image to be de-noised by using a preset image de-noising neural network to obtain a de-noised image; a training data set for training the network is generated by utilizing the optical switch characteristics of optical switch fluorescent molecules, and the optical switch fluorescent molecules are used for marking to-be-imaged structures of samples in the training data set; the emission spectrum of the fluorescent molecules is overlapped with the emission spectrum of the fluorescent molecules of the optical switch; and carrying out deconvolution processing on the denoised image to obtain a super-resolution imaging image. According to the invention, the large-scale and high-quality true value image is obtained by using the optical switch characteristic of the optical switch fluorescent molecules, and the denoising effect of the image denoising neural network is improved; and denoising the image by using the network, and then carrying out deconvolution on the denoised image to realize large-view living cell single-frame super-resolution imaging.

Description

Technical Field

[0001] The present disclosure relates to the field of biological microscopy imaging technology. Specifically, it relates to a deconvolution super-resolution imaging method, apparatus, electronic device, and computer-readable storage medium based on deep learning denoising. Background Art

[0002] Super-resolution imaging technology refers to the process of processing low-resolution images into high-resolution images through computer algorithms or models. It allows us to see more details and obtain a clearer and more realistic visual experience. In recent years, various super-resolution imaging technologies, such as Stimulated emission depletion microscopy (STED), Stochastic optical reconstruction microscopy (STORM), and photoactivated localization microscopy (PALM), have broken through the limitation of the optical diffraction limit, enabling researchers to study the fine features of subcellular structures in cells at the nanoscale resolution level.

[0003] However, these imaging methods are generally multi-frame super-resolution imaging technologies, that is, they utilize the different spatial information presented by the sample in the time series, sacrificing the time resolution in exchange for the spatial resolution, and it is difficult to achieve single-frame super-resolution imaging. For example, the Structured illumination microscopy (SIM), which is most suitable for live cell super-resolution imaging, also requires at least 9 frames of original data to reconstruct 1 ultra-high resolution image. In addition to the defect of sacrificing time resolution in exchange for spatial resolution, the aforementioned various super-resolution imaging technologies are also prone to reconstruction artifacts caused by the rapid movement of the molecules to be observed; moreover, multi-frame sampling is likely to cause fluorescence molecule bleaching, posing a huge challenge to 3D super-resolution imaging of live cells. Therefore, it is of great significance to develop a large-field single-frame super-resolution imaging technology that does not sacrifice time resolution. Summary of the Invention

[0004] In view of the above situation, the embodiments of the present disclosure provide a deconvolution super-resolution imaging method, apparatus, electronic device, and computer-readable storage medium based on deep learning denoising, aiming to solve the above problems or at least partially solve the above problems.

[0005] In a first aspect, the embodiments of the present disclosure provide a deconvolution super-resolution imaging method based on deep learning denoising, the method comprising:

[0006] Obtain a to-be-denoised image, where the to-be-imaged structure of the to-be-denoised image is labeled with a fluorescent molecule;

[0007] Use a preset image denoising neural network to denoise the to-be-denoised image to obtain a denoised image; wherein, the training dataset for training the image denoising neural network is generated using the optical switching characteristics of an optical switching fluorescent molecule, and the optical switching fluorescent molecule is used to label the to-be-imaged structures of the samples in the training dataset; the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the optical switching fluorescent molecule;

[0008] Perform deconvolution processing on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

[0009] In a second aspect, an embodiment of the present disclosure further provides a deconvolution super-resolution imaging device based on deep learning denoising, and the device includes:

[0010] An acquisition module, configured to acquire a to-be-denoised image, where the to-be-imaged structure of the to-be-denoised image is labeled with a fluorescent molecule;

[0011] A denoising module, configured to use a preset image denoising neural network to denoise the to-be-denoised image to obtain a denoised image; wherein, the training dataset for training the image denoising neural network is generated using the optical switching characteristics of an optical switching fluorescent molecule, and the optical switching fluorescent molecule is used to label the to-be-imaged structures of the samples in the training dataset; the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the optical switching fluorescent molecule;

[0012] A super-resolution imaging module, configured to perform deconvolution processing on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, and the executable instructions, when executed, cause the processor to execute the steps of the above-mentioned deconvolution super-resolution imaging method based on deep learning denoising.

[0014] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, and the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to execute the steps of the above-mentioned deconvolution super-resolution imaging method based on deep learning denoising.

[0015] With the above technical solutions, the deconvolution super-resolution imaging method, device, and medium based on deep learning denoising provided by the embodiments of the present disclosure obtain a super-resolution imaging image corresponding to the denoised image by performing deconvolution processing on the denoised image. The denoised image is an image processed by a preset image denoising neural network. The training dataset used by the preset image denoising neural network is generated using the optical switching characteristics of opto-switchable fluorescent molecules. Using this feature, a large-scale, high-quality ground truth image with extremely low noise that is more in line with real images can be obtained, especially with great advantages for deep tissue imaging. Training the network using the training dataset containing the ground truth image can greatly improve the robustness and stability of the preset image denoising neural network. At the same time, since opto-switchable fluorescent molecules can bind to various subcellular structures and do not depend on a specific structure, it is not necessary for the imaging structure of the image to be denoised to be the same or similar to the imaging structure in the training dataset. As long as the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the opto-switchable fluorescent molecule, the preset image denoising neural network can be used to denoise the image to be denoised. Therefore, the preset image denoising neural network has strong generality and universality, which greatly improves the denoising effect of the network, making the noise of the denoised image extremely low. Then, performing deconvolution processing on the denoised image can reverse and restore the convolution phenomenon unique to optical microscopes, thereby improving the spatio-temporal resolution of the image and obtaining a completely restored original image, that is, the super-resolution imaging image corresponding to the denoised image, realizing large-field single-frame super-resolution imaging and solving the defects existing in multi-frame super-resolution imaging methods in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The schematic embodiments and descriptions thereof of the present disclosure are used to explain the present disclosure and do not constitute an improper limitation to the present disclosure. In the drawings:

[0017] Figure 1 The flowchart of the deconvolution super-resolution imaging method based on deep learning denoising provided by the embodiments of the present disclosure is shown;

[0018] Figure 2 The flowchart of the deconvolution super-resolution imaging method based on deep learning denoising provided by another embodiment of the present disclosure is shown;

[0019] Figure 3 The laser modulation timing diagram provided by the embodiments of the present disclosure is shown;

[0020] Figure 4 The comparison diagram of the image processing results of directly deconvolving without denoising and deconvolving after denoising provided by the embodiments of the present disclosure is shown;

[0021] Figure 5The figure shows a schematic structural diagram of a deconvolution super-resolution imaging device based on deep learning denoising provided by an embodiment of the present disclosure;

[0022] Figure 6 The figure shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0023] To make the objectives, technical solutions, and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with specific embodiments of the present disclosure and the corresponding drawings. Apparently, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present disclosure.

[0024] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0025] It should be noted that the terms "first", "second", etc. in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such use can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the term "including" and its variants should be interpreted as open terms meaning "including but not limited to".

[0026] As described above, existing imaging methods generally use multi-frame super-resolution imaging techniques, that is, by using different spatial information presented by a sample in a time series, sacrificing time resolution in exchange for spatial resolution, and it is difficult to achieve single-frame super-resolution imaging. For example, the structured illumination microscopy (SIM), which is most suitable for live cell super-resolution imaging, also requires at least 9 frames of raw data to reconstruct 1 ultra-high-resolution image. In addition to the defect of sacrificing time resolution in exchange for spatial resolution, the foregoing various super-resolution imaging techniques are also prone to reconstruction artifacts caused by the rapid movement of molecules to be observed; moreover, multi-frame sampling is likely to cause fluorescence molecule bleaching, posing a huge challenge to 3D super-resolution imaging of live cells. Therefore, it is of great significance to develop a large-field single-frame super-resolution imaging technique that does not sacrifice time resolution.

[0027] Based on this, in the embodiments of the present disclosure, deconvolution processing is performed on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image. The denoised image is an image processed by a preset image denoising neural network. The training data set used by the preset image denoising neural network is generated using the optical switching characteristics of opto-switchable fluorescent molecules. Using this feature, a large-scale, more realistic, and extremely low-noise high-quality ground truth image can be obtained, especially with great advantages for deep tissue imaging. Training the network using the training data set containing the ground truth image can greatly improve the robustness and stability of the preset image denoising neural network. At the same time, since opto-switchable fluorescent molecules can bind to various subcellular structures without relying on a specific structure, it is not necessary for the imaging structure of the image to be denoised to be the same or similar to the imaging structure in the training data set. As long as the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the opto-switchable fluorescent molecule, the preset image denoising neural network can be applied to denoise the image to be denoised. Therefore, the preset image denoising neural network has strong versatility and universality, which greatly improves the denoising effect of the network, making the noise of the denoised image extremely low. Then, deconvolution processing is performed on the denoised image to reverse and restore the convolution phenomenon unique to optical microscopes, thereby improving the spatio-temporal resolution of the image and obtaining the fully restored original image, that is, the super-resolution imaging image corresponding to the denoised image, realizing large-field single-frame super-resolution imaging and solving the defects existing in the multi-frame super-resolution imaging method in the prior art.

[0028] To facilitate the understanding of this embodiment, first, a deconvolution super-resolution imaging method based on deep learning denoising disclosed in the embodiments of the present disclosure will be introduced in detail. The execution subject of the deconvolution super-resolution imaging method based on deep learning denoising provided in the embodiments of the present disclosure is generally a computer device with certain computing capabilities. Such a computer device includes, for example: a terminal device, a server, or other processing devices. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a computing device, a wearable device, etc. In some possible implementation manners, the deconvolution super-resolution imaging method based on deep learning denoising can be implemented by a processor invoking computer-readable instructions stored in a memory.

[0029] Figure 1 FIG. shows a schematic flow chart of the deconvolution super-resolution imaging method based on deep learning denoising provided in the embodiments of the present disclosure. From Figure 1 it can be seen that the embodiments of the present disclosure at least include step S101-step S103:

[0030] Step S101: Obtain an image to be denoised, and the imaging structure of the image to be denoised is labeled with a fluorescent molecule.

[0031] In this step, the structure to be imaged in the image to be denoised is the cell structure observed by a fluorescence microscope, including but not limited to: endoplasmic reticulum, mitochondria, centrosomes, Golgi apparatus, the morphology of the nucleolus of the cell nucleus, ribosomes, lysosomes, etc. Fluorescent molecules include ordinary fluorescent molecules and photoswitchable fluorescent molecules with photoswitching properties.

[0032] Step S102: Use a preset image denoising neural network to denoise the image to be denoised, obtaining a denoised image; wherein, the training dataset for training the image denoising neural network is generated using the photoswitching characteristics of photoswitchable fluorescent molecules, and the photoswitchable fluorescent molecules are used to label the structures to be imaged in each sample in the training dataset; the emission spectrum of the fluorescent molecules overlaps with the emission spectrum of the photoswitchable fluorescent molecules.

[0033] It has been found through research that deep learning-based methods can learn complex image features and noise distribution rules by establishing deep learning models and utilizing large-scale training data and the ability of parallel computing, thereby achieving efficient noise removal. Deep learning learns directly from the input image to the output image through an end-to-end training method, reducing the dependence on manual feature design and improving the robustness and generalization ability of the algorithm. Deep learning denoising methods also have other advantages. For example, deep learning models can automatically adjust network parameters and learning strategies according to the different characteristics of the input image and noise types to achieve adaptive denoising of different noises; through multi-layer non-linear transformations, model the complex relationship between the image and the noise, thereby better capturing the details and texture information in the image; utilize the context information in the image, that is, the spatial and intensity correlations around the pixels, to more accurately restore the details and structures in the image. Finally, deep learning models can also improve the expressive ability and denoising performance of the model by increasing the depth and width of the network, introducing more hidden layers and parameters. In summary, deep learning denoising methods have strong learning ability and adaptability, can better restore the details and structures in the image, and improve the quality and clarity of the image.

[0034] However, in practical application scenarios, for example, in the field of biological microscopy imaging technology, denoising methods based on deep learning need to use a large number of ground-truth images for training. However, it is difficult to obtain large-scale and high-quality ground-truth images. Moreover, the network models of deep learning methods are usually trained based on specific datasets. When applying the trained network model, the content of the image to be processed needs to be the same or similar to the images in the training dataset. Therefore, the trained neural network model has poor robustness, weak generality and universality. In addition, many deep learning methods introduce translational invariance during the denoising process, that is, they are insensitive to the translation of the input image. This may lead to problems such as blurring or distortion in the denoising results. Deep learning methods are very sensitive to perturbations of the input data. Even a tiny perturbation may cause a huge change in the output result. In the denoising task, the input image may contain some subtle noises, which may cause deep learning methods to be unable to accurately remove the noises. In short, deep learning denoising methods need to develop more effective data augmentation techniques to improve the quality and quantity of ground-truth images, introduce more prior knowledge to enhance the robustness and generality of the model, etc., to achieve better denoising effects. Based on this, in this step, after obtaining the image to be denoised, a preset image denoising neural network can be used to denoise the image to be denoised to obtain a denoised image. Here, the preset image denoising neural network can be a deep learning network model, and the training dataset for training the image denoising neural network is generated using the optical switching characteristics of opto-switchable fluorescent molecules. Among them, opto-switchable fluorescent molecules refer to molecules that can undergo reversible changes in structure and properties under light stimulation. Specifically, opto-switchable fluorescent molecules can achieve the conversion between the bright state and the dark state under different illumination conditions. That is, under the irradiation of a specific wavelength, the opto-switchable fluorescent molecule is in the "on" state, that is, the bright state or the luminescent state. Under the irradiation of another specific wavelength, the opto-switchable fluorescent molecule is in the "off" state, that is, the dark state or the non-luminescent state.

[0035] Step S103: Perform deconvolution processing on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

[0036] After obtaining the denoised image, deconvolution processing can be performed on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image. During implementation, for example, the denoised image can be deconvolved by directly deconvolving to obtain the corresponding super-resolution imaging image, or the denoised image can be deconvolved by iterative deconvolution to obtain the denoised image. The embodiments of the present disclosure do not limit the manner of deconvolution processing.

[0037] From Figure 1As can be seen from the method shown, the embodiments of the present disclosure innovatively propose a deconvolution super-resolution imaging method based on deep learning denoising. By performing deconvolution processing on the denoised image, a super-resolution imaging image corresponding to the denoised image is obtained. The denoised image is an image processed by a preset image denoising neural network. The training data set used by the preset image denoising neural network is generated by using the optical switching characteristics of opto-switchable fluorescent molecules. Using this feature, a large-scale, more realistic, and extremely low-noise high-quality ground truth image can be obtained, especially with great advantages for deep tissue imaging. Training the network with the training data set containing the ground truth image can greatly improve the robustness and stability of the preset image denoising neural network. At the same time, since opto-switchable fluorescent molecules can bind to various subcellular structures without relying on a specific structure, it is not necessary for the structure to be imaged in the image to be denoised to be the same or similar to the structure to be imaged in the training data set. As long as the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the opto-switchable fluorescent molecule, the preset image denoising neural network can be used to denoise the image to be denoised. Therefore, the preset image denoising neural network has strong generality and universality, which greatly improves the denoising effect of the network, making the noise of the denoised image extremely low. Then, by performing deconvolution processing on the denoised image, the convolution phenomenon unique to the optical microscope can be reversed and restored, thereby improving the spatio-temporal resolution of the image and obtaining the fully restored original image, that is, the super-resolution imaging image corresponding to the denoised image, realizing large-field single-frame super-resolution imaging and solving the defects existing in the multi-frame super-resolution imaging method in the prior art.

[0038] Furthermore, in order to better illustrate the process of the above deconvolution super-resolution imaging method based on deep learning denoising, as a refinement and extension of the above embodiments, the embodiments of the present invention provide several alternative embodiments, but not limited thereto, as specifically shown below:

[0039] In a possible implementation manner, the image to be denoised is determined by the following method A1 / method A2 / method A3:

[0040] Method A1: Obtain the original image and use the original image as the image to be denoised, where the original image is obtained by camera sampling.

[0041] Method A2: Perform preprocessing on the original image and use the obtained processed image as the image to be denoised, where the preprocessing methods include: optical lock-in amplification algorithm processing, Gaussian filtering processing, mean filtering processing, Wiener filtering processing, or deconvolution processing, etc.

[0042] Method A3: Use the obtained target super-resolution imaging image as the image to be denoised. Among them, the target super-resolution imaging image includes: Airyscan super-resolution imaging image, ISM super-resolution imaging image, A-POD super-resolution imaging image, MSSR super-resolution imaging image, etc.

[0043] In this embodiment, for Method A2, the original image sampled by the camera can be preprocessed, and the obtained processed image is used as the image to be denoised. Exemplarily, the original image 1 can be processed by an optical phase-locked amplification algorithm to obtain the processed image 1, and the processed image 1 is used as the image to be denoised. The original image 2 can also be processed by Gaussian filtering to obtain the processed image 2, and the processed image 2 is used as the image to be denoised. The original image 3 can also be processed by mean filtering to obtain the processed image 3, and the processed image 3 is used as the image to be denoised. The original image 4 can also be processed by Wiener filtering to obtain the processed image 4, and the processed image 4 is used as the image to be denoised. The original image 5 can also be processed by deconvolution to obtain the processed image 5, and the processed image 5 is used as the image to be denoised. It should be noted that the examples here are only illustrative and do not limit the embodiments of the present application.

[0044] For Method A3, the image 1 obtained by using the Airyscan super-resolution imaging technology can be used as the image to be denoised. The image 2 obtained by using the ISM super-resolution imaging technology can also be used as the image to be denoised. The image 3 obtained by using the A-POD super-resolution imaging technology can also be used as the image to be denoised. The image 4 obtained by using the MSSR super-resolution imaging technology can also be used as the image to be denoised. It should be noted that the examples here are only illustrative and do not limit the embodiments of the present application.

[0045] In a possible implementation manner, the fluorescent molecules label the structure to be imaged in the image to be denoised through the following Method B1 / Method B2:

[0046] Method B1: The fluorescent molecules label the structure to be imaged in the image to be denoised by molecular cloning.

[0047] Method B2: The fluorescent molecules bind to the target antibody and label the structure to be imaged in the image to be denoised, where the target antibody specifically binds to the structure to be imaged in the image to be denoised.

[0048] In this embodiment, the types of target antibodies can be ordinary antibodies and nanobodies. Exemplarily, a fluorescent molecule A can be fused and expressed with the structure to be imaged by molecular cloning (a method for purifying and amplifying specific DNA fragments at the molecular level) (referring to constructing a fusion gene by ligating the exogenous protein gene to the 3' end of another gene for expression), so as to label the structure to be imaged. The structure to be imaged can also be labeled by binding a fluorescent molecule B to a nanobody A, and the target antibody A specifically binds to the structure to be imaged. It should be noted that the examples here are only illustrative and do not limit the embodiments of the present disclosure.

[0049] In a possible implementation manner, in the above step S102, the photoswitchable fluorescent molecule includes: a fluorescent protein with photoswitchable properties, a fluorescent dye with photoswitchable properties, or a quantum dot.

[0050] In this embodiment, for the fluorescent protein with photoswitchable properties, specifically, in a possible implementation manner, the fluorescent protein with photoswitchable properties responds to light modulation and has two states: a bright state and a dark state, including: a photoswitchable fluorescent protein, a reversible photoswitchable fluorescent protein, and a photoblinking fluorescent protein. In a possible implementation manner, the photoswitchable fluorescent protein includes: Skylan-S, rsEGFP2, or Dronpa, etc. Here, Skylan-S (a novel photoswitchable protein) is a monomer that can be used for live cell imaging and has extremely high optical stability and switching characteristics. rsEGFP2 (reversibly switchable enhanced green fluorescent protein 2) has the characteristics of easy crystallization and good optical properties. Dronpa (erasable optical protein) is a monomeric fluorescent protein extracted from Pectiniidae (a kind of stony coral polyp) and has distinctive photochromic properties, and the fluorescence can be controlled to turn on and off with two different excitation lights.

[0051] For the fluorescent dye with photoswitchable properties, specifically, in a possible implementation manner, the fluorescent dye with photoswitchable properties includes: a photoswitchable fluorescent dye and a photoblinking fluorescent dye. During implementation, the fluorescent dye with photoswitchable properties can be, for example, fluorescein, cyanine dyes, rhodamine fluorescent dyes, etc. It should be noted that the types of the fluorescent protein with photoswitchable properties and the fluorescent dye with photoswitchable properties are not limited in the embodiments of the present disclosure.

[0052] Quantum dots, also known as semiconductor nanocrystals, can lock electrons in a very tiny three-dimensional space. The movement of electrons is restricted. Applying a certain electric field or light to it will emit light of a specific frequency, and the emission frequency changes with the change of the particle size. By adjusting the size of the quantum dots, the color of their emission can be controlled.

[0053] In a possible implementation manner, the types of light emitted by the photoswitchable fluorescent molecule include: blue light, green light, yellow light, red light, or light with a far-infrared wavelength, etc. The types of light emitted by the photoswitchable fluorescent molecule are not limited in the embodiments of the present disclosure.

[0054] In a possible implementation manner, in the above step S102, the preset image denoising neural network is generated according to the following method:

[0055] Step C1: Obtain a plurality of fluorescence image sequences, where the fluorescence image sequence includes at least one bright-state image and at least one dark-state image.

[0056] Step C2: Perform first image processing on each bright-state image and each dark-state image to obtain input images of the plurality of fluorescence image sequences.

[0057] Step C3: Perform second image processing on each bright-state image and each dark-state image to obtain ground-truth images of the plurality of fluorescence image sequences, where the signal-to-noise ratio of the input image is less than the signal-to-noise ratio of the ground-truth image.

[0058] Step C4: Generate a training data set according to each input image and each ground-truth image; use the training data set to train an initial image denoising neural network to obtain the preset image denoising neural network.

[0059] In this embodiment, in order to obtain the preset image denoising neural network, a training data set can be obtained first. Specifically, a plurality of fluorescence image sequences can be obtained first. Specifically, in a possible implementation manner, the fluorescence image sequence is obtained by an optical imaging technique, and the optical imaging technique includes: wide-field fluorescence microscopy, confocal fluorescence microscopy, structured illumination imaging, single-molecule localization microscopy, stimulated emission depletion super-resolution microscopy, optical fluctuation super-resolution imaging, single-photon fluorescence imaging, two-photon microscopy, multi-photon microscopy, or photoacoustic imaging, etc.

[0060] For example, after labeling the structure to be imaged with a photoswitchable fluorescent molecule, the structure to be imaged can be irradiated with modulated light. When the photoswitchable fluorescent molecule is in the luminescent state, an optical imaging device, such as a wide-field fluorescence microscope, can be used to collect a fluorescence image to obtain one or more bright-state images. Since the structure to be imaged is labeled with the photoswitchable fluorescent molecule, each bright-state image contains signal data and noise data generated by the structure to be imaged, where the noise data includes noise data such as Gaussian noise, Poisson noise, and sample autofluorescence. When most of the photoswitchable fluorescent molecules are in the non-luminescent state, a fluorescence image is collected to obtain one or more dark-state images, and each dark-state image contains weak signal data and noise data. Finally, multiple fluorescence image sequences can be obtained. For example, fluorescence image sequence a: bright-state image a1, dark-state image a2; fluorescence image sequence b: bright-state image b1, bright-state image b2, dark-state image b3; fluorescence image sequence c: bright-state image c1, bright-state image c2, dark-state image c3, dark-state image c4. The number of fluorescence image sequences is not limited in the embodiments of the present disclosure and can be set according to actual situations. For example, the number can be set to 500. The optical imaging technology used to obtain the fluorescence image sequence is not limited in the embodiments of the present disclosure and can be selected according to actual needs.

[0061] After obtaining multiple fluorescence image sequences, a first image processing can be performed on each bright-state image and each dark-state image to obtain an input image of the fluorescence image sequence. Specifically, in one possible implementation, after obtaining multiple fluorescence image sequences, the following method D1 / method D2 is used to perform a first image processing on each bright-state image and each dark-state image to obtain an input image of multiple fluorescence image sequences, including:

[0062] Method D1: Arbitrarily select one of the bright-state images as the input image.

[0063] Method D2: Arbitrarily select a group of first fluorescence image pairs from multiple fluorescence image sequences, subtract the bright-state image in the first fluorescence image pair from the dark-state image to obtain a first subtracted image; and use the first subtracted image as the input image.

[0064] In this embodiment, for method D1, any one of the bright-state images can be used as the input image. Exemplarily, if the multiple fluorescence image sequences include bright-state image 1, bright-state image 2, and bright-state image 3, then any one can be selected from bright-state image 1, bright-state image 2, and bright-state image 3. For example, bright-state image 1 can be selected as the input image. It should be noted that the examples here are only illustrative and do not limit the embodiments of the present application.

[0065] For method D2, a first pair of fluorescence images can be arbitrarily selected from multiple fluorescence image sequences. The bright-state image and the dark-state image in the first pair of fluorescence images are subtracted to obtain a first subtracted image. The first subtracted image is used as the input image. Exemplarily, the multiple fluorescence image sequences include fluorescence image sequence d and fluorescence image sequence e. Fluorescence image sequence d includes bright-state images d1, d2, d3 and dark-state image d4, and fluorescence image sequence e includes bright-state images e1, e2 and dark-state image e3. A first pair of fluorescence images is arbitrarily selected from each sequence. For example, if the first pair of fluorescence images includes bright-state image d1 and dark-state image e1, the pixel values of the corresponding pixels of bright-state image d1 and dark-state image e1 can be subtracted to obtain a first subtracted image, and the first subtracted image can be used as the input image. It should be noted that the examples here are only illustrative and do not limit the embodiments of the present application.

[0066] Next, second image processing can be performed on each bright-state image and each dark-state image to obtain true-value images of multiple fluorescence image sequences, where the signal-to-noise ratio of the input image is less than that of the true-value image. Specifically, in one possible implementation, after obtaining multiple fluorescence image sequences, the following method E1 / method E2 is used to perform second image processing on each bright-state image and each dark-state image to obtain true-value images of multiple fluorescence image sequences:

[0067] Method E1: For any multiple pairs of second fluorescence images in multiple fluorescence image sequences, the bright-state image and the dark-state image in each pair of second fluorescence images are subtracted to obtain each second subtracted image; a true-value image is generated based on each second subtracted image.

[0068] Method E2: Using an optical lock-in amplification algorithm, multiple fluorescence image sequences are processed to obtain true-value images.

[0069] In this embodiment, for method E1, for any number of pairs of second fluorescence images in multiple fluorescence image sequences, the pixel values of the corresponding pixels of the bright-state image and the dark-state image in each pair of second fluorescence images can be subtracted to obtain each second subtracted image. Then, based on each second subtracted image, a true-value image can be generated. Exemplarily, the multiple fluorescence image sequences include fluorescence image sequence f and fluorescence image sequence g. For fluorescence image sequence f, the pixel values of the corresponding pixels of the bright-state image f1 and the dark-state image f2 in fluorescence image sequence f can be subtracted to obtain the second subtracted image 1; for fluorescence image sequence g, the pixel values of the corresponding pixels of the bright-state image g1 and the dark-state image g3 in fluorescence image sequence g can be subtracted to obtain the second subtracted image 2. Then, weighted average processing can be performed on each second subtracted image to obtain the true-value image. For example, it can be set that the weights of the second subtracted images 1 and 2 are both 1 / 2, then the true-value image = 1 / 2 × the second subtracted image 1 + 1 / 2 × the second subtracted image 2. Regarding the weights of each second subtracted image, the embodiments of the present disclosure are not limited and can be set according to actual needs. It should be noted that the examples here are only illustrative and do not limit the embodiments of the present disclosure.

[0070] For method E2, exemplarily, the fluorescence image sequence h, the fluorescence image sequence i, the fluorescence image sequence j, and the fluorescence image sequence k can be processed using the optical lock-in amplification algorithm (OLID algorithm) to obtain the true-value image. It should be noted that the examples here are only illustrative and do not limit the embodiments of the present disclosure.

[0071] Then, based on each input image and each true-value image, a training data set can be generated. In specific implementation, for example, an image acquisition device can be used to acquire 1000 frames of images. Assuming that 1 input image and 1 true-value image can be generated according to 20 fluorescence image sequences, then the 1000 frames of images can be divided into: fluorescence image sequence group 1, fluorescence image sequence group 2,..., fluorescence image sequence group 50 in the order of acquisition. According to fluorescence image sequence group 1, input image 1 and true-value image 1 can be generated, according to fluorescence image sequence group 2, input image 2 and true-value image 2 can be generated;...; according to fluorescence image sequence group 50, input image 50 and true-value image 50 can be generated. The input image and the true-value image form a training pair, and 50 training pairs constitute the training data set.

[0072] After obtaining the training data set, the initial image denoising neural network can be trained using the training data set to obtain a preset image denoising neural network. In a specific application scenario, in one possible implementation, the deep learning model on which the initial image denoising neural network is based includes: a supervised model, an unsupervised model, or a self-supervised model, etc.; among them, the supervised model includes the DnCNN model or the Noise2Clean model, etc.; the unsupervised model includes the BM3D model or the BM4D model, etc.; the self-supervised model includes the Deep CAD model, the Deep CAD-RT model, the Noise2Noise model, the Noise2Void model, the Self2Self model, the Self2Self+ model, the Neighbor2Neighbor model, the Noise2Fast model, the Dilated Blind-Spot Network model, the blind2unblind model, or the Deep Image Prior model, etc.

[0073] Here, the Denoising Convolutional Neural Network (DnCNN) is a deep residual network specifically designed for image denoising. It learns the noise features in the image by stacking multiple convolutional layers and residual connections and removes the noise.

[0074] The Block Matching 3D (BM3D) can perform simple denoising through original image matching to form a basic estimate, and then perform more detailed denoising through the original image and the basic estimate to further improve the peak signal-to-noise ratio. The Block Matching 4D filtering (BM4D) model is similar in principle to the BM3D model and can extend two-dimensional spatial denoising to three-dimensional space.

[0075] The self-supervised deep learning model (DeepCAD) can achieve network training only using a single low signal-to-noise ratio calcium imaging sequence without any high signal-to-noise ratio observation images, can suppress detection noise, and increase the signal-to-noise ratio by more than 10 times. A self-supervised deep learning model for real-time noise suppression (DeepCAD-RT) has great advantages in functional imaging. Noise2Noise uses paired noisy images of the same scene as the training image pairs and can achieve the same training effect under supervised conditions. The Noise2Void model can use the noisy image itself as supervision and prevent learning the identity mapping through the blind spot network. A single-image denoising network model for self-supervised learning (Self2Self model) only needs to input a single noisy image (without ground truth) for training. The Self2Self+ model is an optimization of the Self2Self model. The Neighbor2Neighbor model is proposed based on the Noise2Noise model, replacing the noise-clean image pairs with noise-noise image pairs, and solving the problem of collecting a large number of noise-clean image data pairs. The self-supervised image denoising model with visible blind spots (blind2unblind) can overcome the information loss in blind spot-driven denoising methods. The dilated blind spot network model (D-BSN) can learn the denoising model only from real noisy images. The DeepImage Prior model is an image denoising method proposed based on the principle of the implicit prior information inherently carried by the structure of the neural network itself.

[0076] During implementation, a neural network for denoising the to-be-trained image can also be generated based on other deep learning models, which is not limited in the embodiments of the present disclosure. For example, a neural network for denoising the to-be-trained image can also be generated based on the Recorrupted2Recorrupted model, etc.

[0077] In this embodiment, the training data set can be trained using the initial image denoising neural network generated based on the above deep learning model. The above deep learning model can efficiently remove various noises in the imaging process, such as Gaussian noise, Poisson noise, etc., thereby greatly improving the denoising effect of the preset image denoising neural network.

[0078] During implementation, each input image in the training sample set can be input into the initial image denoising neural network to obtain multiple predicted images, and then based on a preset loss function, the parameters in the initial image denoising neural network can be updated according to the multiple predicted images and the corresponding ground truth images to obtain the preset image denoising neural network.

[0079] In a possible implementation manner, deconvolution processing is performed on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image, specifically including:

[0080] Step F: Based on the Richardson-Lucy algorithm, perform deconvolution on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image.

[0081] It has been found through research that for a blurred image without any noise, the original image can be completely restored by the deconvolution method of Richardson-Lucy (RL). However, in the actual imaging process, affected by uncertain factors such as optical system noise and point spread function (PSF), the RL deconvolution algorithm often causes more artifacts in the image. Moreover, as the number of deconvolution iterations increases, the noise signal will be amplified, and the deconvolution will further bring more artifacts to the image.

[0082] Based on this, in this embodiment, based on the Richardson-Lucy algorithm, deconvolution can be performed on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image. Since the denoised image is the image processed by a preset image denoising neural network, and the training data set used by the preset image denoising neural network is obtained by using the optical switching characteristics of the opto-switchable fluorescent molecules, the robustness, stability, and generality of the preset image denoising neural network are relatively strong, and the denoising effect has been greatly improved. Therefore, the noise of the denoised image is extremely low. Therefore, by using the RL algorithm to perform deconvolution on the denoised image, a completely restored original image can be obtained, improving the spatio-temporal resolution of the image, so as to achieve the purpose of single-frame super-resolution imaging.

[0083] During specific implementation, the system point spread function can be determined according to the following formula (1):

[0084]

[0085] where I0 is the normalization coefficient, J1 is the Bessel function of the first kind, λ is the wavelength, NA represents the numerical aperture, and ρ represents the radial distance in the polar coordinate system. When the fluorescence emitted by the sample passes through the optical system, due to the effect of diffraction, the image formed on the camera is the convolution of the fluorescence emitted by the sample and the system point spread function. Therefore, in order to better conform to the diffraction process of the optical system, the Bessel function of the first kind can be used to represent the system point spread function.

[0086] Considering the influence of noise in imaging, too many deconvolution iterations will produce artifacts, and insufficient iterations cannot improve the image resolution well. Therefore, the number of deconvolution iterations can be determined according to the quality of the denoised image. Specifically, the number of iterations can be determined by calculating the Fourier Ring Correlation (FRC) resolution of two consecutive frames of images in the actual sample to ensure that the image resolution reaches the highest and there are no obvious reconstruction artifacts.

[0087] Specifically, in a possible implementation, the imaging objects of the deconvolution super-resolution imaging method based on deep learning denoising include: fixed cells, living cells, tissues, living animals, fungi, or fluorescent standard samples, etc. The present disclosure embodiments do not limit the imaging objects of the method. The present disclosure embodiments can achieve fast 3D super-resolution imaging for fixed cells, and for living cell tissues, etc., a series of dynamic images changing with time can be obtained to reveal the rapid dynamic changes of subcellular structures.

[0088] Figure 2 FIG. shows a schematic flow chart of a deconvolution super-resolution imaging method based on deep learning denoising provided by another embodiment of the present disclosure. As can be seen from Figure 2 this, this embodiment includes the following steps S201 - step S209:

[0089] Step S201: Use a photoswitchable molecular marker structure to obtain multiple fluorescence image sequences, where the fluorescence image sequences include at least one bright-state image and at least one dark-state image. During implementation, the fluorescence image sequences are obtained through optical imaging techniques, and the optical imaging techniques include: wide-field fluorescence microscopy imaging technique, confocal fluorescence microscopy imaging technique, structured illumination imaging technique, single-molecule localization microscopy imaging technique, stimulated emission depletion super-resolution microscopy imaging technique, optical fluctuation super-resolution imaging technique, single-photon fluorescence imaging technique, two-photon microscopy imaging technique, multi-photon microscopy imaging technique, or photoacoustic imaging technique, etc.

[0090] Next, taking a photoswitchable fluorescent protein as an example, specifically describe how to obtain multiple fluorescence image sequences of target molecules. First, through cell transfection technology (which refers to the technology of introducing exogenous molecules such as DNA, RNA, etc. into eukaryotic cells), fuse F-actin with a photoswitchable fluorescent protein (RSFP) and express it in eukaryotic cells: human osteosarcoma cell U-2OS. After 36 hours, fix the cytoskeleton with a fixing solution, and use a self-made or commercial fluorescence microscopy imaging platform to collect fluorescence images. Among them, the microscope main body model is model A, using 405nm laser and 488nm laser, 150×, 1.45NA objective lens, and the camera model is model B, with a pixel size of 6.5μm. Figure 3 FIG. shows a laser modulation timing diagram provided by an embodiment of the present disclosure. Refer to Figure 3 the shown laser modulation mode. The 405nm and 488nm lasers can be turned on simultaneously. After starting to take pictures and collect images, turn off the 405nm laser to obtain an image sequence; turn on the 488nm laser, and after starting to take pictures and collect images, turn on the 405nm laser to obtain an image sequence. Finally, a series of fluorescence images with repeatedly changing switch states can be obtained, that is, multiple fluorescence image sequences.

[0091] Step S202: Use any one of the bright-state images as the input image. In implementation, a first pair of fluorescence images can also be arbitrarily selected from multiple fluorescence image sequences, subtract the dark-state image from the bright-state image in the first pair of fluorescence images to obtain a first subtracted image; and use the first subtracted image as the input image.

[0092] Step S203: For any multiple pairs of second fluorescence images in multiple fluorescence image sequences, subtract the dark-state image from the bright-state image in each pair of second fluorescence images to obtain respective second subtracted images.

[0093] Step S204: Generate a ground-truth image based on each second subtracted image. In implementation, an optical lock-in amplification algorithm can also be used to process multiple fluorescence image sequences to obtain a ground-truth image. Among them, the signal-to-noise ratio of the input image is less than that of the ground-truth image.

[0094] Step S205: Generate a training dataset based on the input image and the ground-truth image.

[0095] Step S206: Use the training dataset to train an initial image denoising neural network to obtain a preset image denoising neural network. Here, the training dataset is generated using the optical switching characteristics of an optical switching fluorescent molecule, and the optical switching fluorescent molecule is used to label the structure to be imaged in each sample in the training dataset. The optical switching fluorescent molecule includes: a fluorescent protein with optical switching properties, a fluorescent dye with optical switching properties, or a quantum dot. Among them, the fluorescent protein with optical switching properties responds to light modulation and has two states, bright state and dark state, including: a photoswitchable fluorescent protein, a reversibly photoswitchable fluorescent protein, or a photoblinking fluorescent protein, etc. Among them, the photoswitchable fluorescent protein includes: Skylan-S, rsEGFP2, or Dronpa, etc. The fluorescent dye with optical switching properties includes: a photoswitchable dye or a photoblinking fluorescent dye, etc. The types of light emitted by the optical switching fluorescent molecule include: blue light, green light, yellow light, red light, or light with a far-infrared wavelength, etc. Among them, the deep learning model on which the initial image denoising neural network is based includes: a supervised model, an unsupervised model, or a self-supervised model, etc.; among them, the supervised model includes the DnCNN model or the Noise2Clean model, etc.; the unsupervised model includes the BM3D model or the BM4D model, etc.; the self-supervised model includes the Deep CAD model, the Deep CAD-RT model, the Noise2Noise model, the Noise2Void model, the Self2Self model, the Self2Self+ model, the Neighbor2Neighbor model, the Noise2Fast model, the Dilated Blind-SpotNetwork model, the blind2unblind model, or the Deep Image Prior model, etc.

[0096] Step S207: Preprocess the original image sampled by the camera, and use the obtained processed image as the image to be denoised. Among them, the preprocessing methods include: optical lock-in amplification algorithm processing, Gaussian filtering processing, mean filtering processing, Wiener filtering processing, deconvolution processing, etc. During implementation, the original image can also be directly used as the image to be denoised. The obtained target super-resolution imaging image can also be used as the image to be denoised, where the target super-resolution imaging image includes: Airyscan super-resolution imaging image, ISM super-resolution imaging image, A-POD super-resolution imaging image, MSSR super-resolution imaging image, etc. The imaging structure to be imaged of the image to be denoised is labeled with fluorescent molecules. For example, the fluorescent molecules can label the imaging structure to be imaged in the image to be denoised by molecular cloning. For another example, the fluorescent molecules bind to the target antibody to label the imaging structure to be imaged in the image to be denoised, where the target antibody specifically binds to the imaging structure to be imaged in the image to be denoised. During implementation, the imaging object, that is, the imaging structure to be imaged includes: fixed cells, live cells, tissues, living animals, fungi, or fluorescent standards, etc.

[0097] Step S208: Use a preset image denoising neural network to denoise the image to be denoised to obtain a denoised image. Here, the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the photoswitchable fluorescent molecule.

[0098] Step S209: Based on the Richardson-Lucy algorithm, perform deconvolution processing on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image.

[0099] Figure 4 The figure shows a comparison chart of the image processing results of direct deconvolution without denoising and deconvolution after denoising provided by the embodiments of the present disclosure. Among them, a corresponds to the wide-field image (upper left), the image obtained by directly deconvolving the wide-field image (middle right), the image obtained by deconvolving after denoising (upper right), b is the corresponding enlarged view of a (scale bar: 2μm (a); 1μm (b)), and c is the image resolution comparison chart of direct deconvolution without denoising and deconvolution after denoising. Refer to Figure 4 As shown, according to a and b, it can be seen that the image processed by deconvolution after denoising is clearer, representing higher image resolution and fewer artifacts. According to c, the nm value of the image processed by deconvolution after denoising is lower, indicating higher resolution. Thus, it can be seen that the image denoising neural network based on the photoswitching property of the photoswitchable molecule can effectively remove the noise in the image, and then through deconvolution processing, the spatio-temporal resolution of the denoised image can be greatly improved to obtain a super-resolution imaging image, thereby realizing large-field single-frame super-resolution imaging.

[0100] Those skilled in the art can understand that in the above method of the specific embodiment, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined according to its function and possible internal logic. It should be noted that in practical applications, all the above possible implementation manners can be combined arbitrarily to form possible embodiments of the present disclosure, which will not be elaborated one by one here.

[0101] Based on the same concept, an embodiment of the present disclosure also provides a deconvolution super-resolution imaging device based on deep learning denoising. Figure 5 FIG. shows a schematic structural diagram of the deconvolution super-resolution imaging device based on deep learning denoising provided by an embodiment of the present disclosure. Refer to Figure 5 As shown, the deconvolution super-resolution imaging device 500 provided by an embodiment of the present disclosure includes:

[0102] An acquisition module 501, configured to acquire an image to be denoised, and a to-be-imaged structure of the image to be denoised is labeled with a fluorescent molecule;

[0103] A denoising module 502, configured to denoise the image to be denoised by using a preset image denoising neural network to obtain a denoised image; wherein, the training data set for training the image denoising neural network is generated by using the optical switching property of an optical switching fluorescent molecule, and the optical switching fluorescent molecule is used to label the to-be-imaged structure of each sample in the training data set; the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the optical switching fluorescent molecule;

[0104] A super-resolution imaging module 503, configured to perform deconvolution processing on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

[0105] In a possible implementation manner, in the above device, the optical switching fluorescent molecule includes: a fluorescent protein with optical switching properties, a fluorescent dye with optical switching properties, or a quantum dot.

[0106] In a possible implementation manner, in the above device, the fluorescent protein with optical switching properties responds to light modulation and has two states, a bright state and a dark state, including: a photoswitchable fluorescent protein, a reversibly photoswitchable fluorescent protein, or a photoblinking fluorescent protein, etc.

[0107] In a possible implementation manner, in the above device, the photoswitchable fluorescent protein includes: Skylan-S, rsEGFP2, or Dronpa, etc.

[0108] In a possible implementation manner, in the above device, the fluorescent dye with optical switching properties includes: a photoswitchable dye or a photoblinking fluorescent dye, etc.

[0109] In a possible implementation, in the above device, the types of light emitted by the optical switch fluorescent molecules include: blue light, green light, yellow light, red light, or light with a far-infrared wavelength, etc.

[0110] In a possible implementation, the device further includes a network training module, and the network training module is used for:

[0111] Obtain a plurality of fluorescence image sequences, where the fluorescence image sequences include at least one bright-state image and at least one dark-state image;

[0112] Perform first image processing on each of the bright-state images and each of the dark-state images to obtain the input images of the plurality of fluorescence image sequences;

[0113] Perform second image processing on each of the bright-state images and each of the dark-state images to obtain the ground-truth images of the plurality of fluorescence image sequences, where the signal-to-noise ratio of the input images is less than the signal-to-noise ratio of the ground-truth images;

[0114] Generate the training dataset according to each of the input images and each of the ground-truth images; use the training dataset to train the initial image denoising neural network to obtain the preset image denoising neural network.

[0115] In a possible implementation, when the network training module performs first image processing on each of the bright-state images and each of the dark-state images to obtain the input images of the plurality of fluorescence image sequences, it is used for:

[0116] Take any one of the bright-state images as the input image; or

[0117] Select any group of first fluorescence image pairs from the plurality of fluorescence image sequences, subtract the dark-state image from the bright-state image in the first fluorescence image pair to obtain a first subtracted image; and take the first subtracted image as the input image.

[0118] In a possible implementation, when the network training module performs second image processing on each of the bright-state images and each of the dark-state images to obtain the ground-truth images of the plurality of fluorescence image sequences, it is used for:

[0119] For any multiple groups of second fluorescence image pairs in the plurality of fluorescence image sequences, subtract the dark-state image from the bright-state image in the second fluorescence image pair to obtain a second subtracted image; generate the ground-truth image according to each of the second subtracted images; or

[0120] Use the optical lock-in amplification algorithm to process the plurality of fluorescence image sequences to obtain the ground-truth images.

[0121] In a possible implementation manner, in the above-mentioned network training module, the deep learning model on which the initial image denoising neural network is based includes: a supervised model, an unsupervised model, a self-supervised model, etc.; among them, the supervised model includes a DnCNN model or a Noise2Clean model, etc.; the unsupervised model includes a BM3D model or a BM4D model, etc.; the self-supervised model includes a Deep CAD model, a Deep CAD-RT model, a Noise2Noise model, a Noise2Void model, a Self2Self model, a Self2Self+ model, a Neighbor2Neighbor model, a Noise2Fast model, a DilatedBlind-Spot Network model, a blind2unblind model, or a Deep Image Prior model, etc.

[0122] In a possible implementation manner, in the above-mentioned device, the obtaining module 501 is configured to:

[0123] Obtain an original image, and use the original image as the image to be denoised, where the original image is obtained by camera sampling; or

[0124] Preprocess the original image, and use the obtained processed image as the image to be denoised, where the preprocessing methods include: optical lock-in amplification algorithm processing, Gaussian filtering processing, mean filtering processing, Wiener filtering processing, or deconvolution processing, etc.; or

[0125] Use the obtained target super-resolution imaging image as the image to be denoised, where the target super-resolution imaging image includes: an Airyscan super-resolution imaging image, an ISM super-resolution imaging image, an A-POD super-resolution imaging image, or an MSSR super-resolution imaging image, etc.

[0126] In a possible implementation manner, in the above-mentioned device, the fluorescent molecule labels the structure to be imaged in the image to be denoised through the following method:

[0127] The fluorescent molecule labels the structure to be imaged in the image to be denoised through the method of molecular cloning; or

[0128] The fluorescent molecule binds to a target antibody to label the structure to be imaged in the image to be denoised, where the target antibody specifically binds to the structure to be imaged in the image to be denoised.

[0129] In a possible implementation manner, in the above-mentioned device, the super-resolution imaging module 503 is configured to:

[0130] Based on the Richardson-Lucy algorithm, perform deconvolution processing on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image.

[0131] In a possible implementation manner, in the above network training module, the fluorescence image sequence is acquired by optical imaging technology, and the optical imaging technology includes: wide-field fluorescence microscopy imaging technology, confocal fluorescence microscopy imaging technology, structured illumination imaging technology, single-molecule localization microscopy imaging technology, stimulated emission depletion super-resolution microscopy imaging technology, optical fluctuation super-resolution imaging technology, single-photon fluorescence imaging technology, two-photon microscopy imaging technology, multi-photon microscopy imaging technology, or photoacoustic imaging technology, etc.

[0132] In a possible implementation manner, the imaging objects of the method device include: fixed cells, living cells, tissues, living animals, fungi, or fluorescence standards, etc.

[0133] It should be noted that any of the above deconvolution super-resolution imaging devices based on deep learning denoising can implement the foregoing deconvolution super-resolution imaging method based on deep learning denoising one by one, which will not be elaborated here.

[0134] Based on the same technical concept, the embodiments of the present disclosure also provide an electronic device. Referring to Figure 6 As shown, it is a schematic structural diagram of the electronic device provided by the embodiments of the present disclosure, including a processor 601, a memory 602, and a bus 603. Among them, the memory 602 is used to store execution instructions, including an internal memory 6021 and an external memory 6022; here, the internal memory 6021 is also called the main memory, which is used to temporarily store the operation data in the processor 601 and the data exchanged with the external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the internal memory 6021. When the electronic device 600 runs, the processor 601 communicates with the memory 602 through the bus 603, so that the processor 601 executes the following instructions:

[0135] Obtain an image to be denoised, and the structure to be imaged of the image to be denoised is labeled with a fluorescent molecule;

[0136] Use a preset image denoising neural network to denoise the image to be denoised to obtain a denoised image; wherein, the training data set for training the image denoising neural network is generated using the optical switching characteristics of an optical switch fluorescent molecule, and the optical switch fluorescent molecule is used to label the structure to be imaged of each sample in the training data set; the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the optical switch fluorescent molecule;

[0137] Perform deconvolution processing on the denoised image to obtain the super-resolution imaging image corresponding to the denoised image.

[0138] Among them, the specific processing procedure of the processor 601 may refer to the description in the foregoing method embodiment, and will not be elaborated here.

[0139] In addition, the embodiments of the present disclosure further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the deconvolution super-resolution imaging method based on deep learning denoising in the foregoing method embodiment. Among them, the storage medium may be a volatile or non-volatile computer-readable storage medium.

[0140] The embodiments of the present disclosure further provide a computer program product, which carries program codes. The instructions included in the program codes can be used to execute the steps of the deconvolution super-resolution imaging method based on deep learning denoising in the foregoing method embodiment. For details, refer to the foregoing method embodiment and will not be elaborated here.

[0141] Among them, the foregoing computer program product can be specifically implemented in the form of hardware, software, or a combination thereof. In an alternative embodiment, the computer program product is specifically embodied as a computer storage medium. In another alternative embodiment, the computer program product is specifically embodied as a software product, such as a Software Development Kit (SDK), etc.

[0142] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here. In the several embodiments provided by the present disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces. The indirect coupling or communication connection of the devices or units may be in an electrical, mechanical, or other form.

[0143] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, in each of the embodiments of the present disclosure, each functional unit may be integrated into one processing unit, may exist separately physically for each unit, or two or more units may be integrated into one unit.

[0145] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present disclosure. The aforementioned storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0146] The above are only the specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present disclosure can easily think of changes or substitutions, which should all be covered by the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. A deconvolution super-resolution imaging method based on deep learning denoising, characterized in that The method includes: Obtaining a to-be-denoised image, where the to-be-imaged structure of the to-be-denoised image is labeled with a fluorescent molecule; Denoising the to-be-denoised image by using a preset image denoising neural network to obtain a denoised image; wherein, the training data set for training the image denoising neural network is generated by using the optical switching characteristics of an optically switchable fluorescent molecule, and the optically switchable fluorescent molecule is used to label the to-be-imaged structures of the samples in the training data set; the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the optically switchable fluorescent molecule; Performing deconvolution processing on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

2. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 1, wherein, The optically switchable fluorescent molecule includes: a fluorescent protein with optical switching properties, a fluorescent dye with optical switching properties, or a quantum dot.

3. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 2, wherein The fluorescent protein with optical switching properties responds to light modulation and has two states, a bright state and a dark state, including: an optically switchable fluorescent protein, a reversibly optically switchable fluorescent protein, or a photoblinking fluorescent protein, etc.

4. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 3, characterized in that The optically switchable fluorescent protein includes: Skylan-S, rsEGFP2, or Dronpa, etc.

5. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 2, wherein The fluorescent dye with optical switching properties includes: an optically switchable dye or a photoblinking fluorescent dye, etc.

6. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 1, wherein The types of light emitted by the optically switchable fluorescent molecule include: blue light, green light, yellow light, red light, or light with a far-infrared wavelength, etc.

7. The deconvolution super-resolution imaging method based on deep learning denoising according to claim 1, characterized in that The preset image denoising neural network is generated according to the following method: Obtaining a plurality of fluorescence image sequences, where each fluorescence image sequence includes at least one bright state image and at least one dark state image; Performing first image processing on each of the bright state images and each of the dark state images to obtain input images of the plurality of fluorescence image sequences; Performing second image processing on each of the bright state images and each of the dark state images to obtain ground truth images of the plurality of fluorescence image sequences, where the signal-to-noise ratio of the input images is less than the signal-to-noise ratio of the ground truth images; Generating the training data set according to each of the input images and each of the ground truth images; training an initial image denoising neural network by using the training data set to obtain the preset image denoising neural network.

8. A deconvolution super-resolution imaging device based on deep learning denoising, characterized in that, The device includes: An obtaining module, configured to obtain a to-be-denoised image, where the to-be-imaged structure of the to-be-denoised image is labeled with a fluorescent molecule; A denoising module, configured to denoise the to-be-denoised image by using a preset image denoising neural network to obtain a denoised image; wherein, the training data set for training the image denoising neural network is generated by using the optical switching characteristics of an optically switchable fluorescent molecule, and the optically switchable fluorescent molecule is used to label the to-be-imaged structures of the samples in the training data set; the emission spectrum of the fluorescent molecule overlaps with the emission spectrum of the optically switchable fluorescent molecule; A super-resolution imaging module, configured to perform deconvolution processing on the denoised image to obtain a super-resolution imaging image corresponding to the denoised image.

9. An electronic device, including: A processor; And a memory arranged to store computer-executable instructions, characterized in that when the executable instructions are executed, the processor executes the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium storing one or more programs, characterized in that, When the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to perform the steps of the method according to any one of claims 1-7.

Images