Self-supervised microscopic image super-resolution processing method and system
Through self-supervised learning and denoising deconvolution neural network structure, the microscopic image super-resolution network is trained using physical prior information of optical imaging models, which solves the problems of cumbersome parameter debugging and poor noise robustness in the existing technology, and realizes the High resolution improvement and application scope expansion of microscopic images.
Patent Information
- Application Number
- CN202310735660.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-20
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-06-20
AI Technical Summary
The existing computational super-resolution method has problems such as cumbersome parameter debugging, poor noise robustness and dependence on high-quality training data in microscopic image processing. Especially in the dynamic characteristics of living biological samples and photobleaching conditions, it is difficult to obtain sufficient training sets, which limits the Application of deep learning super-resolution image processing.
Using self-supervised learning method, the image super-resolution neural network is trained using physical prior information of the optical imaging model. Through the neural network structure of the denoising and deconvolution parts, the original microscopic image data with low signal-to-noise ratio and low resolution are used. Perform training and even complete model training without high and low signal-to-noise ratio/resolution training data.
The high fidelity and quantifiable characteristics of microscopic images are achieved, without the need for specific adjustment of hyperparameters, greatly expanding the application range of computational super-resolution methods, and is suitable for the long-term range of low excitation light power for living biological samples. Lossless imaging and high-speed dynamic process observation.
Smart Images

Figure CN116721017B_ABST
Abstract
Description
Technical Field
[0001] The present application generally relates to a microscopic image super-resolution processing method and system, and in particular to a self-supervised microscopic image super-resolution processing method and system. Background Art
[0002] Fluorescence microscopy is an important tool to promote the development of life science research. In recent years, super-resolution microscopy has been able to break through the optical diffraction limit and dynamically observe the fine structure of living cells with a resolution of less than 200 nanometers. However, the hardware improvement of optical microscopy systems in the prior art has reached its limit. In this case, the use of super-resolution microscopy to observe biological samples will be accompanied by the sacrifice of other imaging performance indicators such as imaging speed and time course while achieving improved spatial resolution. These imaging performance indicators are equally important for studying and analyzing the process of biological samples. In recent years, computational super-resolution methods can immediately improve the resolution of microscopic images by calculation, thereby significantly expanding the scope of application of optical microscopy systems.
[0003] Generally speaking, existing computational super-resolution methods can be divided into two categories: the first category is super-resolution algorithms based on traditional analytical models, such as deconvolution algorithms; the second category is methods based on deep learning, such as super-resolution convolutional neural networks. The first category of methods is usually based on prior assumptions about biological samples and microscopic fluorescence images, and uses specific analytical models to iteratively optimize the resolution of microscopic fluorescence images. However, such methods often have multiple hyperparameters, and the algorithm output depends to a large extent on the selection of parameters. Therefore, users need to specifically debug these parameters for the images to be processed, which is time-consuming and laborious. In addition, since the validity of the prior of biological samples or microscopic fluorescence images cannot be guaranteed in actual experiments, such analytical algorithms lack robustness to noise in microscopic fluorescence images, especially under low signal-to-noise ratio input conditions, which are prone to reconstruction artifacts. On the other hand, in recent years, super-resolution algorithms based on deep learning can learn the end-to-end mapping relationship between high-resolution and low-resolution images from a large number of high-quality training sets, so that the resolution of microscopic images can be improved without physical modeling of imaging or noise models, and can achieve processing effects far exceeding traditional analytical algorithms. However, the training of deep learning super-resolution network models requires a large number of paired low-resolution input images and high-quality true super-resolution images, and the performance of the super-resolution neural network depends largely on the quality and quantity of the training data. Due to the highly dynamic characteristics of living biological samples and photobleaching, collecting high-quality training sets often requires a lot of manpower and time costs, and even in some specific applications, it is impossible to obtain enough training sets. These factors have seriously hindered the application of super-resolution image processing methods based on deep learning in super-resolution analysis of microscopic images. Summary of the invention
[0004] In view of the above problems, the present application aims to propose a novel image super-resolution reconstruction technology solution. The solution can fully utilize the physical prior information of the optical imaging model to train the image super-resolution neural network, or even train the image super-resolution neural network in a self-supervised learning manner, so that even if there is no high or low signal-to-noise ratio / resolution training data, only low signal-to-noise ratio and low resolution original microscopic image data can be used to complete the training of the microscopic image super-resolution neural network model.
[0005] According to one aspect of the present application, a self-supervised microscopic image super-resolution processing method is provided, comprising:
[0006] Using an optical imaging system to collect raw fluorescence image data of biological samples;
[0007] Depending on the type of the optical imaging system, the collected raw fluorescence image data is preprocessed by a computer to obtain a training set, and the training set is used in the computer to train a neural network for image denoising and super-resolution processing, wherein the neural network includes a denoising part and a deconvolution part, wherein the training set includes input image data and true value input image data, pixel blocks at the same position are randomly extracted from the input image data and the true value input image data, and are randomly rotated and flipped to serve as a training input image and a training target image respectively, the training input image is first processed by the denoising part to become a denoised image, and then processed by the deconvolution part to become a deconvolution image, the denoising image and the training target image are used to calculate the denoising loss together, the deconvolution image is subjected to degradation processing depending on the type of the optical imaging system, and the deconvolution loss is calculated together with the training target image, and in the training process of the neural network, the loss function includes the denoising loss and the deconvolution loss or only includes the deconvolution loss;
[0008] The trained neural network is used to process the original fluorescent image data or the additional fluorescent image data collected from the same biological sample by the optical imaging system.
[0009] Optionally, when the optical imaging system is a wide-field microscopy imaging system, a scanning confocal microscopy imaging system, a light sheet illumination microscopy imaging system, or a two-photon scanning imaging system, the original fluorescence image data is one or more two-dimensional noise images of a biological sample independently collected by the optical imaging system, and the training set includes a plurality of groups of training noise image pairs, wherein the plurality of groups of training noise images are generated by performing image re-degradation processing on the original fluorescence image data;
[0010] The image re-degradation processing comprises:
[0011] For a two-dimensional noise image, a computer is used to randomly generate a normal distribution random variable with the same image dimension as the two-dimensional noise image, with a mean of 0 and a variance of 1, and a pair of fluorescence perturbation images are generated using the following formula:
[0012]
[0013] y A =y+α·Σ·z (2)
[0014] y B =y-α -1 ·Σ·z (3)
[0015] In the above formulas (1), (2), and (3), α, β1, and β2 are constants randomly generated by a computer, and α = 0.2 to 5.0, β1 = 0.2 to 5, and β2 = 0.5σ 2 ~1.5σ 2 , y represents the two-dimensional noise image, z represents the normal distribution random variable, y A represents one of the pair of fluorescence perturbation images, y B represents the other fluorescence perturbation image in the pair of fluorescence perturbation images, σ 2 is the background noise variance of the camera used in the optical imaging system. This pair of fluorescence perturbation images constitutes a set of training noise image pairs;
[0016] When training the neural network, one noise image in a set of training noise image pairs is selected from the plurality of sets of training noise image pairs as input image data, and another noise image in the selected set of training noise image pairs is used as true value input image data;
[0017] After the neural network is trained, the trained neural network is used to process the two-dimensional noise image of the biological sample collected by the optical imaging system or to process the original fluorescence image data.
[0018] Optionally, when training the neural network, the degradation processing of the deconvolved image is the convolution of the deconvolved image with a point spread function of the optical imaging system.
[0019] Optionally, when the optical imaging system is a wide-field microscopic imaging system or a scanning confocal microscopic imaging system or a light sheet illumination microscopic imaging system or a two-photon scanning imaging system, the original fluorescence image data is a three-dimensional noise image volume stack of a biological sample independently collected by the optical imaging system, the training set includes a first noise image volume stack and a second noise image volume stack amplified by the three-dimensional noise image volume stack through axial interval sampling processing, and the denoised image and the deconvolved image are a denoised image volume stack and a deconvolved image volume stack, respectively;
[0020] The axial interval sampling process includes: extracting odd-numbered layers or even-numbered layers of a three-dimensional noise image volume stack independently collected by the optical imaging system as the first noise image volume stack, and correspondingly extracting even-numbered layers or odd-numbered layers as the second noise image volume stack;
[0021] When training the neural network, one of the first noise image volume stack and the second noise image volume stack corresponding to each other is selected as the input image data, and the other of the first noise image volume stack and the second noise image volume stack corresponding to each other is selected as the true value input image data;
[0022] After the neural network is trained, the trained neural network is used to process the three-dimensional noise image stack of the biological sample collected by the optical imaging system or the original fluorescence image data.
[0023] Optionally, when training the neural network, the degradation processing of the deconvolved image is the convolution of the deconvolved image volume stack with the three-dimensional point spread function of the optical imaging system.
[0024] Optionally, in the case where the optical imaging system is a structured light illumination super-resolution microscopy imaging system, the original fluorescence image data is a structured light illumination image of a biological sample collected by the optical imaging system, and the training set is two structured light illumination super-resolution images reconstructed from the structured light illumination image by structured light illumination super-resolution;
[0025] When training the neural network, one of the two structured light illumination super-resolution images is used as the input image data, and the other of the two structured light illumination super-resolution images is used as the true value input image data;
[0026] After the neural network training is completed, the trained neural network is used to process the structured light illumination image of the biological sample collected by the optical imaging system after structured light illumination super-resolution reconstruction.
[0027] Optionally, the two structured light illumination super-resolution images are generated by using two groups of structured light illumination original image sequences independently collected by the optical imaging system from the biological sample and reconstructing them through structured light illumination super-resolution.
[0028] Optionally, after the optical imaging system independently collects a group of structured light illumination original image sequences for the biological sample, each structured light illumination original image or multiple structured light illumination original images in the group of structured light illumination original images are respectively subjected to image re-degradation processing, thereby generating multiple pairs of re-degraded images, and the first re-degraded image in each pair of re-degraded images is extracted to form a first group of structured light illumination re-degraded image sequences, and the second re-degraded image in each pair of re-degraded images is extracted to form a second group of structured light illumination re-degraded image sequences, and the two structured light illumination super-resolution images are generated by structured light illumination super-resolution reconstruction using the first group of structured light illumination re-degraded image sequences and the second group of structured light illumination re-degraded image sequences.
[0029] Optionally, when training the neural network, the degradation processing of the deconvolved image is the convolution of the deconvolved image with a super-resolution point spread function of the optical imaging system.
[0030] Optionally, the model of the neural network for image denoising and super-resolution processing includes but is not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.
[0031] According to another aspect of the present application, a self-supervised microscopic image super-resolution processing system is provided, which is configured to process raw fluorescence image data of a biological sample collected by an optical imaging system, comprising:
[0032] Microscopic image preprocessing module; and
[0033] A denoising and super-resolution reconstruction module, wherein the microscopic image preprocessing module is configured to perform image preprocessing on the collected raw fluorescence image data using a computer to obtain a training set depending on the type of the optical imaging system, and the denoising and super-resolution module is configured with a neural network to perform denoising and super-resolution processing on the image, wherein the neural network includes a denoising part and a deconvolution part,
[0034] The neural network is trained in a computer using the training set, wherein the training set includes input image data and true value input image data, pixel blocks at the same position are randomly extracted from the input image data and the true value input image data, and are randomly rotated and flipped to serve as a training input image and a training target image respectively, the training input image is first processed by the denoising part to be a denoised image, and then processed by the deconvolution part to be a deconvolution image, the denoising loss is calculated together with the training target image, the deconvolution loss is calculated together with the training target image after the deconvolution image is subjected to degradation processing depending on the type of the optical imaging system, and in the training process of the neural network, the loss function includes the denoising loss and the deconvolution loss or only includes the deconvolution loss;
[0035] The denoising and super-resolution reconstruction module is configured to process the original fluorescence image data using the trained neural network or the additional fluorescence image data collected from the same biological sample using the optical imaging system.
[0036] Optionally, when the optical imaging system is a wide-field microscopy imaging system, a scanning confocal microscopy imaging system, a light sheet illumination microscopy imaging system, or a two-photon scanning imaging system, the original fluorescence image data is one or more two-dimensional noise images of a biological sample independently collected by the optical imaging system, and the training set includes a plurality of groups of training noise image pairs, wherein the plurality of groups of training noise images are generated by performing image re-degradation processing on the original fluorescence image data;
[0037] The image re-degradation processing comprises:
[0038] For a two-dimensional noise image, a computer is used to randomly generate a normal distribution random variable with the same image dimension as the two-dimensional noise image, with a mean of 0 and a variance of 1, and a pair of fluorescence perturbation images are generated using the following formula:
[0039]
[0040] y A =y+α·Σ·z (2)
[0041] y B =y-α -1 ·Σ·z (3)
[0042] In the above formulas (1), (2), and (3), α, β1, and β2 are constants randomly generated by computers, α = 0.2 to 5.0, β1 = 0.2 to 5, and β2 = 0.5σ. 2 ~1.5σ 2 , σ 2is the noise variance of the camera used in the optical imaging system, y represents the two-dimensional noise image, z represents the normal distribution random variable, y A represents one of the pair of fluorescence perturbation images, y B represents another fluorescence perturbation image in the pair of fluorescence perturbation images, and the pair of fluorescence perturbation images constitute a set of noise image pairs for training;
[0043] When training the neural network, one noise image in a set of training noise image pairs is selected from the plurality of sets of training noise image pairs as input image data, and another noise image in the selected set of training noise image (A / B) pairs is used as true value input image data;
[0044] After the neural network is trained, the trained neural network is used to process the two-dimensional noise image of the biological sample collected by the optical imaging system or to process the original fluorescence image data.
[0045] Optionally, when training the neural network, the degradation processing of the deconvolved image is the convolution of the deconvolved image with a point spread function of the optical imaging system.
[0046] Optionally, when the optical imaging system is a wide-field microscopic imaging system or a scanning confocal microscopic imaging system or a light sheet illumination microscopic imaging system or a two-photon scanning imaging system, the original fluorescence image data is a three-dimensional noise image volume stack of a biological sample independently collected by the optical imaging system, the training set includes a first noise image volume stack and a second noise image volume stack amplified by the three-dimensional noise image volume stack through axial interval sampling processing, and the denoised image and the deconvolved image are a denoised image volume stack and a deconvolved image volume stack, respectively;
[0047] The axial interval sampling process includes: extracting odd-numbered layers or even-numbered layers of a three-dimensional noise image volume stack independently collected by the optical imaging system as the first noise image volume stack, and correspondingly extracting even-numbered layers or odd-numbered layers as the second noise image volume stack;
[0048] When training the neural network, one of the first noise image volume stack and the second noise image volume stack corresponding to each other is selected as the input image data, and the other of the first noise image volume stack and the second noise image volume stack corresponding to each other is selected as the true value input image data;
[0049] After the neural network is trained, the trained neural network is used to process the three-dimensional noise image stack of the biological sample collected by the optical imaging system or the original fluorescence image data.
[0050] Optionally, when training the neural network, the degradation processing of the deconvolved image is the convolution of the deconvolved image volume stack with the three-dimensional point spread function of the optical imaging system.
[0051] Optionally, in the case where the optical imaging system is a structured light illumination super-resolution microscopy imaging system, the original fluorescence image data is a structured light illumination image of a biological sample collected by the optical imaging system, and the training set is two structured light illumination super-resolution images reconstructed from the structured light illumination image by structured light illumination super-resolution;
[0052] When training the neural network, one of the two structured light illumination super-resolution images is used as the input image data, and the other of the two structured light illumination super-resolution images is used as the true value input image data;
[0053] After the neural network training is completed, the trained neural network is used to process the structured light illumination image of the biological sample collected by the optical imaging system after structured light illumination super-resolution reconstruction.
[0054] Optionally, the two structured light illumination super-resolution images are generated by using two groups of structured light illumination original image sequences independently collected by the optical imaging system from the biological sample and reconstructing them through structured light illumination super-resolution.
[0055] Optionally, after the optical imaging system independently collects a group of structured light illumination original image sequences for the biological sample, each structured light illumination original image or multiple structured light illumination original images in the group of structured light illumination original images are respectively subjected to image re-degradation processing, thereby generating multiple pairs of re-degraded images, and the first re-degraded image in each pair of re-degraded images is extracted to form a first group of structured light illumination re-degraded image sequences, and the second re-degraded image in each pair of re-degraded images is extracted to form a second group of structured light illumination re-degraded image sequences, and the two structured light illumination super-resolution images are generated by structured light illumination super-resolution reconstruction using the first group of structured light illumination re-degraded image sequences and the second group of structured light illumination re-degraded image sequences.
[0056] Optionally, when training the neural network, the degradation processing of the deconvolved image is the convolution of the deconvolved image with a super-resolution point spread function of the optical imaging system.
[0057] Optionally, the model of the neural network includes but is not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.
[0058] The use of a neural network trained using the technical means of this application can significantly improve the optical resolution of microscopic images with high fidelity and quantifiable characteristics. In addition, the technical means of this application does not require specific adjustment of hyperparameters when applied to different biological samples or different signal-to-noise ratio data, thereby greatly expanding the scope of application of computational super-resolution methods. The method and system of this application are particularly suitable for long-term non-destructive imaging of living biological samples with low excitation light power, and can improve the optical resolution of the collected image information while ensuring the activity of the biological samples, so that its high-speed dynamic process can be clearly observed, which is conducive to the research and development of new life phenomena. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] The principles and various aspects of the present application can be more fully understood from the following detailed description in conjunction with the following drawings. It should be noted that the scales of the drawings may be different for the purpose of clear description, but this will not affect the understanding of the present application. In the drawings:
[0060] Figure 1 A basic block diagram of a microscopic imaging system is schematically shown;
[0061] Figure 2A The process of training a neural network of a denoising and super-resolution module of a self-supervised microscopic image super-resolution processing system according to an embodiment of the present application is schematically shown;
[0062] Figure 2B The process of prediction using the denoising and super-resolution modules trained by the neural network is schematically shown;
[0063] Figure 3A The process of training a neural network of a denoising and super-resolution module of a self-supervised microscopic image super-resolution processing system according to another embodiment of the present application is schematically shown;
[0064] Figure 3B The process of prediction using the denoising and super-resolution modules trained by the neural network is schematically shown;
[0065] Figure 4A The process of training a neural network of a denoising and super-resolution module of a self-supervised microscopic image super-resolution processing system according to another embodiment of the present application is schematically shown;
[0066] Figure 4B The process of prediction using the denoising and super-resolution modules trained by the neural network is schematically shown;
[0067] Figure 5A The process of training the neural network of the denoising and super-resolution module of the self-supervised microscopic image super-resolution processing system according to another embodiment of the present application is schematically shown; and
[0068] Figure 5B The process of prediction using the denoising and super-resolution modules trained by the neural network is schematically shown. DETAILED DESCRIPTION
[0069] In the various figures of the present application, features with the same structure or similar functions are represented by the same reference numerals.
[0070] Figure 1 The basic block diagram of a microscopic imaging system is schematically shown, which generally includes an optical imaging system 100 and a control and data processing system 200. The optical imaging system 100 can be, but is not limited to, a wide-field microscopic imaging system, a scanning confocal microscopic imaging system, a light sheet illumination microscopic imaging system, a two-photon scanning imaging system, and a structured light illumination microscopic imaging system. Taking the structured light illumination microscopic imaging system as an example, the optical imaging system 100 includes an excitation light path and a detection light path, wherein the excitation light path includes an excitation objective lens and other optical components for generating excitation light, and the excitation light beam can be emitted through the excitation objective lens so as to excite fluorescence on a biological sample, and the detection light path includes a detection objective lens and other optical components for imaging, which are used to receive and detect the excited fluorescence. It should be clear to those skilled in the art that, depending on the configuration of the microscopic imaging system, the excitation objective lens and the detection objective lens can be the same objective lens or different objective lenses. The optical imaging system 100 can perform two-dimensional or three-dimensional fluorescence microscopic imaging on biological samples, especially living biological samples. When three-dimensional fluorescence microscopy imaging is performed on biological samples, especially living biological samples, multiple layers of fluorescence images are continuously scanned and sampled along the optical axis direction of the detection objective lens, i.e., the axial direction. In this way, each time a scanning sampling is completed, the acquired multiple layers of fluorescence images constitute a fluorescence image stack (also referred to as a "sequence"). It should be pointed out that in the context of this application, according to the requirements of the specific technical solution, the term "image" can be understood as a two-dimensional image or a three-dimensional image stack.
[0071] The control and data processing system 200 mainly includes a computer and related components (such as a data storage device, etc.), which can control the operation of the optical imaging system 100 and can receive image data from the optical imaging system 100 and perform corresponding post-processing. For example, the acquired fluorescent image volume stack is provided to the control and data processing system 200, and reconstructed into a high signal-to-noise ratio, super-resolution microscopic image after a series of data processing. For this purpose, the control and data processing system 200 may include a microscopic image super-resolution processing module or system or a self-supervised microscopic image super-resolution processing module or system. The microscopic image super-resolution processing module or system or the self-supervised microscopic image super-resolution processing module or system mainly includes a microscopic image preprocessing module 210 and a denoising and super-resolution reconstruction module 220. It should be pointed out that within the scope of the present application, the modules and / or submodules described here can be understood to include a data storage device, such as a computer-readable storage medium, in which programs or subroutines and denoising neural network models called and run by a computer, especially a computer of the control and data processing system 200, can be stored. These programs or subroutines and denoising neural network models can implement the methods / steps described below, especially the self-supervised microscopic image super-resolution processing methods / steps, when called and executed by a computer. The specific programming methods for the programs and / or subroutines are not discussed in this application, and those skilled in the art can implement the relevant functions with any well-known programming software and / or commercial software. Therefore, when describing the operation of the relevant system or the operation or method of the module below, it should be understood that they can also be written as programs to be called and executed by a computer.
[0072] The microscopic image preprocessing module 210 is configured to preprocess the fluorescence image acquired by the optical imaging system 100 to obtain raw data for neural network training and prediction in the denoising and super-resolution processing module 220. The processing methods include but are not limited to image re-degradation, axial interval sampling, structured light super-resolution reconstruction, etc. For specific algorithms of image re-degradation, axial interval sampling, and structured light super-resolution reconstruction, please refer to the references: such as Pang, T., Zheng, H., Quan, Y. & Ji, H. in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition 2043-2052 (2021) (image re-degradation); Qiao, C., Li, D., Liu, Y. et al. Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes. Nature Biotechnology, 41, 367–377 (2023) (axial interval sampling); Gustafsson, M. Get al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys J 94, 4957-4970 (2008) (structured light super-resolution reconstruction).
[0073] It should be noted that when the image is subjected to image re-degradation or axial interval sampling in the microscopic image preprocessing module 210, the image (or image data) targeted is two noise-independent data pairs that can be used for training obtained from a single image or a single three-dimensional image stack directly obtained from the optical imaging system 100, rather than paired training data directly obtained from the optical imaging system 100 (e.g., high / low signal-to-noise ratio data pairs, or low / low signal-to-noise ratio data pairs independently sampled from the same scene). In addition, it should be noted that when the image is subjected to structured light super-resolution reconstruction in the microscopic image preprocessing module 210, the image (or image data) targeted is the original fluorescence image data obtained by the optical imaging system 100 in a structured light illumination manner, rather than the original fluorescence image data obtained by the optical imaging system 100 in other imaging modalities (e.g., wide-field microscopy, scanning confocal microscopy, two-photon microscopy, light sheet microscopy, etc.), which do not require structured light super-resolution reconstruction.
[0074] The denoising and super-resolution module 220 can be implemented by using any neural network architecture as a basic model. For example, the neural network model used in the denoising and super-resolution module 220 includes but is not limited to a U-type neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model. When training the neural network of the denoising and super-resolution module 220, the relevant network model is optimized using a loss function, and the loss function includes but is not limited to mean square error (MSE), mean absolute error (MAE), structural similarity (SSIM) or their weighted sum, etc.
[0075] Therefore, the control and data processing system 200 may also be referred to as a microscopic image super-resolution processing system, which includes a microscopic image pre-processing module 210 and a denoising and super-resolution module 220 . Figure 2A The process of training the neural network of the denoising and super-resolution module 220 of the microscopic image super-resolution processing system using the original fluorescence image obtained by the optical imaging system 100 according to one embodiment of the present application is schematically shown; and Figure 2B The schematic diagram shows a process of performing denoising and super-resolution processing on the original fluorescence image obtained by the optical imaging system 100 using the microscopic image super-resolution processing system 200 with a trained neural network according to an embodiment of the present application.
[0076] against Figure 2A , a wide-field microscopic imaging system will be introduced as an example of the optical imaging system 100. Those skilled in the art should be aware that other imaging system examples described in the present application are also applicable. The optical imaging system 100 first completes two independent fluorescence image scans on the same biological sample to sample and obtain noise image A and noise image B, which can also be called raw fluorescence image data. Since the two samples are independent of each other, the noise in noise image A and noise image B is also independent of each other. If the signal-to-noise ratio of noise image A and noise image B is the same as each other, the neural network training mode described below corresponds to the noise-noise training mode; if the signal-to-noise ratio of noise image A is higher than the signal-to-noise ratio of noise image B, the neural network training mode described below corresponds to the supervised training mode.
[0077] According to an embodiment of the present application, the optical imaging system 100 can be used to complete multiple sampling of the same biological sample at different times, thereby obtaining multiple groups of noise image A / B pairs distributed over time, so as to form a training set for training the neural network of the denoising and super-resolution module 220. In the training set, each group of noise image A / B pairs is divided by time, and the noise image A and the noise image B in each group of noise image A / B pairs correspond to each other one by one. In the scope of the present application, the term "one-to-one correspondence" means that when training a neural network, if one noise image A or B in a group of noise image A / B pairs is selected as a training input, the other noise image B in the same group of noise image A / B pairs is used as a training true value.
[0078] In an alternative embodiment of the present application, the noise image A / B pair can also be obtained from an independent noise image through image re-degradation, axial interval sampling (applied to three-dimensional data) and other data augmentation methods, as long as the noise of the noise image A and the noise image B in the pair are independently distributed. In another alternative embodiment of the present application, if the optical imaging system 100 is a structured light microscopy imaging system, then Figure 2A and 2B The noise image A and the noise image B shown in the figure refer to the structured light super-resolution images after the collected original microscopic fluorescence images are reconstructed by structured light super-resolution.
[0079] Return to Figure 2A , randomly extract multiple groups of noise images A / B from the training set to train the neural network of the denoising and super-resolution module 220. For example, in each group of noise image A / B pairs extracted, the noise image A is used as the input image data for training, and the noise image B is used as the true value image data for training. In each training cycle, for the selected one-to-one corresponding noise image A and noise image B, randomly extract pixel blocks at the same image position, randomly rotate and flip them, and then use them as the training input image and training target image of the neural network respectively.
[0080] The neural network of the denoising and super-resolution module 220 includes two parts, namely, a denoising module part and a deconvolution module part. The denoising module part is used to denoise the input noisy image A and output the denoised image A. q The deconvolution module is used to denoise the image A. q Perform deconvolution processing and output the deconvolution image A j Correspondingly, in this training process, the loss function of the neural network can include denoising loss and deconvolution loss, where the denoising loss is expressed as the denoised image A q The difference between the noisy image B and the deconvolution loss is shown as the degraded deconvolution image A. jThe difference between the deconvolution image A and the noise image B is particularly significant for this embodiment. j It should be degraded first, that is, it is convolved with the point spread function of the optical imaging system 100 and then the difference between the convolved result and the noise image B is determined. The difference is calculated in a manner including but not limited to mean square error (MSE), mean absolute error (MAE), structural similarity (SSIM) or a weighted sum thereof.
[0081] In an alternative embodiment of the present application, considering the coordination between various factors such as computing processing time and result accuracy, the denoising loss calculation content can also be omitted. That is to say, when training the neural network, the loss function can also retain only the deconvolution loss, which can also realize the training of the neural network of the denoising and super-resolution module 220. In this mode, although only the deconvolution image can be output, the network efficiency and operation speed can be improved by reducing the scale of the neural network parameters.
[0082] In each iteration of network training, the gradient is calculated according to the loss function, and its gradient is back-propagated to update the network parameters. After the network input error converges, the training is stopped and the network parameters are stored. It can be seen that when the above-mentioned neural network training method of the present application is adopted, a high signal-to-noise ratio and high-resolution true value image is not required, so it can also be considered as a self-supervised training.
[0083] After the neural network training of the denoising and super-resolution module 220 is completed, the microscopic image super-resolution processing system can perform denoising and super-resolution reconstruction (or prediction) on the original fluorescence image sequence obtained by the optical imaging system 100, such as Figure 2B As shown. The noisy image is used as the input of the neural network of the denoising and super-resolution module 220, so as to obtain the final denoised and super-resolution image. Of course, if denoising and super-resolution reconstruction are performed on multiple original fluorescence images, the processing process can be repeated for each original fluorescence image.
[0084] Figure 3A and 3BA microscopic image super-resolution processing method according to an embodiment of the present application is schematically shown. In the illustrated embodiment, the optical imaging system 100 is a two-dimensional wide-field microscope, and the training set for training the neural network of the denoising and super-resolution module 220 consists of a plurality of two-dimensional noise images (two-dimensional images containing noise). In the illustrated embodiment, before training the neural network of the denoising and super-resolution module 220, the microscopic image preprocessing module 210 can be used to perform image re-degradation processing on each or any other number of two-dimensional noise images in the plurality of two-dimensional noise images (or referred to as original fluorescence image data) collected by the optical imaging system 100 for biological samples, so as to generate a plurality of groups of training noise image A / B pairs as training sets, wherein each group of noise image A / B pairs includes a noise image A and a noise image B corresponding thereto one by one. An example of image re-degradation processing is as follows.
[0085] Assuming that there is a two-dimensional noise image y, a computer can be used to randomly generate a normal distribution random variable z with the same image dimension (or image pixel size) as the two-dimensional noise image y, with a mean of 0 and a variance of 1. Then, based on each original fluorescence image y, a pair of fluorescence perturbation images (or re-degraded images) y are generated using the following formula: A and B .
[0086]
[0087] y A =y+α·Σ·z (2)
[0088] y B =y-α -1 ·Σ·z (3)
[0089] In the above formulas (1), (2), and (3), α, β1, and β2 are constants randomly generated by a computer within the range specified in Table 1 below. In Table 1, σ 2 is the background noise variance of the camera used by the selected optical imaging system 100. The value ranges of α, β1, and β2 in Table 1 are only the values in this example, and their value ranges can also be adjusted as needed in other application scenarios.
[0090] parameter α <![CDATA[β1]]> <![CDATA[β2]]> scope 0.2~5.0 0.2~5 <![CDATA[0.5σ 2 ~1.5σ 2- ]]>
[0091] Table 1 Parameter ranges used in “Image perturbation”
[0092] In the context of this application, the term "image y" or "image y A or B"Mathematically, it can be understood as a two-dimensional matrix that can reflect the image that the human eye can observe. Therefore, the multiplication operation related to the image or matrix in the above formula should be understood as a dot multiplication operation.
[0093] For the selected original fluorescence image obtained by using the optical imaging system 100, the formulas (1) to (3) are repeatedly used to perform image re-degradation processing on each original noisy image. In this way, all the re-degraded images y A Construct the input image dataset for training, and generate all the degraded images y B Constitute a dataset of true value images for training; as an alternative, it can also be all the generated degraded images y A Construct the real value image dataset for training, and generate all the degraded images y B It should be noted that after each image re-degradation process, a re-degraded image y is generated. A With a re-degraded image y B It forms a one-to-one correspondence, that is, when the degraded image y A When it is selected as the input image data during neural network training, only the degraded image y B Select as the true value image data.
[0094] Based on the above input image dataset and the true value image dataset, the neural network of the denoising and super-resolution module 220 is trained. Figure 3A As shown, in each training cycle, a one-to-one corresponding re-degraded image y is randomly taken from the input dataset and the training dataset. A and B As input image data and true value image data, the pixel blocks at the same position in them are randomly extracted, and after random rotation and flipping, they are used as input images and target images for training of the neural network. In the embodiment shown, the basic neural network architecture can select a U-shaped neural network, that is, the denoising module part and the deconvolution module part each select a U-shaped neural network. The denoising module part outputs the denoised image A q , and calculate the denoising loss with the noisy image B. The deconvolution module outputs the deconvolved image A j It should be pointed out that for the “image re-degradation” method to generate the training input image dataset and the training true value image dataset, before calculating the deconvolution loss, the deconvolution image A must be j After degradation processing, the deconvolution image A is first jConvolve with the point spread function of the optical imaging system 100, and then calculate the deconvolution loss with the convolution result and the noise image B. In each training iteration, the training of the neural network is completed by loss calculation (for example, using both denoising loss and deconvolution loss or only using deconvolution loss) and back propagation. When the "image re-degradation" method is used to train the neural network of the denoising and super-resolution module 220, since the network training can be completed by only using the original fluorescent image sequence acquired once, it is avoided that the living cells are damaged by multiple samplings, and since multiple acquisitions are not required, it can also be used to process the data recorded by video observation of living cells.
[0095] After the neural network of the denoising and super-resolution module 220 is trained, the noise images that are not selected for training or the resampled noise images (noise images not in the training set) in the training set of the neural network of the denoising and super-resolution module 220 can be used for denoising and super-resolution processing (e.g. Figure 3B shown).
[0096] Figure 4A and 4B A method for super-resolution processing of microscopic images according to another embodiment of the present application is schematically shown. In this embodiment, the optical imaging system 100 is a light sheet illumination microscopic imaging system. Before training the neural network of the denoising and super-resolution module 220, the optical imaging system 100 is used to collect multiple three-dimensional noise image stacks (three-dimensional image stacks containing noise, or referred to as raw fluorescence image data). Figure 4A As shown, a three-dimensional noise image stack (ie, the original noise image stack) is processed by axial interval sampling in the microscopic image preprocessing module 210 and is amplified into a noise image stack A. v (odd-numbered layers of the original noise image volume stack or even-numbered layers of the original noise image volume stack) and noise image volume stack B v (even-numbered layers of the original noise image stack or odd-numbered layers of the original noise image stack). Next, in the neural network training of the denoising and super-resolution module 220, the noise image stack A v With the noise image volume stack B v The denoising loss and the deconvolution loss or only the deconvolution loss are calculated as the input image data and the true value image data respectively. In the embodiment shown, the neural network model adopts a three-dimensional residual channel attention neural network architecture, and the denoising module part and the deconvolution module part respectively output the denoised image volume stack A vq With the deconvolved image volume stack A vd For example, a denoised image stack A can be used vq With the noise image volume stack B v To calculate the denoising loss, before calculating the deconvolution loss, the denoised image stack A must bevq After degradation processing, the denoised image stack A vq The result is convolved with the three-dimensional point spread function of the optical imaging system 100 and then convolved with the noise image volume stack B v The error is calculated to obtain the deconvolution loss. In each iteration of training, the gradient is calculated using the denoising loss and the deconvolution loss (or only the deconvolution loss), and the neural network weights are updated based on back propagation to complete the training process.
[0097] like Figure 4B As shown, after the neural network of the denoising and super-resolution module 220 is trained, the original noisy image volume stack is used as input, and the trained neural network of the denoising and super-resolution module 220 can simultaneously output the corresponding denoised image volume stack and deconvolution image volume stack, thereby achieving signal-to-noise ratio and resolution enhancement compared with the original data.
[0098] Figure 5A and 5B A microscopic image super-resolution processing method according to another embodiment of the present application is schematically shown. In this embodiment, the optical imaging system 100 is a structured light illumination super-resolution microscopic imaging system, and the microscopic image preprocessing module 210 is configured to perform structured light super-resolution reconstruction on the original structured light illumination image collected by the optical imaging system 100. Figure 5A In the training process shown, the "noise-noise" training mode is used to train the neural network of the denoising and super-resolution module 220. Specifically, before preparing data for training the neural network of the denoising and super-resolution module 220, two sets of structured light illumination original image sequences y are collected independently for each structural region of the biological sample or the biological sample. A,i and B,i , where i is an integer greater than 1. Then, the microscopic image preprocessing module 210 is used to transform the structured light illumination original image sequence y A,i and B,i Reconstructed into noisy structured light illumination super-resolution images Y A and Y B Then, when training the neural network of the denoising and super-resolution module 220, the noisy structured light illumination super-resolution image (or noisy SIM image) Y A and Y B The network loss (e.g., denoising loss and deconvolution loss or just deconvolution loss) is calculated as the input image data and true value image data of the neural network training, thereby completing the gradient calculation and network weight update. In this embodiment, the neural network model selection and loss function calculation are all the same as Figure 3A and 3BIn each training cycle, the super-resolution image Y A and Y B Pixel blocks at the same position are randomly extracted from them, and after random rotation and flipping, they are used as the training input image and training target image of the neural network. In the embodiment shown, the basic neural network architecture can select a U-shaped neural network, that is, the denoising module part and the deconvolution module part each select a U-shaped neural network. The denoising module part outputs the denoised SIM image Y Aq and the noisy SIM image Y B Calculate the denoising loss. The deconvolution module outputs the deconvolved SIM image Y Aj Before calculating the deconvolution loss, the deconvolved SIM image Y Aj When performing the degradation processing, the point spread function used is not the point spread function of the optical imaging system 100, but a super-resolution point spread function (i.e., the full width at half maximum is about half of the wide-field point spread function, rather than the point spread function under wide-field illumination). Then, the deconvolved SIM image Y after the degradation processing is Aj SIM image Y with noise B Compute the deconvolution loss.
[0099] like Figure 5B As shown, after the neural network training of the denoising and super-resolution module 220 is completed, for the noisy structured light illumination original image that needs to be denoised and super-resolution processed, the microscopic image preprocessing module 210 is first used to reconstruct it into a noisy structured light illumination super-resolution image, and then the reconstructed noisy structured light illumination super-resolution image is input into the denoising and super-resolution module 220 whose neural network has been trained, thereby completing the denoising and resolution enhancement processing of the structured light illumination super-resolution image.
[0100] In a Figure 5A In an alternative embodiment of the illustrated embodiment, before preparing data for training the neural network of the denoising and super-resolution module 220, only one set of structured light illumination original image sequences may be collected for each structural region of the biological sample or the biological sample, and then, using formulas (1), (2), and (3), an image re-degradation processing method is adopted so that each structured light illumination original image or multiple structured light illumination original images in the set of structured light illumination original image sequences respectively generates a pair of re-degraded images y A and B , thereby generating multiple pairs of re-degraded images, and the re-degraded image y in each pair of re-degraded images A Extracted to form a set of structured light illumination and degraded image sequence y A,i And the re-degraded image y in each pair of re-degraded images BExtracted to form a set of structured light illumination and degraded image sequence y B,i , where i is an integer greater than 1. Then, the microscopic image preprocessing module 210 is used to transform the structured light illumination original image sequence y A,i and B,i Reconstructed into noisy structured light illumination super-resolution images Y A and Y B As training images. The neural network training content of the denoising and super-resolution module 220 of this alternative embodiment can refer to Figure 5A After the neural network training of the denoising and super-resolution module 220 is completed, the original image with noisy structured light illumination that needs to be denoised and super-resolution processed can refer to the following Figure 5B The content shown.
[0101] It can be seen from the above description of this application that Figure 3A , 3B as well as Figure 4A , 4B In the embodiment shown, during the neural network training process, the original fluorescence image data must first be processed by the microscopic image preprocessing module 210 before being used for the neural network training. In the prediction process, only the original fluorescence image data needs to be input into the denoising and super-resolution reconstruction module 220 after the neural network training to obtain the final image processed by denoising and super-resolution. Figure 5A , 5B In the embodiment shown, during the neural network training process, the original fluorescence image data must also first be processed by the microscopic image preprocessing module 210 before being used for the neural network training. However, during the prediction process, the original fluorescence image data still needs to be processed by the microscopic image preprocessing module 210 and then input into the denoising and super-resolution reconstruction module 220 to obtain the final image after denoising and super-resolution processing.
[0102] It can be seen from the description of the above embodiments that the present application proposes a self-supervised microscopic image super-resolution processing method, comprising:
[0103] Collecting raw fluorescence image data of a biological sample using the optical imaging system 100;
[0104] Depending on the type of the optical imaging system 100, the collected raw fluorescence image data is preprocessed by a computer to obtain a training set, and the training set is used in the computer to train a neural network for image denoising and super-resolution processing, wherein the neural network includes a denoising part and a deconvolution part, wherein the training set includes input image data and true value input image data, pixel blocks at the same position are randomly extracted from the input image data and the true value input image data, and are randomly rotated and flipped to serve as a training input image and a training target image respectively, the training input image is first processed by the denoising part to become a denoised image, and then processed by the deconvolution part to become a deconvolution image, the denoising image and the training target image are used to calculate the denoising loss together, the deconvolution image is subjected to a degradation process depending on the type of the optical imaging system 100, and the deconvolution loss is calculated together with the training target image, and in the training process of the neural network, the loss function includes the denoising loss and the deconvolution loss or only includes the deconvolution loss;
[0105] The trained neural network is used to process the original fluorescent image data or the additional fluorescent image data collected from the same biological sample by the optical imaging system 100 .
[0106] In addition, it can also be considered that the present application proposes a self-supervised microscopic image super-resolution processing system, which is configured to process raw fluorescence image data of a biological sample collected by the optical imaging system 100, including:
[0107] Microscopic image preprocessing module 210; and
[0108] A denoising and super-resolution reconstruction module 220, wherein the microscopic image preprocessing module 210 is configured to perform image preprocessing on the collected raw fluorescence image data using a computer to obtain a training set depending on the type of the optical imaging system 100, and the denoising and super-resolution module 220 is configured with a neural network to perform denoising and super-resolution processing on the image, wherein the neural network includes a denoising part and a deconvolution part;
[0109] The neural network is trained in a computer using the training set, wherein the training set includes input image data and true value input image data, pixel blocks at the same position are randomly extracted from the input image data and the true value input image data, and are randomly rotated and flipped to serve as a training input image and a training target image respectively, the training input image is first processed by the denoising part to be a denoised image, and then processed by the deconvolution part to be a deconvolution image, the denoising loss is calculated together with the training target image, the deconvolution loss is calculated together with the training target image after the deconvolution image is subjected to degradation processing depending on the type of the optical imaging system 100, and in the training process of the neural network, the loss function includes the denoising loss and the deconvolution loss or only includes the deconvolution loss;
[0110] The denoising and super-resolution reconstruction module 220 is configured to process the original fluorescence image data using the trained neural network or the additional fluorescence image data collected from the same biological sample using the optical imaging system 100 .
[0111] The main advantages of the technical solution of this application are as follows:
[0112] (1) The technical solution of the present application can complete the training of the super-resolution neural network model without high and low signal-to-noise ratio data pairs.
[0113] (2) This application is based on two microscopic images (volume stacks) with independent noise as the input and true value of neural network training, thereby improving the image resolution while denoising the image, and has extremely strong noise robustness.
[0114] (3) The technical solution of the present application can be applied to a variety of different microscopic imaging systems to achieve multimodal denoising and super-resolution imaging.
[0115] (4) Based on data augmentation methods such as image re-degradation and axial interval sampling, the technical solution of the present application can be directly applied to long-term video data, directly construct a training set from the data itself, and perform denoising and super-resolution processing on it, which is very convenient to use.
[0116] The diffraction-limited resolution of a traditional wide-field illuminated optical microscope is about 200 nanometers, while the resolution of a microscopic image processed by the self-supervised denoising and super-resolution scheme of the present application can be increased to about 120 nanometers. In other words, the optical resolution of the microscopic image can be increased by more than 1.5 times. In particular, for super-resolution microscopic imaging technologies such as structured light illumination microscopy, the technology of the present application can continue to improve the resolution based on its resolution. For example, the resolution of structured light illumination super-resolution microscopic imaging technology is about 100 nanometers, and the technology of the present application can further improve its resolution to about 60 nanometers.
[0117] In summary, this application designs a new self-supervised denoising and super-resolution processing scheme, which improves the optical resolution of microscopic images by more than 1.5 times in the absence of high signal-to-noise ratio and high-resolution data. The output results have high fidelity and quantifiable characteristics, and there is no need to specifically adjust the hyperparameters when applied to different biological samples or different signal-to-noise ratio data, thereby greatly expanding the application scope of computational super-resolution methods.
[0118] Although the specific embodiments of the present application are described in detail herein, they are provided only for the purpose of explanation and should not be considered to limit the scope of the present application. In addition, it should be clear to those skilled in the art that the various embodiments described in this specification can be used in combination with each other. Various replacements, changes and modifications can be conceived without departing from the spirit and scope of the present application.
Claims
1. A self-supervised microscopic image super-resolution processing method, comprising: Using an optical imaging system (100) to collect raw fluorescence image data of a biological sample; Depending on the type of the optical imaging system (100), the collected raw fluorescence image data is subjected to image preprocessing by using a computer to obtain a training set, and the training set is used in the computer to train a neural network for image denoising and super-resolution processing, wherein the neural network includes a denoising part and a deconvolution part, wherein the training set includes input image data and true value input image data, pixel blocks at the same position are randomly extracted from the input image data and the true value input image data, and are randomly rotated and flipped to serve as a training input image and a training target image respectively, the training input image is first processed by the denoising part to become a denoised image, and then processed by the deconvolution part to become a deconvolution image, the denoising image and the training target image are used to calculate the denoising loss together, the deconvolution image is subjected to a degradation process depending on the type of the optical imaging system (100), and the deconvolution loss is calculated together with the training target image, and during the training of the neural network, the loss function includes the denoising loss and the deconvolution loss or only includes the deconvolution loss; The trained neural network is used to process the original fluorescent image data or the additional fluorescent image data collected from the same biological sample using the optical imaging system (100).
2. The self-supervised microscopic image super-resolution processing method according to claim 1, characterized in that: In the case where the optical imaging system (100) is a wide-field microscopic imaging system, a scanning confocal microscopic imaging system, a light sheet illumination microscopic imaging system, or a two-photon scanning imaging system, the original fluorescence image data is one or more two-dimensional noise images of a biological sample independently collected by the optical imaging system (100), and the training set includes a plurality of sets of training noise image (A / B) pairs, wherein the plurality of sets of training noise images (A / B) are generated by performing image re-degradation processing on the original fluorescence image data; The image re-degradation processing comprises: For a two-dimensional noise image, a computer is used to randomly generate a normal distribution random variable with the same image dimension as the two-dimensional noise image, with a mean of 0 and a variance of 1, and a pair of fluorescence perturbation images are generated using the following formula: and A =y+α·∑·z(2) and B =y-α -1 ·∑·z (3) In the above formulas (1), (2), and (3), α, β1, and β2 are constants randomly generated by a computer, and α = 0.2 to 5.0, β1 = 0.2 to 5, and β2 = 0.5σ 2 ~1.5σ 2 , y represents the two-dimensional noise image, z represents the normal distribution random variable, y A represents one of the pair of fluorescence perturbation images, y B represents the other fluorescence perturbation image in the pair of fluorescence perturbation images, σ 2 is the background noise variance of the camera used in the optical imaging system (100), and the pair of fluorescence perturbation images constitutes a set of training noise image (A / B) pairs; When training the neural network, one noise image in a pair of training noise images (A / B) is selected from the plurality of pairs of training noise images (A / B) as input image data, and another noise image in the selected pair of training noise images (A / B) is used as true value input image data; After the neural network is trained, the trained neural network is used to process the two-dimensional noise image of the biological sample collected by the optical imaging system (100) or to process the original fluorescence image data.
3. The self-supervised microscopic image super-resolution processing method according to claim 2, characterized in that: When training the neural network, the degradation process of the deconvolved image is the convolution of the deconvolved image with the point spread function of the optical imaging system (100).
4. The self-supervised microscopic image super-resolution processing method according to claim 1, characterized in that: In the case where the optical imaging system (100) is a wide-field microscopic imaging system, a scanning confocal microscopic imaging system, a light sheet illumination microscopic imaging system, or a two-photon scanning imaging system, the original fluorescence image data is a three-dimensional noise image stack of a biological sample independently collected by the optical imaging system (100), and the training set includes a first noise image stack (A) amplified by the three-dimensional noise image stack through axial interval sampling processing. v ) and the second noise image stack (B v ), and the denoised image and the deconvolved image are respectively a denoised image volume stack and a deconvolved image volume stack; The axial interval sampling process comprises: extracting an odd-numbered layer or an even-numbered layer of a three-dimensional noise image volume stack independently collected by the optical imaging system (100) as the first noise image volume stack (A v ), and correspondingly extract its even-numbered layers or odd-numbered layers as the second noise image volume stack (B v ); When training the neural network, select the first noise image stacks (A) corresponding to each other one by one v ) and the second noise image stack (B v ) as the input image data, and select the first noise image volume stack (A) corresponding to each other one by one v ) and the second noise image stack (B v ) as the true value input image data; After the neural network is trained, the trained neural network is used to process the three-dimensional noise image stack of the biological sample collected by the optical imaging system (100) or the original fluorescence image data.
5. The self-supervised microscopic image super-resolution processing method according to claim 3, characterized in that: When training the neural network, the degradation process of the deconvolved image is the convolution of the deconvolved image volume stack with the three-dimensional point spread function of the optical imaging system (100).
6. The self-supervised microscopic image super-resolution processing method according to claim 1, characterized in that: In the case where the optical imaging system (100) is a structured light illumination super-resolution microscopic imaging system, the original fluorescence image data is a structured light illumination image of a biological sample collected by the optical imaging system (100), and the training set is two structured light illumination super-resolution images (Y A and Y B ); When training the neural network, one of the two structured light illumination super-resolution images is used as the input image data, and the other of the two structured light illumination super-resolution images is used as the true value input image data; After the neural network is trained, the trained neural network is used to process the structured light illumination image of the biological sample collected by the optical imaging system (100) after structured light illumination super-resolution reconstruction.
7. The self-supervised microscopic image super-resolution processing method according to claim 6, characterized in that: The two structured light illumination super-resolution images (Y A and Y B ) using two sets of structured light illumination original image sequences (y) independently collected by the optical imaging system (100) for the biological sample A,i and B,i ) is produced by super-resolution reconstruction of structured light illumination.
8. The self-supervised microscopic image super-resolution processing method according to claim 6, characterized in that: A set of structured light illumination original image sequences (y) independently collected from the biological sample by the optical imaging system (100) A ,i and y B ,i), each structured light illumination original image or multiple structured light illumination original images in the group of structured light illumination original images are respectively subjected to image re-degradation processing, thereby generating multiple pairs of re-degraded images, and the first re-degraded image (yA) in each pair of re-degraded images is extracted to form a first group of structured light illumination re-degraded image sequences (y A ,i) and the second re-degraded image (y B ) is extracted to form the second set of structured light illumination and degraded image sequence (y B,i ), the two structured light illumination super-resolution images (Y A and Y B ) Using the first set of structured light illumination to re-degrade the image sequence (y A,i ) and the second set of structured light illumination and degraded image sequence (y B ,i) Produced by super-resolution reconstruction using structured light illumination.
9. The self-supervised microscopic image super-resolution processing method according to claim 6, 7 or 8, characterized in that: When training the neural network, the degradation process of the deconvolved image is the convolution of the deconvolved image with the super-resolution point spread function of the optical imaging system (100).
10. The self-supervised microscopic image super-resolution processing method according to any one of claims 1 to 8, characterized in that: The model of the neural network for image denoising and super-resolution processing includes but is not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.
11. The self-supervised microscopic image super-resolution processing method according to claim 9, characterized in that: The model of the neural network for image denoising and super-resolution processing includes but is not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.
12. A self-supervised microscopic image super-resolution processing system, configured to process raw fluorescence image data of a biological sample collected using an optical imaging system (100), comprising: Microscopic image preprocessing module (210); as well as A denoising and super-resolution reconstruction module (220), wherein the microscopic image preprocessing module (210) is configured to perform image preprocessing on the collected raw fluorescence image data using a computer to obtain a training set depending on the type of the optical imaging system (100), and the denoising and super-resolution module (220) is configured with a neural network to perform denoising and super-resolution processing on the image, wherein the neural network includes a denoising part and a deconvolution part, The neural network is trained in a computer using the training set, wherein the training set includes input image data and true value input image data, pixel blocks at the same position are randomly extracted from the input image data and the true value input image data, and are randomly rotated and flipped to serve as a training input image and a training target image respectively, the training input image is first processed by the denoising part to become a denoised image, and then processed by the deconvolution part to become a deconvolution image, the denoising loss is calculated together with the training target image, the deconvolution loss is calculated together with the training target image after the deconvolution image is subjected to degradation processing depending on the type of the optical imaging system (100), and in the training process of the neural network, the loss function includes the denoising loss and the deconvolution loss or only includes the deconvolution loss; The denoising and super-resolution reconstruction module (220) is configured to process the original fluorescence image data using the trained neural network or the additional fluorescence image data collected from the same biological sample using the optical imaging system (100).
13. The self-supervised microscopic image super-resolution processing system according to claim 12, characterized in that: In the case where the optical imaging system (100) is a wide-field microscopic imaging system, a scanning confocal microscopic imaging system, a light sheet illumination microscopic imaging system, or a two-photon scanning imaging system, the original fluorescence image data is one or more two-dimensional noise images of a biological sample independently collected by the optical imaging system (100), and the training set includes a plurality of sets of training noise image (A / B) pairs, wherein the plurality of sets of training noise images (A / B) are generated by performing image re-degradation processing on the original fluorescence image data; The image re-degradation processing comprises: For a two-dimensional noise image, a computer is used to randomly generate a normal distribution random variable with the same image dimension as the two-dimensional noise image, with a mean of 0 and a variance of 1, and a pair of fluorescence perturbation images are generated using the following formula: and A =y+α·∑·z (2) and B =y-α -1 ·∑·z (3) In the above formulas (1), (2), and (3), α, β1, and β2 are constants randomly generated by computers, α = 0.2 to 5.0, β1 = 0.2 to 5, and β2 = 0.5σ. 2 ~1.5σ 2 , σ 2 is the background noise variance of the camera used in the optical imaging system (100), y represents the two-dimensional noise image, z represents the normal distribution random variable, y A represents one of the pair of fluorescence perturbation images, y B represents another fluorescence perturbation image in the pair of fluorescence perturbation images, and the pair of fluorescence perturbation images constitute a set of training noise image pairs (A / B); When training the neural network, one noise image in a pair of training noise images (A / B) is selected from the plurality of pairs of training noise images (A / B) as input image data, and another noise image in the selected pair of training noise images (A / B) is used as true value input image data; After the neural network is trained, the trained neural network is used to process the two-dimensional noise image of the biological sample collected by the optical imaging system (100) or to process the original fluorescence image data.
14. The self-supervised microscopic image super-resolution processing system according to claim 13, characterized in that: When training the neural network, the degradation process of the deconvolved image is the convolution of the deconvolved image with the point spread function of the optical imaging system (100).
15. The self-supervised microscopic image super-resolution processing system according to claim 12, characterized in that: In the case where the optical imaging system (100) is a wide-field microscopic imaging system, a scanning confocal microscopic imaging system, a light sheet illumination microscopic imaging system, or a two-photon scanning imaging system, the original fluorescence image data is a three-dimensional noise image stack of a biological sample independently collected by the optical imaging system (100), and the training set includes a first noise image stack (A) amplified by the three-dimensional noise image stack through axial interval sampling processing. v ) and the second noise image stack (B v ), and the denoised image and the deconvolved image are respectively a denoised image volume stack and a deconvolved image volume stack; The axial interval sampling process comprises: extracting an odd-numbered layer or an even-numbered layer of a three-dimensional noise image volume stack independently collected by the optical imaging system (100) as the first noise image volume stack (A v ), and correspondingly extract its even-numbered layers or odd-numbered layers as the second noise image volume stack (B v ); When training the neural network, select the first noise image stacks (A) corresponding to each other one by one v ) and the second noise image stack (B v ) as the input image data, and select the first noise image volume stack (A) corresponding to each other one by one v ) and the second noise image stack (B v ) as the true value input image data; After the neural network is trained, the trained neural network is used to process the three-dimensional noise image stack of the biological sample collected by the optical imaging system (100) or the original fluorescence image data.
16. The self-supervised microscopic image super-resolution processing system according to claim 14, characterized in that: When training the neural network, the degradation process of the deconvolved image is the convolution of the deconvolved image volume stack with the three-dimensional point spread function of the optical imaging system (100).
17. The self-supervised microscopic image super-resolution processing system according to claim 12, characterized in that: In the case where the optical imaging system (100) is a structured light illumination super-resolution microscopic imaging system, the original fluorescence image data is a structured light illumination image of a biological sample collected by the optical imaging system (100), and the training set is two structured light illumination super-resolution images (Y A and Y B ); When training the neural network, one of the two structured light illumination super-resolution images is used as the input image data, and the other of the two structured light illumination super-resolution images is used as the true value input image data; After the neural network is trained, the trained neural network is used to process the structured light illumination image of the biological sample collected by the optical imaging system (100) after structured light illumination super-resolution reconstruction.
18. The self-supervised microscopic image super-resolution processing system according to claim 17, characterized in that: The two structured light illumination super-resolution images (Y A and Y B ) using two sets of structured light illumination original image sequences (y) independently collected by the optical imaging system (100) for the biological sample A ,i and y B ,i) Produced by super-resolution reconstruction using structured light illumination.
19. The self-supervised microscopic image super-resolution processing system according to claim 17, characterized in that: A set of structured light illumination original image sequences (y) independently collected from the biological sample by the optical imaging system (100) A,i and B,i ), each structured light illumination original image or multiple structured light illumination original images in the group of structured light illumination original images are respectively subjected to image re-degradation processing, thereby generating multiple pairs of re-degraded images, and the first re-degraded image (y A ) is extracted to form the first set of structured light illumination and degraded image sequence (y A,i ) and the second re-degraded image (y B ) is extracted to form the second set of structured light illumination and degraded image sequence (y B,i ), the two structured light illumination super-resolution images (Y A and Y B ) Using the first set of structured light illumination to re-degrade the image sequence (y A,i ) and the second set of structured light illumination and degraded image sequence (y B,i ) is produced by super-resolution reconstruction of structured light illumination.
20. The self-supervised microscopic image super-resolution processing system according to claim 17, 18 or 19, characterized in that: When training the neural network, the degradation process of the deconvolved image is the convolution of the deconvolved image with the super-resolution point spread function of the optical imaging system (100).
21. The self-supervised microscopic image super-resolution processing system according to any one of claims 12 to 19, characterized in that: The model of the neural network includes but is not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.
22. The self-supervised microscopic image super-resolution processing system according to claim 20, characterized in that: The model of the neural network includes but is not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.
