Self-supervised microscopy image super-resolution processing method and system

Abstract

This application discloses a self-supervised microscopy image super-resolution processing method. The self-supervised microscopy image super-resolution processing method includes the steps of: collecting original fluorescence image data of a biological sample using an optical imaging system (100);, depending on the type of optical imaging system (100), using a computer to perform image preprocessing on the collected original fluorescence image data to obtain a training set; training a neural network for image denoising and super-resolution processing using the training set on the computer, wherein the neural network includes a denoising portion and a deconvolution portion; the training set includes input image data and true input image data; randomly extracting pixel blocks at the same position from the input image data and true input image data, randomly rotating and inverting them, and then creating a training input image and a training target image, respectively; and training The training input image is first processed by the denoising section to become a denoised image, then processed by the deconvolution section to become a deconvolved image, the denoised image is used to calculate the denoising loss together with the training target image, the deconvolved image is subjected to degradation processing according to the type of optical imaging system (100), and then the deconvolved image is used to calculate the deconvolved loss together with the training target image, the neural network training process includes the steps of: the loss function including the denoising loss and the deconvolved loss or including only the deconvolved loss; and processing the original fluorescence image data or additional fluorescence image data collected from the same biological sample by the optical imaging system (100) using the trained neural network.

Description

[Technical Field]

[0001] This application relates to a method and system for super-resolution processing of microscope images, and more particularly to a self-supervised method and system for super-resolution processing of microscope images. [Background technology]

[0002] Fluorescence microscopy imaging is an important tool for advancing research in life sciences. In recent years, super-resolution microscopy imaging has broken through the optical diffraction limit, enabling dynamic observation of the fine tissues of living cells with a resolution of less than 200 nanometers. However, improvements in the hardware of optical microscope imaging systems in conventional techniques are already approaching their limits. In this case, using super-resolution microscopy imaging for observing biological samples can achieve improved spatial resolution, but at the expense of other imaging performance indicators such as imaging speed and time progression. These imaging performance indicators are equally important for the research and analysis processes of biological samples. In recent years, computational super-resolution methods have greatly expanded the applicability of optical microscope imaging systems because they can instantly improve the resolution of microscopic images through computation.

[0003] Generally, existing computational super-resolution methods can be divided into two categories. The first category is super-resolution algorithms based on conventional analytical models, such as the inverse convolution algorithm. The second category is methods based on deep learning, such as super-resolution convolutional neural networks. In the first category, usually, based on prior assumptions about biological samples and microscopic fluorescence images, a specific analytical model is used to iteratively optimize the resolution of microscopic fluorescence images. However, these methods often have multiple hyperparameters, and the output of the algorithm largely depends on the selection of these parameters. Therefore, users need to individually adjust these parameters according to the processed images, which is time-consuming and laborious. In addition, since the effectiveness of prior assumptions about biological samples and microscopic fluorescence images cannot be guaranteed in actual experiments, this analytical algorithm lacks robustness against noise in microscopic fluorescence images, and reconstruction artifacts are likely to occur especially under input conditions with a low signal-to-noise ratio. On the other hand, recent super-resolution algorithms based on deep learning can learn the end-to-end mapping relationship between high-resolution images and low-resolution images from a large amount of high-quality training data. Thereby, it is possible to improve the resolution of microscopic images without the need for physical modeling of imaging and noise models, achieving results far superior to those of conventional analytical algorithms. However, training a super-resolution network model based on deep learning requires a pair of a large number of low-resolution input images and high-quality true-value super-resolution images. Ultimately, the performance of the super-resolution neural network largely depends on the quality and quantity of the training data. Since living biological samples are very dynamic and photobleach, obtaining a high-quality training set often requires a great deal of human and time costs, and in certain applications, a sufficient training set may not be obtained. These factors significantly hinder the application of super-resolution image processing methods based on deep learning to the super-resolution analysis of microscopic images.

Summary of the Invention

Problems to be Solved by the Invention

[0004] In view of the above problems, the purpose of this application is to propose a novel image super-resolution reconstruction technology solution. This technology solution can train an image super-resolution neural network by making the most of the physical prior information of an optical imaging model or by training the image super-resolution neural network in a self-supervised learning manner. Therefore, even when there is no training data with a high signal-to-noise ratio / resolution, it is possible to train a microscope image super-resolution neural network model using only the original microscope image data with a low signal-to-noise ratio and low resolution.

Means for Solving the Problems

[0005] According to one aspect of this application, collecting original fluorescence image data of a biological sample using an optical imaging system, and Depending on the type of optical imaging system, a computer is used to perform image preprocessing on the collected original fluorescence image data to obtain a training set, and the computer is used to train a neural network for image denoising and super-resolution processing, the neural network including a denoising portion and a deconvolution portion, the training set including input image data and true input image data, randomly extracting pixel blocks at the same position from the input image data and the true input image data, randomly rotating and flipping them to obtain the training input image and training target image, respectively, the training input image is first processed by the denoising portion to become a denoised image, then processed by the deconvolution portion to become a deconvolved image, the denoised image is used together with the training target image to calculate the denoising loss, the deconvolved image is degraded according to the type of optical imaging system and then used together with the training target image to calculate the deconvolution loss, and in the neural network training process, the loss function includes a step of including the denoising loss and the deconvolution loss, or including only the deconvolution loss, A self-supervised microscopy image super-resolution processing method is provided, comprising the step of processing the original fluorescence image data, or additional fluorescence image data collected from the same biological sample by the optical imaging system, using a trained neural network.

[0006] Selectively, if the optical imaging system is a wide-field microscopy imaging system, a scanning confocal microscopy imaging system, a light-sheet illumination microscopy imaging system, or a two-photon scanning imaging system, the original fluorescence image data is one or more two-dimensional noise images of a biological sample independently collected by the optical imaging system, and the training set includes a plurality of training noise image pairs generated by performing image degradation processing on the original fluorescence image data. The aforementioned image degradation process is performed as follows: For a single 2D noise image, a computer is used to randomly generate a normally distributed random variable with the same dimensions as the 2D noise image, a mean of 0, and a variance of 1, and then a pair of fluorescence perturbation images is generated using the following formula:

number

[0007] Selectively, when training the neural network, the degradation of the deconvolved image is performed by convolution of the deconvolved image with the point spread function of the optical imaging system.

[0008] Selectively, if the optical imaging system is a wide-field microscopy imaging system, a scanning confocal microscopy imaging system, a light-sheet illumination microscopy imaging system, or a two-photon scanning imaging system, the original fluorescence image data is a three-dimensional noise image volume stack of a biological sample independently collected by the optical imaging system, the training set includes a first noise image volume stack and a second noise image volume stack expanded by axially spaced sampling of the three-dimensional noise image volume stack, and the denoising image and the deconvolution image are a denoising image volume stack and a deconvolution image volume stack, respectively. The axial interval sampling process described above is: For a single 3D noise image volume stack independently acquired by an optical imaging system, the odd or even layers are extracted as the first noise image volume stack, and accordingly, the even or odd layers are extracted as the second noise image volume stack. When training the neural network, one of the first noise image volume stack and the second noise image volume stack, which correspond to each other one-to-one, is selected as the input image data, and the other of the first noise image volume stack and the second noise image volume stack, which correspond to each other one-to-one, is selected as the true image data. After the neural network has been trained, the trained neural network is used to process the 3D noise image volume stack or the original fluorescence image data of the biological sample collected by the optical imaging system.

[0009] Selectively, when training the neural network, the deconvolution of the deconvolved image is performed by convolution of the deconvolved image volume stack with the 3D point spread function of the optical imaging system.

[0010] Selectively, if the optical imaging system is a structured light illumination super-resolution microscopy imaging system, the original fluorescence image data is a structured light illumination image of a biological sample collected by the optical imaging system, and the training set is two structured light illumination super-resolution images obtained by performing structured light illumination super-resolution reconstruction on the structured light illumination image. When training the neural network, one of the two structured light illumination super-resolution images is used as the input image data, and the other of the two structured light illumination super-resolution images is used as the true image data. After the neural network has been trained, the structured light-illuminated images of biological samples collected by the optical imaging system are processed using the trained neural network after undergoing structured light-illuminated super-resolution reconstruction.

[0011] Selectively, the two structured light illumination super-resolution images are generated by performing structured light illumination super-resolution reconstruction on image sequences of two structured light illumination sources of a biological sample, which are independently collected by the optical imaging system.

[0012] Selectively, the optical imaging system independently collects an image sequence from one structured light source of a biological sample, then performs image degradation processing on each structured light source image or multiple structured light source images in the image of the structured light source to generate multiple pairs of degraded images, extracts the first degraded image from each pair to constitute the first structured light illumination degraded image sequence, extracts the second degraded image from each pair to constitute the second structured light illumination degraded image sequence, and generates the two structured light illumination super-resolution images by performing structured light illumination super-resolution reconstruction on the first structured light illumination degraded image sequence and the second structured light illumination degraded image sequence.

[0013] Selectively, when training the neural network, the degradation process of the deconvolved image is the convolution of the deconvolved image with the super-resolution point spread function of the optical imaging system.

[0014] Selectively, the neural network models for image denoising and super-resolution processing include, but are not limited to, U-shaped neural network models, residual neural network models, residual channel attention convolutional neural network models, or Fourier channel attention convolutional neural network models.

[0015] According to another aspect of this application, a self-supervised microscopy image super-resolution processing system is configured to collect and process original fluorescence image data of a biological sample using an optical imaging system, It includes a microscope image preprocessing module and a denoising and super-resolution reconstruction module, The microscope image preprocessing module is configured to use a computer to perform image preprocessing on the original fluorescence image data to obtain a training set, depending on the type of optical imaging system, and the denoising and super-resolution module is configured with a neural network for image denoising and super-resolution processing, the neural network including a denoising portion and a deconvolutional portion. The neural network is trained on a computer using the training set, the training set includes input image data and true input image data, pixel blocks at the same position are randomly extracted from the input image data and true input image data, randomly rotated and flipped, and then obtained as the training input image and training target image, respectively, the training input image is first processed by the denoising section to become a denoised image, then processed by the deconvolution section to become a deconvolved image, the denoised image is used together with the training target image to calculate the denoising loss, the deconvolved image is degraded according to the type of optical imaging system and then used together with the training target image to calculate the deconvolution loss, in the neural network training process, the loss function includes the denoising loss and the deconvolution loss, or includes only the deconvolution loss, A self-supervised microscopy image super-resolution processing system is provided, in which the noise reduction and super-resolution reconstruction module is configured to process the original fluorescence image data or additional fluorescence image data collected from the same biological sample by the optical imaging system using a trained neural network.

[0016] Optionally, when the optical imaging system is a wide-field microscope imaging system, or a confocal scanning microscope imaging system, or a light sheet illumination microscope imaging system, or a two-photon scanning imaging system, the original fluorescence image data is one or more two-dimensional noisy images of a biological sample independently collected by the optical imaging system, and the training set includes a plurality of training noisy image pairs generated by performing image re-degradation processing on the original fluorescence image data. The image re-degradation processing is In the case of one two-dimensional noisy image, using a computer, a normal distribution random variable with the same image dimension as the one two-dimensional noisy image, an average value of 0, and a variance of 1 is randomly generated, and a pair of fluorescence perturbation images is generated by the following formula:

Equation

[0017] Selectively, when training the neural network, the degradation of the deconvolved image is performed by convolution of the deconvolved image with the point spread function of the optical imaging system.

[0018] Selectively, if the optical imaging system is a wide-field microscopy imaging system, a scanning confocal microscopy imaging system, a light-sheet illumination microscopy imaging system, or a two-photon scanning imaging system, the original fluorescence image data is a three-dimensional noise image volume stack of a biological sample independently collected by the optical imaging system, the training set includes a first noise image volume stack and a second noise image volume stack expanded by axially spaced sampling of the three-dimensional noise image volume stack, and the denoising image and the deconvolution image are a denoising image volume stack and a deconvolution image volume stack, respectively. The axial interval sampling process described above is: For a single 3D noise image volume stack independently acquired by an optical imaging system, the odd or even layers are extracted as the first noise image volume stack, and accordingly, the even or odd layers are extracted as the second noise image volume stack. When training the neural network, one of the first noise image volume stack and the second noise image volume stack, which correspond to each other one-to-one, is selected as the input image data, and the other of the first noise image volume stack and the second noise image volume stack, which correspond to each other one-to-one, is selected as the true image data. After the neural network has been trained, the trained neural network is used to process the 3D noise image volume stack or the original fluorescence image data of the biological sample collected by the optical imaging system.

[0019] Selectively, when training the neural network, the deconvolution of the deconvolved image is performed by convolution of the deconvolved image volume stack with the 3D point spread function of the optical imaging system.

[0020] Selectively, if the optical imaging system is a structured light illumination super-resolution microscopy imaging system, the original fluorescence image data is a structured light illumination image of a biological sample collected by the optical imaging system, and the training set is two structured light illumination super-resolution images obtained by performing structured light illumination super-resolution reconstruction on the structured light illumination image. When training the neural network, one of the two structured light illumination super-resolution images is used as the input image data, and the other of the two structured light illumination super-resolution images is used as the true image data. After the neural network has been trained, the structured light-illuminated images of biological samples collected by the optical imaging system are processed using the trained neural network after undergoing structured light-illuminated super-resolution reconstruction.

[0021] Selectively, the two structured light illumination super-resolution images are generated by performing structured light illumination super-resolution reconstruction on image sequences of two structured light illumination sources of a biological sample, which are independently collected by the optical imaging system.

[0022] Selectively, the optical imaging system independently collects an image sequence from one structured light source of a biological sample, then performs image degradation processing on each structured light source image or multiple structured light source images in the image of the structured light source to generate multiple pairs of degraded images, extracts the first degraded image from each pair to constitute the first structured light illumination degraded image sequence, extracts the second degraded image from each pair to constitute the second structured light illumination degraded image sequence, and generates the two structured light illumination super-resolution images by performing structured light illumination super-resolution reconstruction on the first structured light illumination degraded image sequence and the second structured light illumination degraded image sequence.

[0023] Selectively, when training the neural network, the degradation process of the deconvolved image is the convolution of the deconvolved image with the super-resolution point spread function of the optical imaging system.

[0024] Selectively, the neural network models for image denoising and super-resolution processing include, but are not limited to, U-shaped neural network models, residual neural network models, residual channel attention convolutional neural network models, or Fourier channel attention convolutional neural network models.

[0025] By using a neural network trained with the technical means of this application, the optical resolution of microscope images can be significantly improved, resulting in high fidelity and quantifiable characteristics. Furthermore, since the technical means of this application do not require specific adjustments to hyperparameters when applied to different biological samples or different signal-to-noise ratio data, the applicability range of computational super-resolution methods is greatly expanded. The method and system of this application are particularly suitable for long-term non-destructive imaging of live biological samples with low excitation light power, and can improve the optical resolution of the collected image information while ensuring the activity of the biological sample, thereby enabling clear observation of its rapid dynamic processes and contributing to the research and development of new biological phenomena. [Brief explanation of the drawing]

[0026] The principles and embodiments of this application can be more comprehensively understood by referring to the following detailed description and drawings. Note that the scale of the drawings may be changed for clarity, but this will not affect the understanding of this application.

[0027] [Figure 1] A schematic diagram of the basic block structure of a single microscope imaging system is shown. [Figure 2A] This invention schematically illustrates the process of training the neural network of the noise reduction and super-resolution module of a self-supervised microscope image super-resolution processing system according to one embodiment of this application. [Figure 2B] This schematic diagram illustrates the process by which a neural network makes predictions using trained denoising and super-resolution modules. [Figure 3A] This invention schematically illustrates the process of training the neural network of the denoising and super-resolution module of a self-supervised microscope image super-resolution processing system according to another embodiment of this application. [Figure 3B] This schematic diagram illustrates the process by which a neural network makes predictions using trained denoising and super-resolution modules. [Figure 4A]This invention schematically illustrates the process of training the neural network of the denoising and super-resolution module of a self-supervised microscope image super-resolution processing system according to another embodiment of this application. [Figure 4B] This schematic diagram illustrates the process by which a neural network makes predictions using trained denoising and super-resolution modules. [Figure 5A] This invention schematically illustrates the process of training the neural network of the denoising and super-resolution module of a self-supervised microscope image super-resolution processing system according to another embodiment of this application. [Figure 5B] This schematic diagram illustrates the process by which a neural network makes predictions using trained denoising and super-resolution modules. [Modes for carrying out the invention]

[0028] In the drawings of this application, features having the same structure or similar function are indicated by the same reference numerals.

[0029] Figure 1 schematically shows a basic block diagram of a microscope imaging system. The microscope imaging system includes an optical imaging system 100 and a control and data processing system 200. The optical imaging system 100 may be, but is not limited to, a wide-field microscope imaging system, a scanning confocal microscope imaging system, a light-sheet illumination microscope imaging system, a two-photon scanning microscope imaging system, or a structured light illumination microscope imaging system. Taking a structured light illumination microscope imaging system as an example, the optical imaging system 100 includes an excitation optical path and a detection optical path, the excitation optical path including an excitation objective lens and other optical assemblies for generating excitation light. The excitation light beam is emitted through the excitation objective lens to excite fluorescence in a biological sample. The detection optical path includes a detection objective lens and other optical assemblies for imaging and is used to receive and detect the excited fluorescence. As will be apparent to those skilled in the art, depending on the configuration of the microscope imaging system, the excitation objective lens and the detection objective lens may be the same objective lens or different objective lenses. The optical imaging system 100 can perform two-dimensional or three-dimensional fluorescence microscopy imaging of biological samples, particularly raw biological samples. When performing three-dimensional fluorescence microscopy imaging of biological samples, particularly raw biological samples, multiple layers of fluorescence images are sampled by continuously scanning along the optical axis of the detection objective lens, i.e., along the axial direction. Thus, after each scan and sampling is completed, the acquired multiple layers of fluorescence images constitute a single fluorescence image volume stack (also called a "sequence"). In the context of this application, the term "image" may be understood as a two-dimensional image volume stack or a three-dimensional image volume stack, depending on the requirements of the specific technical invention.

[0030] The control and data processing system 200 mainly comprises a computer and related components (e.g., data memory) and can control the operation of the optical imaging system 100, as well as receive image data from the optical imaging system 100 and perform corresponding post-processing. For example, an acquired fluorescence image volume stack is provided to the control and data processing system 200 and, after a series of data processing steps, is reconstructed into a high signal-to-noise ratio super-resolution microscope image. For this purpose, the control and data processing system 200 may include a microscope image super-resolution processing module or system, or a self-supervised microscope image super-resolution processing module or system. The microscope image super-resolution processing module or system, or a self-supervised microscope image super-resolution processing module or system, mainly comprises a microscope image pre-processing module 210 and a denoising and super-resolution reconstruction module 220. Within the scope of this application, the modules and / or submodules described herein may be understood as including data memory, such as a computer-readable medium, that can store programs or subprograms and denoising neural network models that are called and executed by the computer, in particular the computer of the control and data processing system 200. When these programs or subprograms and denoising neural network models are invoked and executed by a computer, the methods / steps described below, in particular, the self-supervised microscopy image super-resolution processing methods / steps, can be realized. Specific programming methods for the programs and / or subprograms are not discussed. Those skilled in the art can implement the relevant functions using any well-known programming software and / or commercially available software. Therefore, it should be understood that when the operation of the relevant systems, the operation of modules, or methods are described below in this application, they can also be described as programs invoked and executed by a computer.

[0031] The microscope image preprocessing module 210 is configured to obtain original data for training and prediction of neural networks in the denoising and super-resolution processing module 220 by preprocessing fluorescence images acquired by the optical imaging system 100. Processing methods include, but are not limited to, image degradation, axial spacing sampling, and structured optical super-resolution reconstruction. Specific algorithms for image degradation, axial spacing sampling, and structured optical super-resolution reconstruction can be found in the following references, respectively. For example, Pang, T., Zheng, H., Quan, Y. & Ji, H. in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition 2043-2052 (2021) (image degradation); Qiao, C., Li, D., Liu, Y. et al. Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes. Nature Biotechnology, 41, 367-377 (2023) (axial spacing sampling); Gustafsson, MG et al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys J 94, 4957-4970 (2008) (structured light super-resolution reconstruction).

[0032] Furthermore, when the microscope image preprocessing module 210 performs image degradation or axial spacing sampling on an image, the target image (or image data) is a single image acquired directly from the optical imaging system 100 or two noise-independent data pairs available for training, acquired from a single 3D image volume stack, and not training data pairs acquired directly from the optical imaging system 100 (e.g., a high / low signal vs. noise ratio data pair, or a low / low signal vs. noise ratio data pair sampled independently from the same scenario). Also, when the microscope image preprocessing module 210 performs structured light super-resolution reconstruction on an image, the target image (or image data) is the original fluorescence image data acquired by the optical imaging system 100 using structured light illumination, and not the original fluorescence image data acquired by the optical imaging system 100 using other imaging techniques (e.g., wide-field microscopy, scanning confocal microscopy, two-photon microscopy, light-sheet microscopy, etc.), the latter of which require structured light super-resolution reconstruction.

[0033] The denoising and super-resolution module 220 can select any neural network architecture as its base model. For example, the neural network models used in the denoising and super-resolution module 220 include, but are not limited to, U-shaped neural network models, residual neural network models, residual channel attention convolutional neural network models, or Fourier channel attention convolutional neural network models. When training the neural network of the denoising and super-resolution module 220, the relevant network model is optimized using a loss function, which includes, but is not limited to, mean squared error (MSE), mean absolute error (MAE), structural similarity (SSIM), or their weights and sums.

[0034] Therefore, the control and data processing system 200 may also be called a microscope image super-resolution processing system and includes a microscope image preprocessing module 210 and a denoising and super-resolution module 220. Figure 2A schematically shows the process of training the neural network of the denoising and super-resolution module 220 of the microscope image super-resolution processing system using the original fluorescence image acquired by the optical imaging system 100 according to one embodiment of the present application. Figure 2B schematically shows the process of performing denoising and super-resolution processing on the original fluorescence image acquired by the optical imaging system 100 using the microscope image super-resolution processing system 200 with the trained neural network according to one embodiment of the present application.

[0035] With respect to Figure 2A, a wide-field microscope imaging system is described as an example of the optical imaging system 100. As those skilled in the art will see, the other imaging system examples described in this application are similarly applicable. The optical imaging system 100 first performs two independent fluorescence image scans on the same biological sample and samples and acquires noise image A and noise image B (also called the original fluorescence image data). Since the two samples are independent of each other, the noise in noise image A and noise image B is also independent of each other. When the signal-to-noise ratio of noise image A and noise image B are the same, the neural network training mode described later corresponds to the noise-noise training mode. When the signal-to-noise ratio of noise image A is higher than the signal-to-noise ratio of noise image B, the neural network training mode described later corresponds to the supervised training mode.

[0036] According to one embodiment of this application, by using an optical imaging system 100 to sample the same biological sample multiple times at different times, multiple sets of time-distributed noise image A / B pairs can be obtained, and a training set for training the neural network of the denoising and super-resolution module 220 can be constructed. In this training set, each noise image A / B pair is divided in time, and noise image A and noise image B of each noise image A / B pair correspond to each other one-to-one. Within the scope of this application, the term "one-to-one correspondence" means that when training a neural network, if one noise image A or B of a noise image A / B pair is selected as the training input, the other noise image B or A of the same noise image A / B pair becomes the training true value.

[0037] In one alternative embodiment of this application, the noise image A / B pair may be obtained from a single independent noise image using a data augmentation method such as image degradation or axial spacing sampling (applicable to 3D data), provided that the noise in noise image A and noise image B of the pair is distributed independently of each other. In another alternative embodiment of this application, if the optical imaging system 100 is a structured light microscope imaging system, then noise image A and noise image B shown in Figures 2A and 2B refer to structured light super-resolution images obtained by performing structured light super-resolution reconstruction on the collected original microscope fluorescence images.

[0038] Returning to Figure 2A, multiple pairs of noise images A / B are randomly selected from the training set to train the neural network of the denoising and super-resolution module 220. For example, noise image A from each extracted noise image A / B pair is used as the training input image data, and noise image B is used as the training true image data. In each training cycle, pixel blocks at the same image position are randomly selected from the selected one-to-one corresponding noise image A and noise image B, randomly rotated and flipped, and then used as the training input image and training target image for the neural network, respectively.

[0039] The neural network of the denoising and super-resolution module 220 includes two parts: a denoising module part and a deconvolution module part. The denoising module part performs denoising on the input noise image A, and the denoised image A q It is used to output the denoised image A. q The inverse convolution process is performed, and the inverse convolution image A is obtained. j This is used to output the denoised image A. Accordingly, in such a training process, the loss function of the neural network includes denoising loss and deconvolution loss. The denoising loss is used to output the denoised image A. q The inverse convolution loss is expressed as the difference between the degraded inverse convolution image A and the noisy image B. j It is expressed as the difference between and noise image B. In particular, in this embodiment, the inverse convolution image A j The image should first be degraded; that is, it is first convolved with the point spread function of the optical imaging system 100, and then the difference between the convolved result and the noise image is checked. The method for calculating the difference includes, but is not limited to, the mean squared error (MSE), mean absolute error (MAE), structural similarity (SSIM), or their weights and sums.

[0040] In one alternative embodiment of this application, the calculation of the denoising loss portion may be omitted, taking into account the adjustment of various factors such as computation time and result accuracy. In other words, when training the neural network, the loss function only needs to retain the deconvolution loss. This also enables the training of the neural network of the denoising and super-resolution module 220. In this mode, only the deconvolution image can be output, but network efficiency and computation speed can be improved by reducing the scale of the neural network parameters.

[0041] In each iteration of network training, the gradient is calculated based on the loss function, and the network parameters are updated by backpropagating this gradient. After the network input error converges, training is stopped and the network parameters are stored. As can be seen from this, when using the neural network training method described in this application, a high signal-to-noise ratio and high-resolution true images are not required, and therefore it can be considered self-supervised training.

[0042] After the neural network of the denoising and super-resolution module 220 has been trained, the microscope image super-resolution processing system can perform denoising and super-resolution reconstruction (or prediction) on the original fluorescence image sequence acquired by the optical imaging system 100, as shown in Figure 2B. By using the noised image as input to the neural network of the denoising and super-resolution module 220, the final denoised and super-resolution image is obtained. Of course, if denoising and super-resolution reconstruction are to be performed on multiple original fluorescence images, the processing process can be repeated for each original fluorescence image.

[0043] Figures 3A and 3B schematically illustrate a microscope image super-resolution processing method according to one embodiment of the present application. In the illustrated embodiment, the optical imaging system 100 is a two-dimensional wide-field microscope, and the training set for training the neural network of the denoising and super-resolution module 220 consists of a plurality of two-dimensional noise images (two-dimensional images containing noise). In the illustrated embodiment, before training the neural network of the denoising and super-resolution module 220, the microscope image preprocessing module 210 can be used to perform image degradation processing on each or any number of two-dimensional noise images (or original fluorescence image data) collected from a biological sample by the optical imaging system 100, thereby generating a plurality of training noise image A / B pairs to form a training set. Here, each noise image A / B pair includes one noise image A and one noise image B that corresponds to it one-to-one. An example of the image degradation processing is as follows.

[0044] Assuming we have a 2D noise image y, we can use a computer to randomly generate a normally distributed random variable z with the same image dimension (or image pixel size) as the 2D noise image y, with a mean of 0 and a variance of 1. Next, using each original fluorescence image y, we can generate a pair of fluorescence perturbation images (or degraded images) y using the following formula. A and y B Generates.

number

[0045] [Table 1]

[0046] Within the scope of this application, "image y" or "image y A or y B The term "image" can be understood mathematically as a two-dimensional matrix capable of representing an image observable by the human eye. Therefore, the multiplication operation related to the image or matrix in the above formula should be understood as a point multiplication operation.

[0047] For the selected original fluorescence images acquired by the optical imaging system 100, the image degradation process is repeated for each original noisy image by using equations (1) to (3). All the degraded images y generated in this way are then processed. A This constructs the training input image dataset and generates all the degraded images y B This constitutes the true image dataset for training. Instead, all the generated degraded images y A This constitutes the true image dataset for training, and all the generated degraded images y B This may constitute a training input image dataset. Note that one degraded image y generated after each image has undergone degradation processing. A and one further degraded image y B This means that a one-to-one correspondence is formed. In other words, when training a neural network, the single degraded image y A If selected as the input image data, the single degraded image y B This has no choice but to be selected as the true image data.

[0048] Based on the above input image dataset and true image dataset, the neural network of the denoising and super-resolution module 220 is trained. For example, as shown in Figure 3A, in each training cycle, a one-to-one correspondence is obtained between the input dataset and the training dataset and the degraded image y Aand y B Randomly selected images are used as the input image data and true image data, respectively. Pixel blocks at the same positions are randomly extracted, randomly rotated and flipped, and then used as the training input image and training target image for the neural network, respectively. In the illustrated embodiment, the basic neural network architecture can be, for example, a U-shaped neural network. That is, the denoising module and the deconvolution module each utilize one U-shaped neural network. The denoising module uses denoised image A. q Outputs noise-reduced image A q The denoising loss is calculated using the denoised image B. The deconvolution module portion uses the deconvolution image A. j The output is as follows. Note that when generating the training input image dataset and training true image dataset using the "image degradation" method, before calculating the deconvolution loss, first the deconvolution image A j The image must be degraded. That is, first, the deconvolved image A j The point spread function of the optical imaging system 100 is convolved with the image, and then the deconvolution loss is calculated using the convolution result and the noise image B. In each training iteration, the neural network is trained by loss calculation (e.g., using both denoising loss and deconvolution loss or only deconvolution loss) and backpropagation. When training the neural network of the denoising and super-resolution module 220 in the "image degradation" method, network training can be completed using the original fluorescence image sequence acquired only once, thus avoiding damage caused by repeated sampling of raw cells and also making it possible to process raw cell video observation recording data as repeated collection is not required.

[0049] After the denoising and super-resolution module 220's neural network has been trained, denoising and super-resolution processing can be performed using noisy images not used in training in the training set of the trained neural network of the denoising and super-resolution module 220, or noisy images that have been sampled again (not the noisy images in the training set) (as shown in Figure 3B).

[0050] Figures 4A and 4B schematically illustrate a microscope image super-resolution processing method according to another embodiment of the present application. In this embodiment, the optical imaging system 100 is a light-sheet illumination microscope imaging system. Before training the neural network of the denoising and super-resolution module 220, the optical imaging system 100 is used to collect multiple 3D noise image volume stacks (referred to as 3D image volume stacks containing noise, or original fluorescence image data). As shown in Figure 4A, one of the collected 3D noise image volume stacks (i.e., the original noise image volume stack) is subjected to axial spacing sampling by the microscope image preprocessing module 210 to create a noise image volume stack A v (Odd layers of the original noise image volume stack or even layers of the original noise image volume stack) and noise image volume stack B v (The even-numbered layers or odd-numbered layers of the original noise image volume stack are extended.) Next, in the process of training the neural network of the denoising and super-resolution module 220, the noise image volume stack A v and noise image volume stack B v These are used as the input image data and true image data, respectively, and the denoising loss and deconvolution loss, or only the deconvolution loss, are calculated. In the illustrated embodiment, the neural network model utilizes a 3D residual channel attention neural network architecture, and the denoising module and deconvolution module are respectively denoising image volume stack Avq and deconvolved image volume stack A vd Outputs the following: For example, denoising image volume stack A vq and noise image volume stack B v The denoising loss can be calculated using this method. On the other hand, before calculating the deconvolution loss, the denoising image volume stack A vq The image volume stack A must be degraded. That is, noise reduction is required. vq The 3D point spread function of the optical imaging system 100 is convolved with the obtained result and the noise image volume stack B v The error is calculated as the deconvolution loss. At each iteration of training, the gradient is calculated using the denoising loss and the deconvolution loss (or using only the deconvolution loss), the neural network weights are updated based on the backpropagation, and the training process is completed.

[0051] As shown in Figure 4B, after the neural network training of the denoising and super-resolution module 220 is completed, the trained neural network of the denoising and super-resolution module 220 can take the original noisy image volume stack as input and simultaneously output the corresponding denoised image volume stack and deconvolved image volume stack, achieving an improvement in signal-to-noise ratio and resolution compared to the original data.

[0052] Figures 5A and 5B schematically illustrate a microscopy image super-resolution processing method according to another embodiment of the present application. In this embodiment, the optical imaging system 100 is a structured light illumination super-resolution microscopy imaging system, and the microscopy image preprocessing module 210 is configured to perform structured light super-resolution reconstruction on the original structured light illumination image collected by the optical imaging system 100. Therefore, in the training process shown in Figure 5A, the neural network of the denoising and super-resolution module 220 is trained in a "noise-noise" training mode. Specifically, before preparing data for training the neural network of the denoising and super-resolution module 220, two image sequences y from structured light illumination sources are prepared for each structural region of a biological sample. A,i and y B,i The data are collected independently of each other, where i is an integer greater than 1. Subsequently, the image sequence y of the structured light illumination source is processed using the microscope image preprocessing module 210. A,i and y B,i Noised structured light illumination super-resolution image y A and y B These are then reconstructed and used as training images. Subsequently, when training the neural network of the denoising and super-resolution module 220, the noisy structured light illumination super-resolution image (or noise SIM image) Y A and Y B The input image data and true image data are used as the neural network training input image data and true image data respectively, the network loss (e.g., denoising loss and deconvolution loss, or deconvolution loss only) is calculated, and then the gradient calculation and network weight update are completed. In this embodiment, the selection of the neural network model and the calculation of the loss function are the same as in the embodiments shown in Figures 3A and 3B. In each training cycle, the noisy structured light-illuminated super-resolution image Y A and Y BPixel blocks at the same position are randomly extracted, randomly rotated and flipped, and then used as the training input image and training target image for the neural network, respectively. In the illustrated embodiment, the basic neural network architecture can be, for example, a U-shaped neural network. That is, the denoising module and the deconvolution module each utilize one U-shaped neural network. The denoising module uses the denoising SIM image y Aq Outputs the denoised SIM image yAq and the noisy SIM image y B The noise reduction loss is calculated using the following: The inverse convolution module part is the inverse convolution SIM image y Aj Outputs the deconvolution SIM image y before calculating the deconvolution loss. Aj The point spread function used when degrading the image should be the super-resolution point spread function (i.e., the full width at half maximum is approximately half that of the wide-field point spread function, and not the point spread function under non-wide-field illumination), rather than the point spread function of the optical imaging system 100. Subsequently, the degraded inverse convolution SIM image Y Aj and noise SIM image Y Bと Using 、 Calculate the deconvolution loss.

[0053] As shown in Figure 5B, after the neural network training of the denoising and super-resolution module 220 is completed, for the noisy structured light illumination source image requiring denoising and super-resolution processing, the microscope image preprocessing module 210 is first used to reconstruct it into a noisy structured light illumination super-resolution image. Then, the reconstructed noisy structured light illumination super-resolution image is input to the denoising and super-resolution module 220, which has been trained by the neural network, thereby completing the denoising and resolution enhancement processing of the structured light illumination super-resolution image.

[0054] In one alternative embodiment of the example shown in Figure 5A, before preparing data for training the neural network of the denoising and super-resolution module 220, only an image sequence from one structured light source may be collected for each structural region of a biological sample or biological sample. Then, using the image degradation method of equations (1), (2), and (3), one degraded image pair y is obtained from each image of the structured light source in the image sequence of the structured light source or from multiple images of the structured light source. A and y B This generates a number of degraded image pairs, and each degraded image y of the degraded image pair generates a number of degraded image pairs. A Extract and combine into a single structured light-illuminated degraded image sequence y A,i The structure consists of the degraded image y of each degraded image pair. B Extract and combine into a single structured light-illuminated degraded image sequence y B,i This constructs a sequence where i is an integer greater than 1. Next, the microscope image preprocessing module 210 is used to process the image sequence y of the structured light illumination source. A,i and y B,i Noised structured light illumination super-resolution image Y A and Y B The image is reconstructed and used as a training image. In this alternative embodiment, the training content for the neural network of the denoising and super-resolution module 220 can be found in the content shown in Figure 5A. Furthermore, after the training of the neural network of the denoising and super-resolution module 220 is complete, the image of the noisy structured light source requiring denoising and super-resolution processing can be found in the content shown in Figure 5B.

[0055] As can be seen from the above description of this application, in the embodiments shown in Figures 3A, 3B and 4A, 4B, in the neural network training process, the original fluorescence image data must be processed by the microscope image preprocessing module 210 before being used to train the neural network. On the other hand, in the prediction process, the original fluorescence image data can be input to the denoising and super-resolution reconstruction module 220, which the neural network has been trained on, to obtain the final image after denoising and super-resolution processing. Similarly, in the embodiments shown in Figures 5A and 5B, in the neural network training process, the original fluorescence image data must be processed by the microscope image preprocessing module 210 before being used to train the neural network. However, in the prediction process, the original fluorescence image data still needs to be processed by the microscope image preprocessing module 210 before being input to the denoising and super-resolution reconstruction module 220, thereby obtaining the final image after denoising and super-resolution processing.

[0056] As can be seen from the description of the above embodiment, this application relates to the step of collecting original fluorescence image data of a biological sample using an optical imaging system 100, Depending on the type of optical imaging system 100, a computer is used to perform image preprocessing on the collected original fluorescence image data to obtain a training set, and the computer is used to train a neural network for image denoising and super-resolution processing, the neural network including a denoising portion and a deconvolution portion, the training set including input image data and true input image data, randomly extracting pixel blocks at the same position from the input image data and true input image data, randomly rotating and flipping them, and then training input image and training target image respectively. The training input image is first processed by the denoising section to become a denoised image, then processed by the deconvolution section to become a deconvolved image, the denoised image is used together with the training target image to calculate the denoising loss, the deconvolved image is degraded according to the type of optical imaging system 100 and then used together with the training target image to calculate the deconvolution loss, and in the training process of the neural network, the loss function includes a step that includes the denoising loss and the deconvolution loss, or the deconvolution loss alone. The present invention provides a self-supervised microscopy image super-resolution processing method, comprising the step of processing the original fluorescence image data or additional fluorescence image data collected from the same biological sample by the optical imaging system 100 using a trained neural network.

[0057] Furthermore, this application relates to a self-supervised microscopy image super-resolution processing system configured to collect and process original fluorescence image data of a biological sample using an optical imaging system 100, It includes a microscope image preprocessing module 210 and a noise reduction and super-resolution reconstruction module 220. The microscope image preprocessing module 210 is configured to use a computer to perform image preprocessing on the collected original fluorescence image data to obtain a training set, depending on the type of optical imaging system 100, and the denoising and super-resolution module 220 is configured with a neural network for image denoising and super-resolution processing, the neural network including a denoising portion and a deconvolutional portion, The neural network is trained on a computer using the training set, the training set includes input image data and true input image data, pixel blocks at the same position are randomly extracted from the input image data and true input image data, randomly rotated and flipped, and then obtained as the training input image and training target image, respectively, the training input image is first processed by the denoising section to become a denoised image, then processed by the deconvolution section to become a deconvolved image, the denoised image is used together with the training target image to calculate the denoising loss, the deconvolved image is degraded according to the type of optical imaging system 100 and then used together with the training target image to calculate the deconvolution loss, in the neural network training process, the loss function includes the denoising loss and the deconvolution loss, or includes only the deconvolution loss, The noise reduction and super-resolution reconstruction module 220 provides a self-supervised microscopy image super-resolution processing system configured to process the original fluorescence image data or additional fluorescence image data collected from the same biological sample by the optical imaging system 100 using a trained neural network.

[0058] The main advantages of the proposed technology in this application are as follows: (1) The proposed technology of this application can complete the training of a super-resolution neural network model without requiring high and low signal-to-noise ratio data pairs. (2) This application improves image resolution and provides very strong noise immunity for image denoising, by using two independent microscope images (volume stacks) containing noise as training inputs and true values ​​for a neural network, respectively. (3) The proposed technology of this application is applicable to a variety of different microscope imaging systems and can achieve multimodal noise reduction and super-resolution imaging. (4) Based on data augmentation methods such as image degradation and axial spacing sampling, the proposed technology of this application can be directly applied to long-duration video data, and a training set can be built directly from the data itself, and noise reduction and super-resolution processing can be performed on it, making it very convenient to use.

[0059] While the diffraction-limited resolution of conventional wide-field illumination optical microscopes is approximately 200 nanometers, the resolution of microscope images processed using the self-supervised noise reduction and super-resolution method described in this application can be improved to approximately 120 nanometers. In other words, the optical resolution of microscope images can be improved by more than 1.5 times. In particular, in the case of super-resolution microscope imaging techniques such as structured light illumination microscopes, the method described in this application can further improve the resolution based on that resolution. For example, the resolution of structured light illumination super-resolution microscope imaging techniques is approximately 100 nanometers, but the method described in this application can further improve that resolution to approximately 60 nanometers.

[0060] In summary, this application designs a novel self-supervised noise reduction and super-resolution processing method that improves the optical resolution of microscope images by more than 1.5 times, even when high signal-to-noise ratio and high-resolution data are unavailable. The output results possess high fidelity and quantifiable characteristics, and since it does not require hyperparameter specificity adjustments when applied to different biological samples or data with different signal-to-noise ratios, the applicability of computational super-resolution methods is greatly expanded.

[0061] While specific embodiments of this application are described in detail herein, these are for illustrative purposes only and should not be construed as limiting the scope of this application. Furthermore, as will be apparent to those skilled in the art, the embodiments described herein can be used in combination with each other. Various substitutions, modifications, and improvements can be made without departing from the spirit and scope of this application.

Claims

1. A self-supervised microscopy image super-resolution processing method, The steps include: collecting the original fluorescence image data of a biological sample using an optical imaging system (100); Depending on the type of optical imaging system (100), a computer is used to perform image preprocessing on the collected original fluorescence image data to obtain a training set, and the computer is used to train a neural network for image denoising and super-resolution processing, the neural network including a denoising portion and a deconvolution portion, the training set including input image data and true input image data, randomly extracting pixel blocks at the same position from the input image data and true input image data, randomly rotating and flipping them, and then training input image and training target image respectively. The training input image is first processed by the denoising section to become a denoised image, then processed by the deconvolution section to become a deconvolved image, the denoised image is used together with the training target image to calculate the denoising loss, the deconvolved image is degraded according to the type of optical imaging system (100) and then used together with the training target image to calculate the deconvolution loss, and in the training process of the neural network, the loss function includes a step of including the denoising loss and the deconvolution loss, or including only the deconvolution loss, The steps include processing the original fluorescence image data or additional fluorescence image data collected from the same biological sample by the optical imaging system (100) using a trained neural network, A self-supervised microscopy image super-resolution processing method, including the above.

2. If the optical imaging system (100) is a wide-field microscopy imaging system, a scanning confocal microscopy imaging system, a light-sheet illumination microscopy imaging system, or a two-photon scanning imaging system, the original fluorescence image data is one or more two-dimensional noise images of a biological sample independently collected by the optical imaging system (100), and the training set includes a plurality of training noise image (A / B) pairs generated by performing an image degradation process on the original fluorescence image data. The aforementioned image degradation process is performed as follows: For a single two-dimensional noise image, a computer is used to randomly generate a normally distributed random variable with the same image dimension as the single two-dimensional noise image, a mean of 0, and a variance of 1, and a pair of fluorescence perturbation images is generated using the following formula: [Math 1] In the above formulas (1), (2), and (3), α and β １ , β ２ are constants randomly generated by a computer, where α = 0.2 to 5.0, β １ = 0.2 to 5, β ２ = 0.5σ ２ to 1.5σ ２ , y represents the two-dimensional noise image, z represents the normal distribution random variable, y Ａ represents one of the fluorescence perturbation images in the fluorescence perturbation image pair, y Ｂ represents the other fluorescence perturbation image in the fluorescence perturbation image pair, σ ２ is the variance of the noise floor of the camera used in the optical imaging system (100), and the fluorescence perturbation image pair constitutes one training noise image (A / B) pair. When training the neural network, one noise image from one of the training noise image (A / B) pairs is selected as input image data, and the other noise image from the selected training noise image (A / B) pair is selected as true input image data. The self-supervised microscopy image super-resolution processing method according to claim 1, wherein, after the training of the neural network is completed, the trained neural network is used to process a two-dimensional noise image of a biological sample collected by the optical imaging system (100), or to process the original fluorescence image data.

3. The self-supervised microscope image super-resolution processing method according to claim 2, wherein when training the neural network, the degradation process of the deconvolved image is the convolution of the deconvolved image with the point spread function of the optical imaging system (100).

4. If the optical imaging system (100) is a wide-field microscopy imaging system, or a scanning confocal microscopy imaging system, or a light-sheet illumination microscopy imaging system, then the original fluorescence image data is a three-dimensional noise image volume stack of a biological sample independently collected by the optical imaging system (100), and the training set is a first noise image volume stack (A) expanded by axially spaced sampling of the three-dimensional noise image volume stack. ｖ ) and the second noise image volume stack (B ｖ ) including the denoising image and the deconvolution image are a denoising image volume stack and a deconvolution image volume stack, respectively. The axial interval sampling process described above is: For a single three-dimensional noise image volume stack independently acquired by the optical imaging system (100), the odd or even layers are assigned to the first noise image volume stack (A ｖ ) is extracted as such, and accordingly, the even or odd layers are selected as the second noise image volume stack (B ｖ Extracted as follows: When training the neural network, a first noise image volume stack (A) has a one-to-one correspondence with each other. ｖ ) and the second noise image volume stack (B ｖ One of the above is selected as the input image data, and the first noise image volume stack (A) corresponds to each other one-to-one. ｖ ) and the second noise image volume stack (B ｖ The other of the two is selected as the true image data, The self-supervised microscopy image super-resolution processing method according to claim 1, wherein, after the training of the neural network is completed, the trained neural network is used to process a three-dimensional noise image volume stack or the original fluorescence image data of a biological sample collected by the optical imaging system (100).

5. The self-supervised microscope image super-resolution processing method according to claim 3, wherein, when training the neural network, the degradation of the deconvolved image is performed by convolution of the deconvolved image volume stack with the three-dimensional point spread function of the optical imaging system (100).

6. If the optical imaging system (100) is a structured light illumination super-resolution microscopy imaging system, the original fluorescence image data is a structured light illumination image of a biological sample collected by the optical imaging system (100), and the training set is two structured light illumination super-resolution images (Y) obtained by performing structured light illumination super-resolution reconstruction on the structured light illumination image. Ａ and Y Ｂ ) and When training the neural network, one of the two structured light illumination super-resolution images is used as the input image data, and the other of the two structured light illumination super-resolution images is used as the true image data. The self-supervised microscopy image super-resolution processing method according to claim 1, wherein, after the training of the neural network is completed, the structured light illumination super-resolution reconstruction of the structured light illumination image of the biological sample collected by the optical imaging system (100) is performed using the trained neural network and then processed.

7. The two structured light illumination super-resolution images (Y Ａ and Y Ｂ ) is an image sequence (image y) of a biological sample independently collected by the optical imaging system (100) under two structured light illumination sources. Ａ，ｉ and y Ｂ，ｉ The self-supervised microscope image super-resolution processing method according to claim 6, which is generated by performing structured light illumination super-resolution reconstruction on ) ).

8. The optical imaging system (100) captures an image sequence (y) of a biological sample from one structured light source. Ａ，ｉ and y Ｂ，ｉ After independently collecting the images of the structured light source, the image degradation process is performed on each image of the structured light source or on multiple images of the structured light source to generate multiple pairs of degraded images, and the first degraded image (y) from each pair of degraded images is selected. Ａ ) is extracted and the first structured light-illuminated degraded image sequence (y Ａ，ｉ ) constitutes the second degraded image (y) of each degraded image pair. Ｂ ) is extracted and the second structured light-illuminated degraded image sequence (y Ｂ，ｉ ) constitutes the first structured light illumination degraded image sequence (y Ａ，ｉ ) and the second structured light-illuminated degraded image sequence (y Ｂ，ｉ By performing structured light illumination super-resolution reconstruction on the two structured light illumination super-resolution images (Y Ａ and Y Ｂ A self-supervised microscopy image super-resolution processing method according to claim 6, which generates ).

9. The self-supervised microscope image super-resolution processing method according to any one of claims 6 to 8, wherein when training the neural network, the degradation processing of the deconvolved image is a convolution of the deconvolved image with the super-resolution point spread function of the optical imaging system (100).

10. The self-supervised microscopy image super-resolution processing method according to any one of claims 1 to 9, wherein the neural network model for image noise reduction and super-resolution processing includes, but is not limited to, a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.

11. A self-supervised microscopy image super-resolution processing system configured to collect and process original fluorescence image data of a biological sample using an optical imaging system (100), It includes a microscope image preprocessing module (210) and a noise reduction and super-resolution reconstruction module (220), The microscope image preprocessing module (210) is configured to use a computer to perform image preprocessing on the original fluorescence image data to obtain a training set, depending on the type of optical imaging system (100), and the denoising and super-resolution module (220) is configured with a neural network for image denoising and super-resolution processing, the neural network including a denoising portion and a deconvolutional portion. The neural network is trained on a computer using the training set, the training set includes input image data and true input image data, pixel blocks at the same position are randomly extracted from the input image data and true input image data, randomly rotated and flipped, and then obtained as the training input image and training target image, respectively, the training input image is first processed by the denoising section to become a denoised image, then processed by the deconvolution section to become a deconvolved image, the denoised image is used together with the training target image to calculate the denoising loss, the deconvolved image is degraded according to the type of optical imaging system (100) and then used together with the training target image to calculate the deconvolution loss, in the neural network training process, the loss function includes the denoising loss and the deconvolution loss, or includes only the deconvolution loss. The noise reduction and super-resolution reconstruction module (220) is configured to process the original fluorescence image data or additional fluorescence image data collected from the same biological sample by the optical imaging system (100) using a trained neural network, in a self-supervised microscopy image super-resolution processing system.

12. If the optical imaging system (100) is a wide-field microscopy imaging system, a scanning confocal microscopy imaging system, a light-sheet illumination microscopy imaging system, or a two-photon scanning imaging system, the original fluorescence image data is one or more two-dimensional noise images of a biological sample independently collected by the optical imaging system (100), and the training set includes a plurality of training noise image (A / B) pairs generated by performing an image degradation process on the original fluorescence image data. The aforementioned image degradation process is performed as follows: For a single two-dimensional noise image, a computer is used to randomly generate a normally distributed random variable with the same image dimension as the single two-dimensional noise image, a mean of 0, and a variance of 1, and a pair of fluorescence perturbation images is generated using the following formula: [Math 1] In the above equations (1), (2), and (3), α and β １ , β ２ These are constants randomly generated by a computer, with α = 0.2 to 5.0 and β = β. １ = 0.2 to 5, β ２ = 0.5σ ２ ~1.5σ ２ , σ ２ is the variance of the noise floor of the camera used in the optical imaging system (100), where y represents the two-dimensional noise image, and z represents the normally distributed random variable, y Ａ represents one of the fluorescence perturbation images in the fluorescence perturbation image pair, and y Ｂ This represents the other fluorescence perturbation image in the fluorescence perturbation image pair, and this fluorescence perturbation image pair constitutes one training noise image (A / B) pair. When training the neural network, one noise image from one of the training noise image (A / B) pairs is selected as input image data, and the other noise image from the selected training noise image (A / B) pair is selected as true input image data. The self-supervised microscopy image super-resolution processing system according to claim 11, wherein, after the training of the neural network is completed, the trained neural network is used to process a two-dimensional noise image of a biological sample collected by the optical imaging system (100) or to process the original fluorescence image data.

13. The self-supervised microscope image super-resolution processing system according to claim 12, wherein, when training the neural network, the degradation process of the deconvolved image is the convolution of the deconvolved image with the point spread function of the optical imaging system (100).

14. If the optical imaging system (100) is a wide-field microscopy imaging system, or a scanning confocal microscopy imaging system, or a light-sheet illumination microscopy imaging system, then the original fluorescence image data is a three-dimensional noise image volume stack of a biological sample independently collected by the optical imaging system (100), and the training set is a first noise image volume stack (A) expanded by axially spaced sampling of the three-dimensional noise image volume stack. ｖ ) and the second noise image volume stack (B ｖ ) including the denoising image and the deconvolution image are a denoising image volume stack and a deconvolution image volume stack, respectively. The axial interval sampling process described above is: For a single three-dimensional noise image volume stack independently acquired by the optical imaging system (100), the odd or even layers are assigned to the first noise image volume stack (A ｖ ) is extracted as such, and accordingly, the even or odd layers are selected as the second noise image volume stack (B ｖ Extracted as follows: When training the neural network, a first noise image volume stack (A) has a one-to-one correspondence with each other. ｖ ) and the second noise image volume stack (B ｖ One of the above is selected as the input image data, and the first noise image volume stack (A) corresponds to each other one-to-one. ｖ ) and the second noise image volume stack (B ｖ The other of the two is selected as the true image data, The self-supervised microscopy image super-resolution processing system according to claim 11, wherein, after the training of the neural network is completed, the trained neural network is used to process a three-dimensional noise image volume stack or the original fluorescence image data of a biological sample collected by the optical imaging system (100).

15. The self-supervised microscope image super-resolution processing system according to claim 13, wherein, when training the neural network, the degradation of the deconvolved image is performed by convolution of the deconvolved image volume stack with the three-dimensional point spread function of the optical imaging system (100).

16. If the optical imaging system (100) is a structured light illumination super-resolution microscopy imaging system, the original fluorescence image data is a structured light illumination image of a biological sample collected by the optical imaging system (100), and the training set is two structured light illumination super-resolution images (Y) obtained by performing structured light illumination super-resolution reconstruction on the structured light illumination image. Ａ and Y Ｂ ) and When training the neural network, one of the two structured light illumination super-resolution images is used as the input image data, and the other of the two structured light illumination super-resolution images is used as the true image data. The self-supervised microscopy image super-resolution processing system according to claim 11, wherein, after the training of the neural network is completed, the trained neural network is used to perform structured light illumination super-resolution reconstruction on structured light illumination images of biological samples collected by the optical imaging system (100) and then process them.

17. The two structured light illumination super-resolution images (Y Ａ and Y Ｂ ) is an image sequence (y ) of a biological sample independently collected by the optical imaging system (100) from two structured light illumination sources. Ａ，ｉ and y Ｂ，ｉ A self-supervised microscope image super-resolution processing system according to claim 16, which is generated by performing structured light illumination super-resolution reconstruction on ) ).

18. The optical imaging system (100) captures an image sequence (y) of a biological sample from one structured light source. Ａ，ｉ and y Ｂ，ｉ After independently collecting the images of the structured light source, the image degradation process is performed on each image of the structured light source or on multiple images of the structured light source to generate multiple pairs of degraded images, and the first degraded image (y) from each pair of degraded images is selected. Ａ ) is extracted and the first structured light-illuminated degraded image sequence (y Ａ，ｉ ) constitutes the second degraded image (y) of each degraded image pair. Ｂ ) is extracted and the second structured light-illuminated degraded image sequence (y Ｂ，ｉ ) constitutes the first structured light illumination degraded image sequence (y Ａ，ｉ ) and the second structured light-illuminated degraded image sequence (y Ｂ，ｉ By performing structured light illumination super-resolution reconstruction on the two structured light illumination super-resolution images (Y Ａ and Y Ｂ A self-supervised microscope image super-resolution processing system according to claim 16, which generates ).

19. A self-supervised microscope image super-resolution processing system according to any one of claims 16 to 18, wherein when training the neural network, the degradation of the deconvolved image is performed by convolution of the deconvolved image with the super-resolution point spread function of the optical imaging system (100).

20. The self-supervised microscopy image super-resolution processing system according to any one of claims 11 to 19, wherein the neural network model includes, but is not limited to, a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.

Images