Method and computer program for reducing noise in self-monitoring structured light-illuminated super-resolution microscopic images

Abstract

This invention provides a method and program for denoising images captured in structural light illumination mode. [Solution] The super-resolution image noise reduction network that performs the noise reduction method comprises a super-resolution reconstruction module 100 that reconstructs a raw image in super-resolution and outputs a super-resolution reconstructed image, and a super-resolution noise reduction module 200 constructed using a neural network that performs noise reduction processing on the super-resolution reconstructed image and outputs a super-resolution noise-reduced image. The network provides at least one set of raw images, and performs downsampling first and then upsampling. The raw images are divided into a training input raw image and a training target raw image, which are then amplified to generate a training input raw image set and a training target raw image set, and a training network model for the deep learning network is constructed, thereby performing self-monitoring rationalized deep learning image noise reduction.

Description

[Technical Field]

[0001] This application relates, in general, to a method for denoising images captured using a structured light illumination mode based on self-monitoring rationalized deep learning using a neural network. [Background technology]

[0002] The invention of the optical microscope has enabled people to observe microscopic life phenomena within cells, greatly advancing the development of life sciences. However, conventional optical microscopes are generally limited by the optical diffraction limit and cannot observe detailed structures on a scale of 200 nm or less. In recent years, from a hardware perspective, various super-resolution fluorescence microimaging techniques have been invented, pushing the resolution of imaging systems to the level of 10-100 nm. Structured illumination microscopy (SIM) is an ideal choice for studying dynamic interactions between cell organelles within cells, and it boasts high imaging speed while maintaining relatively low photobleaching and phototoxicity. However, fluorescent proteins have drawbacks such as a limited number of photons, susceptibility to bleaching, or quenching, and these drawbacks limit the acquisition of high-quality super-resolution fluorescence images over long periods.

[0003] With the rapid iteration of hardware technologies such as computers and graphics, deep learning-based neural network algorithms have achieved significant advancements from a software perspective. In recent years, image noise reduction neural networks based on surveillance learning, such as Content-aware image restoration (CARE) and Deep Fourier channel attention network (DFCAN), have been proposed. Their superior noise reduction performance significantly improves image quality, enabling long-term, high signal-to-noise ratio super-resolution imaging of living cells. Surveillance noise reduction networks typically require the collection of a large number of signal-to-noise ratio (L / H) and structurally consistent (image) data pairs as a training set, with low signal-to-noise ratio data as input images and high signal-to-noise ratio data as output images to train the network model. However, in actual cell samples, there are many cell organelles with strong background signals, weak fluorescence signals, or fast movement speeds, making it difficult to collect high signal-to-noise ratio datasets that meet the requirements. All of these factors limit the applicability of surveillance noise reduction networks. Therefore, researchers are developing methods such as Noise2void, Blind2unblind, and Neighbor2Neighbor. They are further developing self-monitoring noise reduction neural networks (hereinafter, neural networks can also be abbreviated as networks) to reduce the requirements for the training set. The training logic of a self-monitoring noise reduction network generally involves dividing a single image into two or more parts, selecting one as the input image, and training the noise reduction network model with the other as the target image. The effect of monitoring noise reduction can be restored simply by providing a sufficient number of images for training.

[0004] However, when applying noise reduction processing to existing self-monitoring noise reduction networks using structured optical super-resolution images after training and distribution, reconstruction and motion artifacts in the reconstructed images are not effectively removed, image resolution is reduced, and a good noise reduction effect cannot be obtained, especially under conditions where the signal noise of the original data is relatively low. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] In response to the shortcomings of the noise reduction algorithms described above, this invention aims to propose a self-monitoring-based rationalized deep learning image noise reduction method. This method includes a super-resolution noise reduction branch and a raw image noise reduction branch as inputs to two unbalanced branches when training the network using acquired low signal-to-noise ratio images. By utilizing physical prior information constraints such as illumination stripes, it is possible to better remove noise in super-resolution images, more effectively reducing reconstruction and motion artifact problems in the super-resolution image reconstruction process, and simultaneously achieving a more accurate and high-quality super-resolution image noise reduction effect under different super-resolution image modes (2D or 3D super-resolution images) and various imaging conditions (various signal-to-noise ratios, imaging speeds, and imaging time lengths). [Means for solving the problem]

[0006] According to one aspect of the present application, a method for reducing noise in rationalized deep learning images based on self-monitoring or noise reduction in self-monitoring structured optically illuminated super-resolution microscopic images is provided. The process includes providing a super-resolution image denoising network, the super-resolution image denoising network comprising an upstream super-resolution reconstruction module and a downstream super-resolution noise reduction module, wherein the super-resolution reconstruction module is configured to super-reconstruct a raw image captured in a structured light illumination mode using a super-resolution reconstruction algorithm and output a super-resolution reconstructed image, the super-resolution noise reduction module is configured to denoise the super-resolution reconstructed image and output a super-resolution noise reduction image, the super-resolution noise reduction module is constructed using a neural network, and the super-resolution image denoising network is trained in the following manner before deployment: The present invention provides at least one set of primitive images taken of a biological structure in the structural light illumination mode, each set of primitive images comprising N × M primitive images, where N and M are integers of 2 or more, where N represents the direction in the structural light illumination mode, and M represents the structural light stripe phase in the structural light illumination mode. In a method that first downsamples and then upsamples, at least one set of the original images is divided into at least one set of training input original images and at least one set of training target original images. At least one set of the training input primitive images and at least one set of the training target primitive images are amplified to generate a training input primitive image set and a training target primitive image set. A training network model of a self-monitoring deep learning network is constructed for the super-resolution image noise reduction network, and the training network model of the self-monitoring deep learning network comprises a first feature extraction module, a second feature extraction module, a structure recovery module, and a modulation module, wherein the first feature extraction module, the second feature extraction module, and the structure recovery module are constructed using a neural network, the first feature extraction module and the second feature extraction module are configured to perform feature extraction on an input image, and the modulation module is configured to output a stripe-modulated image by performing point multiplication on a stripe with the same direction and phase as the illumination stripe used during structural light illumination mode shooting of the input image. When training the network model, the super-resolution reconstruction module, the super-resolution noise reduction module, the modulation module, and the first feature extraction module sequentially configure a first image processing branch, and the second feature extraction module configures an independent second image processing branch. The training input primitive image selected from the training input primitive image set is input to the first image processing branch and the second image processing branch, respectively. The output of the first image processing branch and the output of the second image processing branch are added together and used as the input to the structure recovery module. The output of the structure recovery module is averaged with the primitive image input to the first image processing branch and the primitive image input to the second image processing branch to obtain the output for network model training. The training target primitive image selected from the training target primitive image set is used as the target for network model training.

[0007] In the above embodiment, in a method that first downsamples and then upsamples, each of the N×M original images in a set of original images is divided into adjacent and identically sized pixel regions, at least one pixel is randomly selected from each pixel region, interpolation is used to generate the training input original image corresponding to that original image, and at least one pixel is selected from the remaining pixels in each pixel region, interpolation is used to generate the training target original image corresponding to that original image, thereby generating a set of training input original images and a set of training target original images.

[0008] Furthermore, in the above embodiment, the size of the pixel region is 2x2, and in a method that performs downsampling first and then upsampling, one pixel may be randomly extracted from each of the pixel regions, and interpolation may be used to generate the training input primitive image.

[0009] Furthermore, in the above embodiment, at least one set of original images captured with respect to the biological structure in the structural light illumination mode may be original images with a low signal-to-noise ratio.

[0010] Furthermore, in the above embodiment, the amplification method may include random clipping, random angular rotation, or image symmetrical operation.

[0011] Furthermore, in the above embodiment, the modulation module may be configured to first point multiply the input image by a stripe having the same direction and phase as the illumination stripe used during structural light illumination mode imaging, then convolve this with the system point diffusion function of the optical imaging system for capturing the original image, and output the stripe-modulated image.

[0012] Furthermore, in the above embodiment, the training input primitive image set and the training target primitive image set may be subjected to normalization processing. In addition, in the above aspect, the loss function of the training network model is the average absolute error (MAE) and the structural similarity (SSIM) between the network output image

Number

Number

[0013] In addition, in the above aspect, the neural network may include a convolutional neural network, a deep neural network, a Transformer, or a neural network based on an attention mechanism.

[0014] In addition, in the above aspect, the super-resolution noise reduction module, the first feature extraction module, the second feature extraction module, and / or the structure restoration module may be constructed using the same neural network or different neural networks.

[0015] In addition, in the above aspect, the neural network model for constructing the super-resolution noise reduction module, the first feature extraction module, the second feature extraction module, and / or the structure restoration module is a neural network model that can converge during training, and may include a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.

[0016] In addition, in the above aspect, any one of the super-resolution noise reduction module, the first feature extraction module, the second feature extraction module, and the structure restoration module includes a shallow feature extraction sub-module, a deep feature extraction sub-module, and an output sub-module connected in series in sequence. The shallow feature extraction sub-module has a convolutional layer and an activation layer connected in series in sequence. The deep feature extraction sub-module has p residual groups, where p is an integer greater than or equal to 1. The input of the deep feature extraction sub-module is added to the output of the last one of the residual groups of the deep feature extraction sub-module. Each of the residual groups has q residual blocks, where q is an integer greater than or equal to 1. The input of each of the residual groups is added to the output of the last one of the residual blocks of the residual group. Each of the residual blocks has a convolutional layer and an activation layer connected in series, and a channel attention module connected in series downstream. The input of the residual block is added to the output of the channel attention module of the residual block. The channel attention module is composed of a global average pooling layer, a convolutional layer, an activation layer, a convolutional layer, and an activation layer connected in series, calculates the weights of each channel of the input image, multiplies each of the weights and each of the channels of the input image to obtain an output image, and may also be configured to add the input of the channel attention module to the output of the last activation layer of the channel attention module.

[0017] In addition, in the above aspect, the shallow feature extraction sub-module may have one pair, two pairs, or more pairs of convolutional layers and activation layers connected in series in sequence, and / or each of the residual blocks may have one pair, two pairs, or more pairs of convolutional layers and activation layers connected in series in sequence.

[0018] In addition, in the above aspect, in the shallow feature extraction sub-module and / or the residual block and / or the output sub-module, the activation function of the activation layer is a LeakyReLU activation function, which is represented as follows.

Equation

[0019] Furthermore, in the above embodiment, the activation function of the upstream activation layer in the channel attention module is the LeakyReLU activation function, which is expressed as follows:

number

number

[0020] In another aspect of the present invention, a computer program is further provided that realizes the steps of the above method when the computer program / instructions are executed by a processor.

[0021] By employing the above-described technical means of this invention, two unbalanced noise reduction branches (a first image processing branch and a second image processing branch) exist during network training. In the two branches, the resolution and noise distribution of the input images of the feature extraction module differ from each other. Since the training target is a primitive image related to the input image (related by the division), the super-resolution image noise reduction branch (i.e., the first image processing branch) can learn more high-frequency information, resulting in better noise reduction and motion artifact removal. Furthermore, the self-monitoring training method does not require the specialized collection of high-quality training datasets, and the network model can be trained using only primitive images with a low signal-to-noise ratio. The noise reduction effect of the noise reduction network presented after training convergence is more faithful, and its applications are broader. [Brief explanation of the drawing]

[0022] The following detailed explanation, combined with the following drawings, will allow for a more comprehensive understanding of the principles and various aspects of this application. Note that the proportions of each drawing may differ for illustrative purposes, but this will not affect the understanding of this application. [Figure 1] A schematic diagram of the super-resolution image noise reduction network in one embodiment of the present invention is shown. [Figure 2] A schematic diagram of the model configuration of a self-monitoring rationalized deep learning network in one embodiment of the present invention is shown. [Figure 3] A non-restrictive example of dividing a single image is schematically shown. [Figure 4] A schematic diagram illustrating an example of a network model is shown. [Figure 5] A schematic diagram of the network configuration for one residual block is shown. [Figure 6] A schematic flowchart of the basic method for training the network in one embodiment of the present invention is shown. [Figure 7] This paper schematically shows the results of performing super-resolution noise reduction on low signal-to-noise ratio super-resolution images using the super-resolution noise reduction module in the super-resolution image noise reduction network published after training. [Modes for carrying out the invention]

[0023] In the drawings of this application, features that are identical in structure or similar in function are denoted by the same reference numerals in the drawings. Hereinafter, the technical proposals in embodiments of the present invention will be clearly and completely described in combination with the drawings of embodiments of the present invention. Clearly, the embodiments described below with reference to the drawings are illustrative and used solely to illustrate the present invention and should not be construed as limitations of the present invention. All other embodiments obtained by those skilled in the art without creative work based on the embodiments of the present invention are within the scope of the protection of the present invention.

[0024] Generally, an optical imaging system is used to image a specific biological structure in structural light illumination mode to acquire at least one set of primitive images. For example, an optical imaging system comprises an excitation path and a detection path, the excitation path comprising an excitation objective lens and other optical components for generating excitation light, the excitation beam being able to pass through the excitation objective lens and be emitted in the form of periodic stripe structure light to excite fluorescence on the biological sample, and the detection path comprising a detection objective lens for receiving and detecting the excited fluorescence and other optical components for imaging. In particular, when performing three-dimensional fluorescence microimaging on biological samples, especially living biological samples, multilayer fluorescence images are obtained by continuous scanning sampling in the optical axis direction of the detection objective lens, i.e., in the axial direction, so that each time scanning sampling is completed, the acquired multilayer fluorescence images constitute a fluorescence image stack (also called a "sequence"). Within the scope of this application, optical imaging systems may include, but are not limited to, two-dimensional structured optical systems (2D-SIM), three-dimensional structured optical systems (3D-SIM), lattice optical sheet structured optical systems (LLS-SIM), and incident illumination structured optical systems (GI-SIM).

[0025] The following description of this application uses the processing of two-dimensional fluorescence images as an example, but those skilled in the art should make it clear that similar techniques can be extended to the processing of three-dimensional fluorescence images.

[0026] Within the scope of this Application, it should be noted that the methods and / or method steps and / or neural network models described herein are programmed as programs / instructions or subprograms that can be invoked and executed by a computer. Specific programming methods for programs / instructions and / or subprograms are not discussed herein, and those skilled in the art can implement the relevant functions using any well-known programming software and / or commercial software. Accordingly, when describing the relevant methods and / or method steps hereafter in this Application, it should be understood that they may also be written as computer programs / instructions for invocation and execution by a computer.

[0027] According to one embodiment of the method of the present invention, first, it is necessary to acquire a biofluorescence image (hereinafter abbreviated as "primitive image") captured in structural light illumination mode for a specific biological sample (e.g., a biological structure such as the mitochondrial outer membrane or endoplasmic reticulum). In the context of the present invention, the term "image" can be understood as a two-dimensional matrix that can be processed by a computer. Those skilled in the art should recognize that when capturing a biological sample in structural light illumination mode using an optical imaging system, it is necessary to acquire at least one set of original images in response to the requirements of subsequent super-resolution reconstruction, and that each set of primitive images includes N × M primitive images, where N and M are integers of 2 or greater, for example, N and M are each 3, where N represents the direction in structural light illumination mode and M represents the structural light stripe phase in structural light illumination mode. Using a super-resolution reconstruction algorithm (such as the conventional Wiener algorithm), the N × M primitive images can be reconstructed into a single super-resolution image. Here, the primitive images are preferably primitive images with a low signal-to-noise ratio, and therefore can be easily obtained for the optical imaging system for the biological sample, especially for living biological samples. In the context of this application, a low signal-to-noise ratio (LMI) primitive image does not need to be defined within an explicit LMI range; in practice, the term “low signal-to-noise ratio” means that the LMI of the input image used to train a neural network is less than or equal to the LMI of the input image used to present the corresponding neural network after training is complete. For example, the i-th primitive image in each set of primitive images may be represented by a two-dimensional matrix [2a × 2b]i, where i is a non-zero integer less than N × M, and a and b are integers greater than 1, where 2a represents the number of pixels in the longitudinal direction of the primitive image and 2b represents the number of pixels in the width direction of the primitive image.

[0028] Figure 1 schematically shows the structure of a super-resolution image denoising network. This super-resolution image denoising network processes at least one set of raw images so that at least one super-resolution denoising image can be obtained. The super-resolution image denoising network comprises a super-resolution reconstruction module 100 and a super-resolution denoising module 200. The input to the super-resolution reconstruction module 100 is at least one set of low signal-to-noise ratio raw images (each set of raw images contains N × M raw images), the super-resolution reconstruction module 100 is implemented using a structured light illumination super-resolution reconstruction algorithm well known to those skilled in the art, and the output is at least one low signal-to-noise ratio super-resolution image. The super-resolution noise reduction module 200 is constructed using, for example, a convolutional neural network, a deep neural network, a Transformer, or a neural network based on an attention mechanism, such as a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model. The input to the super-resolution noise reduction module 200 is at least one low signal-to-noise ratio (e.g., at least one set of low signal-to-noise ratio (R&D) images as described above), and the output (or prediction) is at least one super-resolution noise reduction image.

[0029] The inventors of this application train the super-resolution image denoising network described above by simultaneously extracting features from a low signal-to-noise ratio (Lv) original image as input, and further training the network by having one branch feature the Lv original image, while the other branch features a super-resolution denoising image based on the Lv original image using physical prior information constraints such as illumination stripes. This allows the trained denoising network to be reconstructed in super-resolution with higher quality and motion artifacts removed. In the context of this application, the terms "feature extraction" or "perform feature extraction" mean training a knowledge understanding that is familiar to those skilled in the art by referring to a neural network.

[0030] Figure 2 schematically shows the model structure of a self-supervised rationalized deep learning network used to train the super-resolution image denoising network shown in Figure 1.

[0031] In Figure 2, for example, a self-supervised rationalized deep learning network model is constructed using the Python programming language with a deep learning framework based on PyTorch, and the output of the super-resolution image denoising network (shown in Figure 1) is used as the input to the modulation module 300. The modulation module 300 can be designed, for example, to obtain a stripe-modulated image by point multiplying the input super-resolution denoising image with a stripe that has the same direction and phase as the illumination stripe used when imaging a biological sample in structural light illumination mode using an optical imaging system. Alternatively, the modulation module 300 can be designed, for example, to first multiply the input super-resolution denoising image with a stripe that has the same direction and phase as the illumination stripe used when imaging a biological sample in structural light illumination mode using an optical imaging system, and then convolve this with the system point diffusion function of the optical imaging system for capturing a primitive image to obtain a stripe-modulated image. For example, if at least one set of low signal-to-noise ratio raw images is input to the modulation module 300 and outputs at least one super-resolution noise-reduced image via the super-resolution image noise-free network shown in Figure 1, the output of the modulation module 300 may be at least one set of stripe-modulated images with the same number of low signal-to-noise ratios as the raw images.

[0032] For example, if one image is I m、n(r) When representing the stripe intensity at coordinate r, m is selected from an integer range of 1 or greater and less than or equal to M, n is selected from an integer range of 1 or greater and less than or equal to N, and r can represent a two-dimensional (length and width) coordinate or a three-dimensional coordinate, and the stripe (or stripe intensity) mentioned above can be exemplified by the following formula (those skilled in the art should recognize that formulas for other structured lighting stripes are similarly applicable to the technical proposal of this application):

number

number

[0033] Subsequently, these stripe-modulated images can be used as input to the first feature extraction module 410, which is configured to perform feature extraction on the input stripe-modulated images. In other words, the sequence of low signal-to-noise ratio raw image --> super-resolution noise-reduced image --> stripe-modulated image --> first feature extraction module 410 roughly forms the image processing path at the super-resolution noise-reduced image processing (feature extraction) branch during network training.

[0034] Furthermore, at least one set of low signal-to-noise ratio (LRI) original images are used as input to the second feature extraction module 420, which is configured to extract important features from the input LRI original images. In other words, the LRI original images --> second feature extraction module 420 form the image processing path at the LRI noise reduction image processing (feature extraction) branch during network training. The outputs of the first feature extraction module 410 and the second feature extraction module 420 are added together and then used as input to the structure recovery module 430. The structure recovery module 430 is configured to perform noise reduction on the input image and output at least one set of noise-reduced images. The at least one set of noise-reduced images from the output of the structure recovery module 430, at least one set of low signal-to-noise ratio original images, and at least one set of stripe modulation images are added together and averaged to form the network output image of the training network. Simultaneously, the previously acquired training target original image is used as the target of the training network, and backpropagation iterative optimization is performed using a loss function, preferably using an Adam optimizer. At the same time, the above-mentioned at least one set of low signal-to-noise ratio original images is used as the network input image of the training network. In this application, the loss function is the network output image

number

number

[0035] In this application, the super-resolution noise reduction module 200, the first feature extraction module 410, the second feature extraction module 420, and the structure recovery module 430 described above can be constructed using the same network model, for example, a convolutional neural network, a deep neural network, a Transformer, a neural network based on an attention mechanism, or any other suitable trainable and convergent neural network model, including but not limited to U-shaped neural network models, residual neural network models, residual channel attention convolutional neural network models, or Fourier channel attention convolutional neural network models. Of course, those skilled in the art should recognize that these four modules may be constructed using different network models, and at least two of them may be constructed using different network models.

[0036] For example, when training the model of the rationalized deep learning network based on self-monitoring shown in FIG. 2, first, for a specific biological sample (such as a biological structure like the outer mitochondrial membrane, endoplasmic reticulum, etc.), at least one set of original images, for example, at least one set of original images with a low signal-to-noise ratio, is captured in the structured light illumination mode. Then, in order to perform the self-monitoring training of the network, image segmentation is performed on each of the at least one set of original images. For example, image segmentation is performed in a manner of first downsampling and then upsampling, generating at least two sets of images (for example, at least two sets of original images with a low signal-to-noise ratio). Preferably, the luminance distributions of these images are substantially the same, but the noises are independent of each other. For example, the i-th original image [2a×2b] in one set of original images i can be divided so as to have a×b adjacent pixel regions with a size of 2×2 for each pixel region. Those skilled in the art should be clear that depending on the number of divided pixel regions, the size of the pixel region can be other specifications such as 3×2, 3×3, 4×3, 4×2, 4×2, 2×1, etc., as long as the length dimension and width dimension of the pixel region can be evenly divided by the length dimension 2a and width dimension 2b of the original image [2a×2b]. i It should be noted that the above-mentioned adjacent pixel regions mean that there is no overlap between each pixel region and they fill (or occupy) the whole of the original image [2a×2b]. i

[0037] For example, FIG. 3 schematically shows the i-th original image [8×6] in one set of original images i which is divided into 4×3 adjacent pixel regions with a size of 2×2 for each pixel region. Then, one pixel-constituted downsampled image with a size of 4×3 is randomly selected from each pixel region and upsampled using an interpolation method (such as 2D linear or bi-cubic interpolation), obtaining the training input original image [8×6] i-IN for the subsequent network training input, with the subscript " -INThis means that the image will be used as the network training input; the remaining pixels in each pixel region (e.g., 3 pixels) will form the corresponding downsampled image, which will then be upsampled using an interpolation method (e.g., 2D linear or double cubic interpolation) to create the target raw image for training [8×6] for network training. i-TAG It is obtained as such, and the subscript within it is -TAG This means that the image will be used as a network training target.

[0038] In alternative embodiments (not shown), two pixels are randomly selected from each pixel region to construct a downsampled image of size 4×3, which is then upsampled using an interpolation method (e.g., 2D linear or double cubic interpolation) to obtain the training input raw image [8×6] for network training input. i-IN It can also be obtained as a subscript, and among them the subscript " -IN This means that the image will be used as the network training input; the remaining pixels in each pixel region (e.g., two pixels) will form the corresponding downsampled image, which will then be upsampled using an interpolation method (e.g., 2D linear or double cubic interpolation) to obtain the target raw image for training [8×6] for network training. i-TAG It is obtained as such, and the subscript within it is -TAG" means that the image will be used as the network training target. In alternative embodiments not shown, downsampling can also be implemented, for example, by selecting one pixel from each pixel region, then randomly selecting a different number of pixels (e.g., one, two, or any other number less than or equal to the total number of remaining pixels) from the remaining pixels in the same pixel region, and then upsampling using reinterpolation to obtain the corresponding training input or training target raw image for subsequent network training, or the training target or training input raw image for subsequent network training. In alternative embodiments not shown, one or more pixels can be randomly (and without overlap) selected from each pixel region, and then each can be used to obtain the training input or training target raw image for subsequent network training using interpolation upsampling.

[0039] Therefore, using the image segmentation method described above, one set of original images can be divided into one set of training input original images (having N × M training target original images) and one set of training target original images (having N × M training target original images). A structured light illumination super-resolution reconstruction algorithm well known to those skilled in the art (see, for example, the disclosure by Gustafsson, MG et al., Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys J 94, 4957-4970 (2008)) is used to perform super-resolution processing on this set of training input original images to obtain a single super-resolution image of the training input source.

[0040] To meet the requirements for achieving self-monitoring network training, it is necessary to perform multi-faceted amplification on the aforementioned training input primitive images and training target primitive images. For example, amplification can be performed using pixel denoising techniques (see inventor's patent document CN116402681A) or image degeneration techniques (see inventor's patent document CN116721017A or the disclosures "Self-supervised denoising for structured illumination microscopy enables long-term super-resolution live-cell imaging." bioRxiv, 2023-04 by authors Chen, X., Qiao, C., Jiang, T., Liu, J., Meng, Q., Zeng, Y., ... & Wu, J., and "Zero-shot learning enables instant denoising and super-resolution in optical fluorescence microscopy." bioRxiv, 2023-02 by authors Qiao, C., Zeng, Y., Meng, Q., Chen, X., Chen, H., Jiang, T., ... & Dai, Q.). For example, amplification methods may include, but are not limited to, operations such as random clipping, random angular rotation, and image symmetry. For example, a set of original images obtained by splitting a set of original images can be split into a set of training input original images and a set of training target original images. These can then be amplified by an amplification method to obtain L sets of training input original images and L sets of training target original images, where L may be an integer of 2 or more, preferably 10000, 20000, or 30000 or more or less. The L sets of training target original images can be a set of training target original images, and the L sets of training input original images can be a set of training input original images.It should be recognized that the number of training target primitive images in the training target primitive image set and the number of training input primitive images in the training input primitive image set may be the same or different.

[0041] When training the network shown in Figure 2, at least one set of training input primitive images is selected as input from the training input primitive image set and simultaneously input to a first image processing branch consisting of a super-resolution reconstruction module 100, a super-resolution noise reduction module 200, a modulation module 300, and a first feature extraction module 410, and also to an independent second image processing branch consisting of a second feature extraction module 420. The output of the first image processing branch and the output of the second image processing branch are added together and used as input to the structure recovery module 430. The output of the structure recovery module 430, the primitive images input to the first image processing branch, and the primitive images input to the second image processing branch are added together and averaged to obtain the output for network model training. The training target primitive image selected from the training target primitive image set is used as the target for network model training.

[0042] The training input primitive image set and the training target primitive image set can be normalized. Therefore, when the output of the structure recovery module 430 is added to the primitive image input to the first image processing branch and the primitive image input to the second image processing branch and averaged, the output during network model training can be obtained by first adding the intensity of each pixel point and then calculating the average value.

[0043] For example, when training the network shown in Figure 2, in the first image processing branch, L sets of training input raw images (e.g., low signal-to-noise ratio images) are super-reconstructed via the super-resolution reconstruction module 100, and then L super-resolution images of the training input sources are generated via the super-resolution noise reduction module 200. These L super-resolution images of the training input sources are then used to generate at least L sets of stripe-modulated images via the modulation module 300, which are then used as input to the first feature extraction module 410. In the second image processing branch, L sets of training input raw images (e.g., low signal-to-noise ratio images) are directly used as input to the second feature extraction module 420.

[0044] Figure 4 schematically shows a configuration diagram that allows for the construction of a network model. For example, the super-resolution noise reduction module 200, the first feature extraction module 410, the second feature extraction module 420, and the structure recovery module 430 can be constructed using this network model. The network model structure mainly consists of sequentially connected shallow feature extraction submodules, deep feature extraction submodules, and output submodules, with the output of the shallow feature extraction submodule being the input to the deep feature extraction submodule, and the output of the deep feature extraction submodule being the input to the output submodule. The shallow feature extraction submodule has sequentially connected convolutional layers and activation layers (for example, the activation function of the activation layer may be a LeakyReLU activation function, or abbreviated as LeakyReLU activation layer). The input image extracts shallow features sequentially through the convolutional layers and activation layers, and the LeakyReLU activation function can be expressed as follows:

number

[0045] The deep feature extraction submodule generally has p residual groups, where p is an integer greater than or equal to 1. The input of the deep feature extraction submodule and the output of the last residual group of the deep feature extraction submodule are added together to obtain an output image (abbreviated as long jump connection). Each residual group has q residual blocks, where q is an integer greater than or equal to 1. The input of each residual group and the output of the last residual block of that residual group are added together to obtain an output image (abbreviated as long jump connection). In a preferred embodiment, p and q may each be 4. Figure 5 schematically shows the network configuration of one residual block. As shown in Figure 5, each residual block has a series of convolutional layers, an activation layer (for example, the activation function of the activation layer may be a LeakyReLU activation function), and a downstream series of channel attention modules. The input of the residual block and the output of the channel attention module of the residual block are added together to obtain an output image (abbreviated as long jump connection). The channel attention module generally consists of a global averaging pooling layer, a convolutional layer, a LeakyReLU activation layer, another convolutional layer, and a sigmoid activation layer in series. It calculates the weight of each channel in the input image, multiplies the weight by each channel in the input image to obtain the output image, and adds the input of the channel attention module to the output of the sigmoid activation layer in series to obtain the output image (abbreviated as long jump connection). The sigmoid activation function of the sigmoid activation layer can be expressed by the following equation:

number

[0046] In training using the self-monitoring, streamlined deep learning network shown in Figure 3, two unbalanced branches (i.e., a super-resolution noise reduction branch and a primitive image noise reduction branch) work together as training inputs. The former branch is constrained by physical prior information such as illumination stripes, allowing the network training to converge more quickly. This embodies the streamlining of network training, i.e., during deep learning. Furthermore, the post-production super-resolution image noise reduction network can better remove noise in super-resolution images, more effectively reducing reconstruction and motion artifact problems in the super-resolution image reconstruction process. At the same time, it can achieve more accurate and high-quality super-resolution image noise reduction effects under different super-resolution image modes (2D or 3D super-resolution images) and various imaging conditions (various signal-to-noise ratios, imaging speeds, and imaging time lengths).

[0047] Figure 6 schematically shows a flowchart of a basic method for training the network of the embodiment of the present invention. The order of the steps shown in Figure 6 is not restricted to the method of the present invention, and it is clear that those skilled in the art can adjust the order of the steps as needed. Figure 7 schematically shows the results of performing super-resolution noise reduction on a low signal-to-noise ratio super-resolution image using the super-resolution noise reduction module in the super-resolution image denoising network published after training.

[0048] While specific embodiments of the present application have been described in detail here, they are provided solely for interpretive purposes and do not constitute limitations to the scope of the application. Furthermore, those skilled in the art will understand that each embodiment described herein can be used in combination with one another. Various substitutions, modifications, and alterations can be conceived without departing from the spirit and scope of the application.

Claims

1. A method for reducing noise in self-monitoring structured light-illuminated super-resolution microscopic images, The method includes the step of providing a super-resolution image denoising network, the super-resolution image denoising network comprising an upstream super-resolution reconstruction module (100) and a downstream super-resolution noise reduction module (200), wherein the super-resolution reconstruction module (100) is configured to super-reconstruct a raw image captured in a structured light illumination mode using a super-resolution reconstruction algorithm and output a super-resolution reconstructed image, the super-resolution noise reduction module (200) is configured to denoise the super-resolution reconstructed image and output a super-resolution noise reduction image, the super-resolution noise reduction module (200) is constructed using a neural network, and the super-resolution image denoising network is trained in the following manner before deployment: The present invention provides at least one set of primitive images taken of a biological structure in the structural light illumination mode, each set of primitive images comprising N × M primitive images, where N and M are integers of 2 or more, where N represents the direction in the structural light illumination mode, and M represents the structural light stripe phase in the structural light illumination mode. In a method that first downsamples and then upsamples, at least one set of the original images is divided into at least one set of training input original images and at least one set of training target original images. At least one set of the training input primitive images and at least one set of the training target primitive images are amplified to generate a training input primitive image set and a training target primitive image set. A training network model of a self-monitoring deep learning network is constructed for the super-resolution image noise reduction network, and the training network model of the self-monitoring deep learning network comprises a first feature extraction module (410), a second feature extraction module (420), a structure recovery module (430), and a modulation module (300), wherein the first feature extraction module (410), the second feature extraction module (420), and the structure recovery module (430) are constructed using a neural network, the first feature extraction module (410) and the second feature extraction module (420) are configured to perform feature extraction on an input image, and the modulation module (300) is configured to perform point multiplication on a stripe modulated image by a stripe with the same direction and phase as the illumination stripe used during structural light illumination mode shooting of the input image, and output an image after stripe modulation. A method for reducing noise in self-monitoring structurally illuminated super-resolution microscopic images, characterized in that, when training a network model, the super-resolution reconstruction module (100), the super-resolution noise reduction module (200), the modulation module (300), and the first feature extraction module (410) sequentially configure a first image processing branch, and the second feature extraction module (420) configures an independent second image processing branch, inputs the training input primitive image selected from the training input primitive image set to the first image processing branch and the second image processing branch respectively, adds the outputs of the first image processing branch and the second image processing branch and uses them as input to the structure recovery module (430), averages the output of the structure recovery module (430) with the primitive image input to the first image processing branch and the primitive image input to the second image processing branch and uses this as the output during network model training, and uses the training target primitive image selected from the training target primitive image set as the target during network model training.

2. A method for reducing noise in self-monitoring structured optically illuminated super-resolution microscopic images, characterized in that, in a method of downsampling first and then upsampling, each of the N × M original images in a set of original images is divided into adjacent and same-sized pixel regions, at least one pixel is randomly selected from each pixel region, interpolation is used to generate the training input original image corresponding to that original image, and at least one pixel is selected from the remaining pixels in each pixel region, interpolation is used to generate the training target original image corresponding to that original image, thereby generating a set of training input original images and a set of training target original images.

3. The noise reduction method for self-monitoring structured light-illuminated super-resolution microscopic images according to claim 2, characterized in that the size of the pixel region is 2 x 2, and in a method of downsampling first and then upsampling, one pixel is randomly extracted from each of the pixel regions and interpolation is used to generate the training input primitive image.

4. The noise reduction method for self-monitoring structural light illumination super-resolution microscopic images according to claim 3, characterized in that at least one set of original images captured with respect to the biological structure in the structural light illumination mode are original images with a low signal-to-noise ratio.

5. The noise reduction method for self-monitoring structure light-illuminated super-resolution microscopic images according to claim 4, characterized in that the amplification method includes random clipping, random angular rotation, or image symmetric operation.

6. The noise reduction method for self-monitoring structural light illumination super-resolution microscopic images according to claim 1, characterized in that the modulation module (300) is arranged to first point multiply the input image by a stripe having the same direction and phase as the illumination stripe used when the structural light illumination mode was captured, then convolve with the system point diffusion function of the optical imaging system for capturing the original image, and output the stripe-modulated image.

7. The noise reduction method for self-monitoring structured optically illuminated super-resolution microscopic images according to claim 1, characterized in that the training input primitive image set and the training target primitive image set are subjected to normalization processing.

8. The loss function of the aforementioned training network model is the network output image [Math 12] It consists of two parts: the averaged absolute error (MAE) and the structural similarity (SSIM) between the target image (X), and is expressed by the following formula: [Number 13] The noise reduction method for self-monitoring structural light-illuminated super-resolution microscopic images according to claim 7, characterized in that λ is a weight of structural similarity loss, and the value of λ is 0 or greater.

9. The noise reduction method for self-monitoring structured optically illuminated super-resolution microscopic images according to claim 1, characterized in that the neural network includes a convolutional neural network, a deep neural network, a transformer, and a neural network based on an attention mechanism.

10. The noise reduction method for self-monitoring structure light-illuminated super-resolution microscopic images according to claim 9, characterized in that the super-resolution noise reduction module (200), the first feature extraction module (410), the second feature extraction module (420), and / or the structure recovery module (430) can be constructed using the same neural network or different neural networks.

11. The noise reduction method for self-monitoring structure light-illuminated super-resolution microscopic images according to claim 10, characterized in that the neural network model for constructing the super-resolution noise reduction module (200), the first feature extraction module (410), the second feature extraction module (420), and / or the structure recovery module (430) is a training-convergent neural network model, and includes a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.

12. Any one of the super-resolution noise reduction module (200), the first feature extraction module (410), the second feature extraction module (420), and the structure recovery module (430) comprises a shallow feature extraction submodule, a deep feature extraction submodule, and an output submodule, wherein the shallow feature extraction submodule has a convolutional layer and an activation layer, the deep feature extraction submodule has p residual groups, where p is an integer of 1 or more, and adds the input of the deep feature extraction submodule to the output of the last residual group of the deep feature extraction submodule, each residual group has q residual blocks, where q is an integer of 1 or more, and adds the input of each residual group to the output of the last residual group A noise reduction method for self-monitoring structured optically illuminated super-resolution microscopic images according to claim 1, characterized in that the outputs of the residual blocks are added, each residual block having a series-connected convolutional layer and activation layer, and a series-connected channel attention module located downstream, the input of the residual block and the output of the channel attention module of the residual block are added, the channel attention module is configured in series with a global averaging pooling layer, a convolutional layer, an activation layer, a convolutional layer, and an activation layer, calculates the weight of each channel of the input image, multiplies the weight by each channel of the input image to obtain an output image, and is configured to add the input of the channel attention module and the output of the last activation layer of the channel attention module.

13. The method for reducing noise in a self-monitoring structure light-illuminated super-resolution microscopic image according to claim 12, characterized in that the shallow feature extraction submodule has one, two or more pairs of sequentially arranged convolutional layers and activation layers, and / or the residual block has one, two or more pairs of sequentially arranged convolutional layers and activation layers.

14. In the shallow feature extraction submodule and / or the residual block and / or the output submodule, the activation function of the activation layer is the LeakyReLU activation function, which is expressed as follows: [Number 14] The noise reduction method for self-monitoring structure light illumination super-resolution microscopic images according to claim 13, characterized in that α is a leakage value, and α is greater than or equal to 0 and less than 1, or α = 0.

2.

15. In the channel attention module, the activation function of the upstream activation layer is the LeakyReLU activation function, which is expressed as follows: [Number 15] α is the leakage value, and α is greater than or equal to 0 and less than 1, or α = 0.

2. The activation function of the downstream activation layer is the Sigmoid activation function, which is expressed as follows: [Number 16] The noise reduction method for self-monitoring structured optically illuminated super-resolution microscopic images according to feature 14.

16. A computer program characterized in that, when the computer program / instruction is executed by a processor, a step of the method according to any one of claims 1 to 15 is realized.

Images