Self-supervised structured illumination microscopy image denoising

Abstract

The application discloses self-supervised structured light illumination super-resolution microscopic image denoising, comprising: providing a denoising network with a super-resolution reconstruction module and a super-resolution denoising module, the super-resolution reconstruction module is configured to perform super-resolution reconstruction on an original image to output a super-resolution reconstructed image, the super-resolution denoising module is configured to perform denoising processing on the super-resolution reconstructed image to output a super-resolution denoised image, the super-resolution denoising module is built by using a neural network, and the denoising network is trained in the following manner before deployment: providing at least one set of original images; the original images are split into training input original images and training target original images in the manner of first downsampling and then upsampling; the training input original images and the training target original images are augmented to generate a set of training input original images and a set of training target original images; and a training network model of a self-supervised deep learning network is established for the super-resolution image denoising network.

Description

TECHNICAL FIELD

[0001] The present application generally relates to a method for denoising an image taken by using a structured light illumination pattern by rationalizing deep learning based on self-supervision using a neural network. BACKGROUND

[0002] The invention of optical microscope enables people to observe the life phenomena inside tiny cells, which greatly promotes the development of life science. However, generally speaking, the traditional optical microscope is restricted by the optical diffraction limit and cannot clearly see the details of the structure below 200 nm. In recent years, from the hardware aspect, various super-resolution fluorescence microscopic imaging techniques have been invented, which have pushed the resolution of the imaging system to the level of 10-100 nm. Among them, the structured illumination microscopy (SIM) technique is an ideal choice for studying the dynamic interaction between organelles in living cells, and has a high imaging speed under the premise of maintaining low photobleaching and phototoxicity. However, the fluorescent protein has some defects, such as limited number of photons, easy bleaching or quenching, etc., which limit the acquisition of long-term, high-quality super-resolution fluorescence images.

[0003] With the rapid iteration of hardware technologies such as computers and graphics cards, significant progress has been made in deep learning-based neural network algorithms from a software perspective. In recent years, supervised learning-based image denoising neural networks, such as Content-aware image restoration (CARE) and Deep Fourier channel attention network (DFCAN), have been proposed. Their superior denoising performance greatly improves image quality, making long-term, high signal-to-noise ratio (SNR) super-resolution imaging of live cells possible. Supervised denoising networks typically require the collection of a large number of image data pairs with consistent structure and low and high SNR as the training set, using low SNR data as the input image and high SNR data as the output image to train the network model. However, in real cell samples, there are always many organelles with strong background signals, weak fluorescence signals, or fast-moving structures, making it difficult to collect high SNR datasets that meet the requirements. These factors limit the applicability of supervised denoising networks. Therefore, researchers have further developed self-supervised denoising neural networks (which can also be simply referred to as networks) to reduce the requirements for training sets, such as Noise2void, Blind2unblind, and Neighbor2Neighbor. The training logic of self-supervised denoising networks generally involves splitting an image into two or more parts, selecting one as the input image, and using the other as the target image to train the denoising network model. With enough images provided for training, it is possible to achieve results close to those of supervised denoising.

[0004] However, when existing self-supervised denoising networks are trained and published using structured light super-resolution images for denoising, reconstruction and motion artifacts in the reconstructed images cannot be effectively removed, and the image resolution will also decrease. In particular, good denoising results cannot be obtained under conditions where the signal-to-noise ratio of the original data is low. Summary of the Invention

[0005] To address the shortcomings of the aforementioned denoising algorithms, this application aims to propose a self-supervised, rationalized deep learning-based image denoising method. In this method, the obtained low signal-to-noise ratio image is used as the input to two unbalanced branches during network training: a super-resolution denoising branch and an original image denoising branch. By leveraging physical prior information constraints such as illumination fringes, noise in the super-resolution image can be removed more effectively, and reconstruction and motion artifacts during the super-resolution image reconstruction process can be reduced more efficiently. Simultaneously, it can provide higher image resolution and achieve more accurate and higher-quality super-resolution image denoising results under different super-resolution imaging modalities (two-dimensional or three-dimensional super-resolution imaging) and various imaging conditions (various signal-to-noise ratios, imaging speeds, and imaging durations).

[0006] According to an aspect of the present application, there is provided a self-supervised rationalized deep learning image denoising method or a self-supervised structured illumination super-resolution microscopy image denoising, which comprises:

[0007] providing a super-resolution image denoising network, which generally comprises an upstream super-resolution reconstruction module and a downstream super-resolution denoising module, the super-resolution reconstruction module is configured to perform super-resolution reconstruction on an original image taken in a structured illumination mode by using a super-resolution reconstruction algorithm to output a super-resolution reconstructed image, the super-resolution denoising module is configured to perform denoising processing on the super-resolution reconstructed image to output a super-resolution denoised image, the super-resolution denoising module is built by using a neural network, and the super-resolution image denoising network is trained before deployment by using the following method:

[0008] providing at least one set of original images taken in a structured illumination mode for biological structures, wherein each set of original images comprises N×M original images, N and M are integers greater than or equal to 2, N represents the direction in the structured illumination mode, and M represents the structured light fringe phase in the structured illumination mode;

[0009] splitting the at least one set of original images into at least one set of training input original images and at least one set of training target original images in a manner of first downsampling and then upsampling;

[0010] augmenting the at least one set of training input original images and the at least one set of training target original images to generate a set of training input original images and a set of training target original images;

[0011] establishing a training network model of a self-supervised deep learning network for the super-resolution image denoising network, wherein the training model of the self-supervised deep learning network comprises a first feature extraction module, a second feature extraction module, a structure recovery module, and a modulation module, the first feature extraction module, the second feature extraction module, and the structure recovery module are built by using a neural network, the first feature extraction module and the second feature extraction module are respectively configured to perform feature extraction on the input image, the modulation module is configured to perform point multiplication on the input image and the same direction and phase of the illumination fringe when taking the structured illumination mode to output a fringe-modulated image,

[0012] In training the network model, the super-resolution reconstruction module, the super-resolution denoising module, the modulation module, and the first feature extraction module sequentially constitute a first image processing branch, and the second feature extraction module constitutes an independent second image processing branch, a training input original image selected from a training input original image set is input into the first image processing branch and the second image processing branch, the output of the first image processing branch and the output of the second image processing branch are added to serve as the input of the structure recovery module, and the output of the structure recovery module is added to the original image input into the first image processing branch and the original image input into the second processing branch and then averaged to serve as the output in training the network model, and a training target original image selected from a training target original image set serves as the target in training the network model.

[0013] Optionally, in the manner of first downsampling and then upsampling, each of the N×M original images in the set of original images is split into pixel regions adjacent to each other and of the same size, at least one pixel is randomly selected from each pixel region and interpolation is used to generate a training input original image corresponding to the original image, and for the remaining pixels in each pixel region, at least one pixel is selected therefrom and interpolation is used to generate a training target original image corresponding to the original image, to generate a set of training input original images and a set of training target original images.

[0014] Optionally, the size of the pixel region is 2×2, and in the manner of first downsampling and then upsampling, one pixel is randomly extracted from each pixel region and interpolation is used to generate a training input original image.

[0015] Optionally, the at least one set of original images taken in the structured light illumination mode for biological structures are low signal-to-noise ratio original images.

[0016] Optionally, the augmentation manner includes but is not limited to random cropping, random angle rotation, or image symmetry operation.

[0017] Optionally, the modulation module is configured to multiply the input image by the same direction and phase of the illumination fringe used when taking the original image in the structured light illumination mode, and then convolve the system point spread function of the optical imaging system used to take the original image to output a fringe-modulated image.

[0018] Optionally, the training input original image set and the training target original image set are normalized.

[0019] Optionally, the loss function of the training network model is calculated by network output image The average absolute error (MAE) between the target image (X) and the structure similarity (SSIM) are two parts, which can be expressed by the formula:

[0020]

[0021] Wherein, λ is the weight of the structure similarity loss, for example, the value of λ is greater than or equal to 0.

[0022] Optionally, the neural network includes but is not limited to convolutional neural network, deep neural network, Transformer, attention mechanism based neural network.

[0023] Optionally, the super-resolution denoising module, the first feature extraction module, the second feature extraction module and / or the structure recovery module can be built with the same neural network or different neural networks.

[0024] Optionally, the neural network model used to build the super-resolution denoising module, the first feature extraction module, the second feature extraction module and / or the structure recovery module is a training convergent neural network model, including but not limited to U-shaped neural network model, residual neural network model, residual channel attention convolutional neural network model, or Fourier channel attention convolutional neural network model.

[0025] Optionally, any one of the super-resolution denoising module, the first feature extraction module, the second feature extraction module and the structure recovery module includes a shallow feature extraction submodule, a deep feature extraction submodule and an output submodule connected in series, the shallow feature extraction submodule includes a convolutional layer and an activation layer connected in series, the deep feature extraction module includes p residual groups, where p is an integer greater than or equal to 1, the input of the deep feature extraction module is added to the output of the last residual group of the deep feature extraction module, each residual group includes q residual blocks, where q is an integer greater than or equal to 1, the input of each residual group is added to the output of the last residual block of the residual group, each residual block includes a convolutional layer and an activation layer connected in series and a channel attention module connected downstream in series, the input of the residual block is added to the output of the channel attention module of the residual block, the channel attention module is composed of a global average pooling layer, a convolutional layer, an activation layer, a convolutional layer and an activation layer connected in series, configured to calculate the weight of each channel of the input image and multiply the weight with each channel of the input image respectively to obtain the output image, and the input of the channel attention module is added to the output of the last activation layer of the channel attention module.

[0026] Optionally, one pair, two pairs or more pairs of convolutional layers and activation layers connected in series are included in the shallow feature extraction submodule; and / or one pair, two pairs or more pairs of convolutional layers and activation layers connected in series are included in the residual block.

[0027] Optionally, in the shallow feature extraction submodule and / or in the residual block and / or in the output submodule, the activation function of the activation layer is the LeakyReLU activation function, which can be expressed as follows:

[0028]

[0029] Where α is the leakage value, which is greater than or equal to 0 and less than 1, especially α = 0.2.

[0030] Optionally, in the channel attention module, the activation function of the upstream activation layer is the LeakyReLU activation function, which can be expressed as follows:

[0031]

[0032] Where α is the leakage value, α is greater than or equal to 0 and less than 1, specifically α = 0.2, and

[0033] The activation function of the downstream activation layer is the Sigmoid activation function, which can be expressed as follows:

[0034]

[0035] According to another aspect of this application, a computer program product is also provided, comprising a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the aforementioned method.

[0036] By employing the aforementioned technical methods of this application, because there are two unbalanced denoising branches (a first image processing branch and a second image processing branch) during network training, the resolution and noise distribution of the input images for the feature extraction modules differ between the two branches. Since the training target is the original image associated with the input image (associated due to the splitting), the super-resolution image denoising branch (i.e., the first image processing branch) can learn more high-frequency information, resulting in better denoising and motion artifact elimination. Furthermore, the self-supervised training method does not require the collection of high-quality training datasets; only original images with low signal-to-noise ratios are needed to train the network model. The denoising network released after training convergence exhibits more faithful denoising performance and has wider applications. Attached Figure Description

[0037] A more comprehensive understanding of the principles and aspects of this application will be gained from the detailed description below, in conjunction with the accompanying drawings. It should be noted that the scale of the drawings may vary for clarity, but this will not affect the understanding of this application. In the drawings:

[0038] Figure 1A structural diagram of a super-resolution image denoising network according to an embodiment of the present application is schematically shown;

[0039] Figure 2 A model structural diagram of a rationalization deep learning network based on self-supervision according to an embodiment of the present application is schematically shown;

[0040] Figure 3 A non-limiting example of splitting an image is schematically shown;

[0041] Figure 4 An example structural diagram that can be used to build a network model is schematically shown;

[0042] Figure 5 A network structural diagram of a residual block is schematically shown;

[0043] Figure 6 A basic method flowchart of training a network according to an embodiment of the present application is schematically shown; and

[0044] Figure 7 Results of super-resolution denoising processing of low signal-to-noise ratio super-resolution images by a super-resolution denoising module in a post-training released super-resolution image denoising network are schematically shown. DETAILED DESCRIPTION

[0045] In the drawings of the present application, features with the same structure or similar functions are denoted by the same reference numerals. The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings of the embodiments of the present application. Obviously, the embodiments described below by reference to the drawings are exemplary and are used only to explain the present application, and cannot be explained as a limitation on the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0046] Generally, a certain biological structure is imaged with an optical imaging system in a structured light illumination mode to obtain at least one set of raw images. For example, the optical imaging system can include an excitation light path including an excitation objective and other optical components for generating excitation light, through which excitation light beams can be emitted in a periodic fringe structured light pattern to excite fluorescence on a biological sample, and a detection light path including a detection objective and other optical components for imaging to receive and detect the excited fluorescence. In particular, when performing three-dimensional fluorescence microscopy imaging on a biological sample, especially a live biological sample, a multi-layer fluorescence image is continuously scanned along the optical axis direction of the detection objective, i.e. axially, so that after each scanning, a stack of fluorescence images is obtained, which can also be referred to as a “sequence”. In the context of the present application, the optical imaging system can include, but is not limited to, a two-dimensional structured light system (2D-SIM), a three-dimensional structured light system (3D-SIM), a lattice light sheet structured light system (LLS-SIM), and a grazing incidence illumination structured light system (GI-SIM).

[0047] The following description of the present application is only exemplified by the processing of two-dimensional fluorescence raw images, but it is clear to those skilled in the art that similar techniques can be extended to the processing of three-dimensional fluorescence raw images.

[0048] It should be noted that in the context of the present application, the methods and / or method steps and / or neural network models described herein can be programmed as programs / instructions or subprograms that are invoked and executed by a computer. The specific programming of the programs / instructions and / or subprograms is not discussed in the present application, and those skilled in the art can implement the related functions using any well-known programming software and / or commercial software. Therefore, the following description of the present application when describing the related methods and / or method steps should be understood as they can also be written as computer programs / instructions to be invoked and executed by a computer.

[0049] According to one embodiment of the method of the present application, it is first required to obtain the original fluorescence images (hereinafter referred to as original images) taken in structured illumination mode for a specific biological sample (e.g. biological structure of mitochondrial outer membrane, endoplasmic reticulum, etc.). In the context of the present application, the term related to "image" can be understood as a two-dimensional matrix that can be processed by a computer. It should be clear to those skilled in the art that when taking a biological sample using an optical imaging system in structured illumination mode, at least one set of original images is required according to the requirements of subsequent super-resolution reconstruction, wherein each set of original images includes N x M original images, N and M are integers greater than or equal to 2, for example, N and M can be 3 respectively, N represents the direction in the structured illumination mode, and M represents the phase of the structured light fringe in the structured illumination mode. Using a super-resolution reconstruction algorithm (e.g. traditional Wiener algorithm), N x M original images can be reconstructed into one super-resolution image. Here, the original images are preferably low signal-to-noise ratio original images, so that the optical imaging system is easy to obtain for biological samples, especially for living biological samples. In the context of the present application, the low signal-to-noise ratio original images are not necessarily defined in a specific signal-to-noise ratio range. In fact, the term "low signal-to-noise ratio" means that the signal-to-noise ratio of the input image used when the neural network is trained is equal to or lower than the signal-to-noise ratio of the input image when the corresponding neural network is trained and released for use. For example, the i-th original image in each set of original images can be represented by a two-dimensional matrix [2a x 2b] i where i is a non-zero integer less than N x M, a and b are integers greater than 1 and 2a represents the number of pixels in the length direction of the original image, and 2b represents the number of pixels in the width direction of the original image.

[0050] Figure 1 The structural diagram of the super-resolution image denoising network is schematically shown. Using the super-resolution image denoising network, at least one set of original images is processed to obtain at least one super-resolution denoising image. The super-resolution image denoising network generally includes a super-resolution reconstruction module 100 and a super-resolution denoising module 200, the input of the super-resolution reconstruction module 100 is at least one set of low signal-to-noise ratio original images (each set of original images includes N x M original images) mentioned above, the super-resolution reconstruction module 100 is implemented using a structured illumination super-resolution reconstruction algorithm well known to those skilled in the art, and the output is at least one low signal-to-noise ratio super-resolution image. The super-resolution denoising module 200 can be built using a convolutional neural network, a deep neural network, a Transformer, a neural network based on an attention mechanism, etc., such as a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model. The input of the super-resolution denoising module 200 is at least one low signal-to-noise ratio original image (e.g. at least one set of low signal-to-noise ratio original images mentioned above), and the output (or prediction) is at least one super-resolution denoising image.

[0051] The inventors of the present application found that, when training the super-resolution image denoising network described above, if the low signal-to-noise ratio original image as input is simultaneously input for feature extraction in two independent branches and further training, one of the branches extracts features from the low signal-to-noise ratio original image, and the other branch extracts features from the super-resolution denoising image based on the low signal-to-noise ratio original image using physical prior information constraints such as illumination stripes and further. The trained denoising network can enable super-resolution reconstruction and remove motion artifacts with higher quality. In the context of the present application, the term "feature extraction" or "performing feature extraction" is understood by those skilled in the art with reference to the knowledge familiar to neural network training.

[0052] In order to train the super-resolution image denoising network shown in Figure 1 , the output of the super-resolution image denoising network shown in Figure 2 The model structure diagram of the self-supervised rationalized deep learning network is schematically shown.

[0053] In Figure 2 , for example, a self-supervised rationalized deep learning network model is built using a PyTorch-based deep learning framework and using a Python programming language, in which the output of the super-resolution image denoising network described above (such as shown in Figure 1 ) is input to the modulation module 300. The modulation module 300 can be designed, for example, to point-multiply the input super-resolution denoising image with the same direction and phase of the illumination stripes used when imaging the biological sample in the structured light illumination mode using the optical imaging system to obtain a stripe-modulated image. Alternatively, the modulation module 300 can be designed, for example, to first point-multiply the input super-resolution denoising image with the same direction and phase of the illumination stripes used when imaging the biological sample in the structured light illumination mode using the optical imaging system, and then convolve with the system point spread function of the optical imaging system used to capture the original image to obtain a stripe-modulated image. For example, if the super-resolution image denoising network shown in Figure 1 outputs at least one super-resolution denoising image as input to the modulation module 300, the output of the modulation module 300 can be a stripe-modulated image with the same number of groups as the at least one group of low signal-to-noise ratio original images.

[0054] For example, if I m,n(r) represents the fringe intensity at coordinate r, where m is selected from an integer range greater than or equal to 1 and less than or equal to M, n is selected from an integer range greater than or equal to 1 and less than or equal to N, and r can represent 2-dimensional (length and width direction) coordinates or 3-dimensional coordinates, the above-mentioned fringe (or fringe intensity) can be exemplarily represented by the following formula (other structured illumination fringe formulae are also applicable to the technical solutions of the present application, which should be clear to those skilled in the art):

[0055]

[0056] wherein k n is the fringe vector of the nth direction, is the fringe phase of the image center of the mth phase, MD is the fringe modulation (generally taking the theoretical value 1),

[0057] Therefore, the fringe-modulated image F m,n (r) can be obtained by using the following formula

[0058]

[0059] wherein F SR (r) is the super-resolution image after network denoising (for example, the super-resolution image after denoising by the super-resolution denoising module 200), and PSF is the system point spread function.

[0060] Then, these fringe-modulated images can be used as the input of the first feature extraction module 410, and the first feature extraction module 410 is configured to perform feature extraction on the input fringe-modulated images. That is, the low signal-to-noise ratio original image -> super-resolution denoised image -> fringe-modulated image -> first feature extraction module 410 roughly forms the image processing path in the super-resolution denoised image processing (feature extraction) branch during network training.

[0061] In addition, at least one set of low SNR original images is input into the second feature extraction module 420, and the second feature extraction module 420 is configured to extract key features in the input low SNR original images. That is, the low SNR original images -> the second feature extraction module 420 form an image processing path in the original image denoising image processing (feature extraction) branch of the network training. The outputs of the first feature extraction module 410 and the second feature extraction module 420 are added together and input into the structure recovery module 430, and the structure recovery module 430 is configured to denoise the input images and output at least one set of denoised images. The at least one set of denoised images output by the structure recovery module 430 is added to and averaged with the at least one set of low SNR original images and the at least one set of stripe-modulated images, and is used as the network output image of the training network, while the target original image obtained in advance is used as the target of the training network, and the loss function is used for back propagation and iterative optimization, preferably using the Adam optimizer for back propagation and iterative optimization. At the same time, the at least one set of low SNR original images is used as the network input image of the training network. In this application, the loss function is composed of the mean absolute error (MAE) between the network output image and the target image (X) and the structural similarity (SSIM), which can be expressed by the formula:

[0062]

[0063] wherein λ is the weight of the structural similarity loss, for example, λ is greater than or equal to 0, and is preferably between 0 and 1. During the training process, the initial learning rate can be set to 1x10 -4 The training batch size of each feature extraction module and the structure recovery module is Nxn' (n' is a positive integer), the input channel number is M, the training batch size of the super-resolution image denoising module is n', the input channel number is 1, the channel number of the intermediate hidden layer is generally 64, the training batch size of the final output denoised original image of the network is Nxn', and the output channel number is M. In the context of this application, the addition or addition and averaging between images is based on the common understanding of those skilled in the art. For example, the addition between images can be understood as the direct addition of the intensities of the sub-pixels of each image, and the addition and averaging can be understood as the direct addition of the intensities of the sub-pixels of each image, and then dividing by the number of images to obtain the average.

[0064] ​In the present application, the above super-resolution denoising module 200, the first feature extraction module 410, the second feature extraction module 420, and the structure recovery module 430 can be built with the same network model, for example, built with a convolutional neural network, a deep neural network, a Transformer, a neural network based on an attention mechanism, etc., or built with other suitable neural network models that can be trained to converge, including but not limited to, for example, a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model. Of course, it should be clear to those skilled in the art that the four modules can also be built with different network models or at least two of them are built with different network models.

[0065] For example, when training a model based on a self-supervised rationalization deep learning network as shown in Figure 2 , first, at least one set of original images, for example, at least one set of low signal-to-noise ratio original images, is taken in a structured light illumination mode for a specific biological sample (such as a biological structure of the outer membrane of mitochondria, endoplasmic reticulum, etc.). Then, in order to perform self-supervised training of the network, image splitting is performed on each original image in the at least one set of original images, for example, in a manner of first downsampling and then upsampling, to generate at least two sets of images (for example, at least two sets of low signal-to-noise ratio original images), preferably the brightness distributions of these images are approximately the same but the noises are independent of each other. For example, the i-th original image in the set of original images [2a x 2b] i can be split into a×b pixel regions adjacent to each other, and the size of each pixel region can be 2x2. It should be clear to those skilled in the art that depending on the number of split pixel regions, the size of the above pixel region can also be 3x2, 3x3, 4x3, 4x2, 4x1, 2x1, etc. other specifications, as long as the length dimension and the width dimension of the pixel region can be divided by the length dimension 2a and the width dimension 2b of the original image [2a x 2b] i respectively. It should be clear that the above pixel regions adjacent to each other means that there is no overlap between the pixel regions and the entire original image [2a x 2b] i is fully covered (or occupied).

[0066] For example, Figure 3 illustrates the i-th original image in the set of original images [8 x 6] i which is split into 4x3 pixel regions adjacent to each other, and the size of each pixel region is 2x2. Then, 1 pixel is randomly selected from each pixel region to form a down-sampled image with a size of 4x3, and an interpolation method (such as two-dimensional linear or bicubic interpolation) is used for up-sampling to obtain a training input original image [8 x 6] i-INwhere the subscript "-IN" means that the image is for network training input; and the remaining pixels (e.g., 3 pixels) in each pixel region are up-sampled using interpolation (e.g., two-dimensional linear or bicubic interpolation) to form a corresponding down-sampled image, which is then up-sampled to form a training input original image [8 x 6] for subsequent network training. i-TAG where the subscript "-IN" means that the image is for network training input; and the remaining pixels (e.g., 3 pixels) in each pixel region are up-sampled using interpolation (e.g., two-dimensional linear or bicubic interpolation) to form a corresponding down-sampled image, which is then up-sampled to form a training input original image [8 x 6] for subsequent network training. TAG where the subscript "-IN" means that the image is for network training input; and the remaining pixels (e.g., 3 pixels) in each pixel region are up-sampled using interpolation (e.g., two-dimensional linear or bicubic interpolation) to form a corresponding down-sampled image, which is then up-sampled to form a training input original image [8 x 6] for subsequent network training.

[0067] In an alternative embodiment not shown, 2 pixels can also be randomly selected from each pixel region to form a down-sampled image of size 4 x 3, which is then up-sampled using interpolation (e.g., two-dimensional linear or bicubic interpolation) to form a training input original image [8 x 6] for subsequent network training input. i-IN where the subscript "-IN" means that the image is for network training input; and the remaining pixels (e.g., 3 pixels) in each pixel region are up-sampled using interpolation (e.g., two-dimensional linear or bicubic interpolation) to form a corresponding down-sampled image, which is then up-sampled to form a training input original image [8 x 6] for subsequent network training. -IN where the subscript "-IN" means that the image is for network training input; and the remaining pixels (e.g., 3 pixels) in each pixel region are up-sampled using interpolation (e.g., two-dimensional linear or bicubic interpolation) to form a corresponding down-sampled image, which is then up-sampled to form a training input original image [8 x 6] for subsequent network training. i-TAG where the subscript "-IN" means that the image is for network training input; and the remaining pixels (e.g., 3 pixels) in each pixel region are up-sampled using interpolation (e.g., two-dimensional linear or bicubic interpolation) to form a corresponding down-sampled image, which is then up-sampled to form a training input original image [8 x 6] for subsequent network training. -TAG where the subscript "-IN" means that the image is for network training input; and the remaining pixels (e.g., 3 pixels) in each pixel region are up-sampled using interpolation (e.g., two-dimensional linear or bicubic interpolation) to form a corresponding down-sampled image, which is then up-sampled to form a training input original image [8 x 6] for subsequent network training.

[0068] Thus, using the image splitting method described above, a set of original images can be split into a set of training input original images (having N x M training input original images) and a set of training target original images (having N x M training target original images). The set of training input original images is super-resolved using a structured illumination super-resolution reconstruction algorithm well known to those skilled in the art (see, for example, Gustafsson, M.G., et al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys J 94, 4957-4970 (2008)) to obtain a training input original super-resolved image.

[0069] In order to meet the requirements of implementing self-supervised network training, the number of the training input raw images and the training target raw images needs to be augmented. For example, the augmentation can be performed by using pixel rearrangement technique (see the inventor's patent publication CN116402681A) or image re-degradation technique (see the inventor's patent publication CN116721017A or see the publication by Chen, X., Qiao, C., Jiang, T., Liu, J., Meng, Q., Zeng, Y.,... & Wu, J. Self-supervised denoising for structured illumination microscopy enables long-term super-resolution live-cell imaging. bioRxiv, 2023-04, or the publication by Qiao, C., Zeng, Y., Meng, Q., Chen, X., Chen, H., Jiang, T.,... & Dai, Q. Zero-shot learning enables instant denoising and super-resolution in optical fluorescence microscopy. bioRxiv, 2023-02). For example, the augmentation can include but not limited to random cropping, random angle rotation, image symmetry, etc. For example, a set of raw images can be split into a set of training input raw images and a set of training target raw images, which in turn can be augmented into L sets of training input raw images and L sets of training target raw images by using the augmentation method, where L can be an integer greater than or equal to 2, preferably L is 10000, 20000 or 30000 or larger or smaller. The L sets of training target raw images can be used as a training target raw image set, and the L sets of training input raw images can be used as a training input raw image set. It should be clear that the number of groups of training target raw images in the training target raw image set and the number of groups of training input raw images in the training input raw image set can be the same or different.

[0070] In order to meet the requirements of implementing self-supervised network training, the number of the training input raw images and the training target raw images needs to be augmented. For example, the augmentation can be performed by using pixel rearrangement technique (see the inventor's patent publication CN116402681A) or image re-degradation technique (see the inventor's patent publication CN116721017A or see the publication by Chen, X., Qiao, C., Jiang, T., Liu, J., Meng, Q., Zeng, Y.,... & Wu, J. Self-supervised denoising for structured illumination microscopy enables long-term super-resolution live-cell imaging. bioRxiv, 2023-04, or the publication by Qiao, C., Zeng, Y., Meng, Q., Chen, X., Chen, H., Jiang, T.,... & Dai, Q. Zero-shot learning enables instant denoising and super-resolution in optical fluorescence microscopy. bioRxiv, 2023-02). For example, the augmentation can include but not limited to random cropping, random angle rotation, image symmetry, etc. For example, a set of raw images can be split into a set of training input raw images and a set of training target raw images, which in turn can be augmented into L sets of training input raw images and L sets of training target raw images by using the augmentation method, where L can be an integer greater than or equal to 2, preferably L is 10000, 20000 or 30000 or larger or smaller. The L sets of training target raw images can be used as a training target raw image set, and the L sets of training input raw images can be used as a training input raw image set. It should be clear that the number of groups of training target raw images in the training target raw image set and the number of groups of training input raw images in the training input raw image set can be the same or different. Figure 2When the network shown is trained, at least one set of training input original images is selected from the training input original image set as input, and simultaneously input into a first image processing branch consisting of a super-resolution reconstruction module 100, a super-resolution noise reduction module 200, a modulation module 300 and a first feature extraction module 410, and into an independent second image processing branch consisting of a second feature extraction module 420. The output of the first image processing branch and the output of the second image processing branch are summed and used as the input of the structure restoration module 430. The output of the structure restoration module 430 is averaged with the original images input to the first image processing branch and the original images input to the second processing branch and used as the output of the network model during training. The training target original images selected from the training target original image set are used as the targets during network model training.

[0071] The original image set used for training input and the original image set used for training target can be normalized. Therefore, when the output of the structure recovery module 430 is summed and averaged with the original image input to the first image processing branch and the original image input to the second processing branch, the output of the network model during training can be obtained by summing the intensity of each pixel first and then calculating the average value.

[0072] For example, in the case of Figure 2 When the network shown is trained, in the first image processing branch, L sets of training input original images (e.g., low signal-to-noise ratio images) are super-reconstructed by the super-resolution reconstruction module 100 and then processed by the super-resolution denoising module 200 to generate L training input original super-resolution images. The L training input original super-resolution images are then processed by the modulation module 300 to generate at least L sets of stripe modulation images, which are used as inputs to the first feature extraction module 410. In the second image processing branch, the L sets of training input original images (e.g., low signal-to-noise ratio images) are directly used as inputs to the second feature extraction module 420.

[0073] Figure 4A structural diagram that can be used to build a network model is shown schematically. For example, the network model can be used to build the super-resolution denoising module 200, the first feature extraction module 410, the second feature extraction module 420, and the structure recovery module 430. The network model structure mainly includes a shallow feature extraction submodule, a deep feature extraction submodule, and an output submodule connected in series, wherein the output of the shallow feature extraction submodule is used as the input of the deep feature extraction submodule, and the output of the deep feature extraction submodule is used as the input of the output submodule. The shallow feature extraction submodule includes a convolutional layer and an activation layer (for example, the activation function of the activation layer can be a LeakyReLU activation function, or simply referred to as a LeakyReLU activation layer) connected in series. The input image is sequentially subjected to the convolutional layer and the activation layer to extract shallow features, wherein the LeakyReLU activation function can be expressed as follows:

[0074]

[0075] wherein a is a leakage value, which is usually set to a = 0.2 (for example, it can be greater than or equal to 0 and less than 1). Depending on the calculation requirements, a larger number of convolutional layers and activation functions connected in series can also be set in the shallow feature extraction submodule.

[0076] The deep feature extraction module generally includes p residual groups, wherein p is an integer greater than or equal to 1, the input of the deep feature extraction module is added to the output of the last residual group of the deep feature extraction module to obtain an output image (referred to as a long skip connection), and each residual group includes q residual blocks, wherein q is an integer greater than or equal to 1, the input of each residual group is added to the output of the last residual block of the residual group to obtain an output image (referred to as a long skip connection). In a preferred embodiment, p and q can each be 4. Figure 5 A network structure diagram of a residual block is shown schematically. As shown in Figure 5 each residual block includes two groups of convolutional layers and activation layers (for example, the activation function of the activation layer can be a LeakyReLU activation function) connected in series and a channel attention module connected downstream. The input of the residual block is added to the output of the channel attention module of the residual block to obtain an output image (referred to as a long skip connection). The channel attention module is generally composed of a global average pooling layer, a convolutional layer, a LeakyReLU activation layer, a convolutional layer, and a Sigmoid activation layer connected in series, the weights of each channel of the input image are calculated and multiplied by each channel of the input image to obtain an output image, and the input of the channel attention module is added to the output of the Sigmoid activation layer to obtain an output image (referred to as a long skip connection), wherein the Sigmoid activation function of the Sigmoid activation layer can be expressed as follows:

[0077]

[0078] The output sub-module is composed of a convolutional layer, a LeakyReLU activation layer, a channel attention module, a convolutional layer and a LeakyReLU activation layer in series, for reasonably fusing image features to obtain a final output result. It should be clear that, regardless of the shallow feature extraction sub-module or the deep feature extraction sub-module, or in the output sub-module or in the channel attention module, the number of convolutional layers and the corresponding activation layer pairs connected in series can be determined as needed, for example, more or less.

[0079] In the training of the self-supervised rationalization deep learning network as shown in Figure 3 Because the two unbalanced branches (i.e., the super-resolution denoising branch and the original image denoising branch) jointly act as the training input, the former branch is constrained by physical prior information such as illumination stripes, which can make the network training converge more quickly - this embodies the rationalization in network training, i.e., deep learning, and the published super-resolution image denoising network can better remove noise in the super-resolution image, more effectively reduce the reconstruction and motion artifact problems in the super-resolution image reconstruction process, and at the same time provide higher image resolution, achieving more accurate and higher quality super-resolution image denoising effect under different super-resolution imaging modalities (two-dimensional or three-dimensional super-resolution imaging) and various imaging conditions (various signal-to-noise ratios, imaging speeds, and imaging time lengths).

[0080] Figure 6 The basic method flowchart for training the network according to the embodiments of the present application is schematically shown. It should be clear that, Figure 6 The order of the steps shown in the above method flowchart does not constrain the method of the present application, and the order of the steps can be adjusted as needed by those skilled in the art. Figure 7 The result of the super-resolution denoising processing of the low signal-to-noise ratio super-resolution image by the super-resolution denoising module in the published super-resolution image denoising network is schematically shown.

[0081] Although specific embodiments of the present application are described in detail herein, they are given only by way of illustration and should not be construed as limiting the scope of the present application. In addition, it should be clear to those skilled in the art that the embodiments described in the specification can be used in combination with each other. Various alternatives, modifications and improvements can be conceived without departing from the spirit and scope of the present application.

Claims

1. A method for denoising super-resolution micrographs under self-supervised structured light illumination, comprising: A super-resolution image denoising network is provided, which generally includes an upstream super-resolution reconstruction module (100) and a downstream super-resolution denoising module (200). The super-resolution reconstruction module (100) is configured to use a super-resolution reconstruction algorithm to perform super-resolution reconstruction on the original image captured in structured light illumination mode to output a super-resolution reconstructed image. The super-resolution denoising module (200) is configured to perform denoising processing on the super-resolution reconstructed image to output a super-resolution denoised image. The super-resolution denoising module (200) is built using a neural network, and the super-resolution image denoising network is trained in the following manner before deployment: Provide at least one set of raw images of biological structures taken in a structured light illumination mode, wherein each set of raw images includes N×M raw images, where N and M are integers greater than or equal to 2, N represents the direction in the structured light illumination mode, and M represents the phase of the structured light stripes in the structured light illumination mode. At least one set of original images is split into at least one set of training input original images and at least one set of training target original images by downsampling first and then upsampling. The at least one set of training input original images and the at least one set of training target original images are augmented to generate a training input original image set and a training target original image set; A training network model for a self-supervised deep learning network is established for the super-resolution image denoising network. This training model includes a first feature extraction module (410), a second feature extraction module (420), a structure restoration module (430), and a modulation module (300). The first feature extraction module (410), the second feature extraction module (420), and the structure restoration module (430) are constructed using neural networks. The first feature extraction module (410) and the second feature extraction module (420) are respectively configured to extract features from the input image. The modulation module (300) is configured to multiply the input image by the stripes with the same direction and phase as the illumination stripes used in the structured light illumination mode, outputting a stripe-modulated image. In the process of training the network model, the super-resolution reconstruction module (100), the super-resolution noise reduction module (200), the modulation module (300), and the first feature extraction module (410) sequentially constitute the first image processing branch, and the second feature extraction module (420) constitutes an independent second image processing branch. The training input original images selected from the training input original image set are respectively input into the first image processing branch and the second image processing branch. The output of the first image processing branch and the output of the second image processing branch are summed and used as the input of the structure restoration module (430). The output of the structure restoration module (430) is averaged with the original images input to the first image processing branch and the original images input to the second image processing branch and used as the output of the network model during training. The training target original images selected from the training target original image set are used as the target of the network model during training.

2. The method according to claim 1, characterized in that, In the downsampling followed by upsampling method, each of the N×M original images in a set of original images is divided into adjacent pixel regions of the same size. At least one pixel is randomly selected from each pixel region and interpolation is used to generate a training input original image corresponding to that original image. For the remaining pixels in each pixel region, at least one pixel is selected from them and interpolation is used to generate a training target original image corresponding to that original image, so as to generate a set of training input original images and a set of training target original images.

3. The method according to claim 2, characterized in that, The size of the pixel region is 2×2. In the method of downsampling first and then upsampling, one pixel is randomly extracted from each pixel region and interpolation is used to generate the original input image for training.

4. The method according to claim 3, characterized in that, The at least one set of raw images of biological structures captured in structured light illumination mode are low signal-to-noise ratio raw images.

5. The method according to claim 4, characterized in that, The amplification methods include, but are not limited to, random cropping, random angle rotation, or image symmetry operations.

6. The method according to any one of claims 1 to 5, characterized in that, The modulation module (300) is configured to first multiply the input image with the stripes of the same direction and phase as the illumination stripes used when the structured light illumination mode is captured, and then convolve it with the system point spread function of the optical imaging system used to capture the original image to output a stripe-modulated image.

7. The method according to any one of claims 1 to 5, characterized in that, The original set of input images for training and the original set of target images for training are normalized.

8. The method according to claim 6, characterized in that, The original set of input images for training and the original set of target images for training are normalized.

9. The method according to claim 7, characterized in that, The loss function of the trained network model is determined by the network output image ( ) and target image ( The mean absolute error (MAE) and structural similarity (SSIM) between the two components can be expressed by the formula: in, The weights for structural similarity loss are... The value of is greater than or equal to 0.

10. The method according to claim 8, characterized in that, The loss function of the trained network model is determined by the network output image ( ) and target image ( The mean absolute error (MAE) and structural similarity (SSIM) between the two components can be expressed by the formula: in, The weights for structural similarity loss are... The value of is greater than or equal to 0.

11. The method according to any one of claims 1 to 5, characterized in that, The neural networks include, but are not limited to, convolutional neural networks, deep neural networks, Transformers, and attention-based neural networks.

12. The method according to claim 6, characterized in that, The neural networks include, but are not limited to, convolutional neural networks, deep neural networks, Transformers, and attention-based neural networks.

13. The method according to claim 7, characterized in that, The neural networks include, but are not limited to, convolutional neural networks, deep neural networks, Transformers, and attention-based neural networks.

14. The method according to claim 8, characterized in that, The neural networks include, but are not limited to, convolutional neural networks, deep neural networks, Transformers, and attention-based neural networks.

15. The method according to claim 9, characterized in that, The neural networks include, but are not limited to, convolutional neural networks, deep neural networks, Transformers, and attention-based neural networks.

16. The method according to claim 10, characterized in that, The neural networks include, but are not limited to, convolutional neural networks, deep neural networks, Transformers, and attention-based neural networks.

17. The method according to claim 11, characterized in that, The neural networks include, but are not limited to, convolutional neural networks, deep neural networks, Transformers, and attention-based neural networks.

18. The method according to claim 12, characterized in that, The super-resolution noise reduction module (200), the first feature extraction module (410), the second feature extraction module (420) and / or the structure recovery module (430) can be built using the same neural network or different neural networks.

19. The method according to claim 18, characterized in that, The neural network model used to build the super-resolution noise reduction module (200), the first feature extraction module (410), the second feature extraction module (420) and / or the structure recovery module (430) is a trainable convergent neural network model, including but not limited to U-shaped neural network model, residual neural network model, residual channel attention convolutional neural network model, or Fourier channel attention convolutional neural network model.

20. The method according to any one of claims 1 to 5, characterized in that, Each of the super-resolution noise reduction module (200), the first feature extraction module (410), the second feature extraction module (420), and the structure restoration module (430) includes a shallow feature extraction submodule, a deep feature extraction submodule, and an output submodule connected in series. The shallow feature extraction submodule includes a convolutional layer and an activation layer connected in series. The deep feature extraction module includes p residual groups, where p is an integer greater than or equal to 1. The input of the deep feature extraction module is added to the output of the last residual group of the deep feature extraction module. Each residual group includes q residual blocks. q is an integer greater than or equal to 1. The input of each residual group is added to the output of the last residual block of that residual group. Each residual block includes a series of convolutional layers and activation layers, as well as a channel attention module located downstream in series. The input of the residual block is added to the output of the channel attention module of the residual block. The channel attention module consists of a global average pooling layer, a convolutional layer, an activation layer, a convolutional layer, and an activation layer in series. It is configured to calculate the weight of each channel of the input image and multiply the weight by each channel of the input image to obtain the output image. The input of the channel attention module is added to the output of the last activation layer of the channel attention module.

21. The method according to claim 20, characterized in that, The shallow feature extraction submodule includes one, two, or more pairs of convolutional layers and activation layers connected in series; and / or the residual block includes one, two, or more pairs of convolutional layers and activation layers connected in series.

22. The method according to claim 21, characterized in that, In the shallow feature extraction submodule and / or in the residual block and / or in the output submodule, the activation function of the activation layer is the LeakyReLU activation function, which can be expressed as follows: in, This is the leakage value. Greater than or equal to 0 and less than 1, especially .

23. The method according to claim 22, characterized in that, In the channel attention module, the activation function of the upstream activation layer is the LeakyReLU activation function, which can be expressed as follows: in, This is the leakage value. Greater than or equal to 0 and less than 1, especially ,and The activation function of the downstream activation layer is the Sigmoid activation function, which can be expressed as follows: 。 24. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 23.

Images