Self-supervised structured light illumination super-resolution microscopic image noise reduction

Abstract

The invention discloses a self-supervised structured light illumination super-resolution microscopic image denoising method, which comprises the steps of providing a denoising network with a super-resolution reconstruction module and a super-resolution denoising module, the super-resolution reconstruction module being configured to perform super-resolution reconstruction on an original image to output a super-resolution reconstructed image, the super-resolution noise reduction module is configured to perform noise reduction processing on the super-resolution reconstructed image so as to output a super-resolution noise reduction image, the super-resolution noise reduction module is constructed by using a neural network, and the noise reduction network is trained by using the following modes before deployment: providing at least one group of original images; splitting the original image into an input original image for training and a target original image for training in a mode of down-sampling and then up-sampling; amplifying the input original image for training and the target original image for training to generate an input original image set for training and a target original image set for training; and establishing a training network model of a self-supervised deep learning network for the super-resolution image denoising network.

Description

Technical Field

[0001] The present application generally relates to a method for denoising images captured using structured light illumination patterns using neural networks and self-supervised rationalized deep learning. Background Art

[0002] The invention of the optical microscope allows people to observe life phenomena inside tiny cells, which has greatly promoted the development of life sciences. But generally speaking, traditional optical microscopes are restricted by the optical diffraction limit and cannot see detailed structures below 200nm. In recent years, from the hardware perspective, a variety of super-resolution fluorescence microscopy technologies have been invented, pushing the resolution of imaging systems to the level of 10-100nm. Among them, structured illumination super-resolution microscopy (SIM) technology is an ideal choice for studying the dynamic interactions between organelles in living cells. It has a high imaging speed while maintaining low photobleaching and phototoxicity. However, fluorescent proteins have some defects, such as a limited number of photons, easy bleaching or quenching, etc. These defects limit the acquisition of long-term, high-quality super-resolution fluorescence images.

[0003] With the rapid advancement of hardware technologies such as computers and graphics cards, deep learning-based neural network algorithms have made tremendous progress in software. In recent years, supervised learning-based image denoising neural networks, such as content-aware image restoration (CARE) and deep Fourier channel attention network (DFCAN), have been proposed. Their excellent noise reduction performance has significantly improved image quality, enabling long-term, high-signal-to-noise ratio (SNR) super-resolution imaging of living cells. Supervised denoising networks typically require the collection of a large number of structurally consistent (image) data pairs, one with a low SNR and one with a high SNR, as training sets. The network model is trained using low SNR data as input and high SNR data as output. However, in real cell samples, many organelles with strong background signals, weak fluorescence signals, or fast motion are often present, making it difficult to collect datasets with the required high SNR. These factors limit the applicability of supervised denoising networks. Therefore, researchers have further developed self-supervised denoising neural networks (hereinafter referred to as networks) to reduce the training set requirements, such as Noise2void, Blind2unblind, and Neighbor2Neighbor. The training logic of self-supervised denoising networks generally splits an image into two or more images, selecting one as the input image and the other as the target image to train the denoising network model. With a sufficient number of images for training, the network can achieve results close to those of supervised denoising.

[0004] However, when the existing self-supervised denoising network is trained using structured light super-resolution images and then released for denoising, the reconstruction and motion artifacts in the reconstructed image cannot be removed well, and the image resolution will also be reduced. In particular, when the signal-to-noise ratio of the original data is low, a good denoising effect cannot be achieved. Summary of the Invention

[0005] In response to the shortcomings of the above-mentioned denoising algorithms, the present application aims to propose a rationalized deep learning image denoising method based on self-supervision. In this method, the obtained low signal-to-noise ratio image is used as the input of two unbalanced branches during network training, including a super-resolution denoising branch and an original image denoising branch. With the help of physical prior information constraints such as illumination stripes, the noise in the super-resolution image can be better removed, and the reconstruction and motion artifact problems in the super-resolution image reconstruction process can be more effectively reduced. At the same time, higher image resolution can be provided, and more accurate and higher-quality super-resolution image denoising effects can be achieved under different super-resolution imaging modalities (two-dimensional or three-dimensional super-resolution imaging) and various imaging conditions (various signal-to-noise ratios, imaging speeds, and imaging times).

[0006] According to one aspect of the present application, a self-supervised rationalized deep learning image denoising method or a self-supervised structured light illumination super-resolution microscopy image denoising method is provided, which includes:

[0007] A super-resolution image denoising network is provided. The super-resolution image denoising network generally includes an upstream super-resolution reconstruction module and a downstream super-resolution denoising module. The super-resolution reconstruction module is configured to use a super-resolution reconstruction algorithm to perform super-resolution reconstruction on an original image captured in a structured light illumination mode to output a super-resolution reconstructed image. The super-resolution denoising module is configured to perform denoising on the super-resolution reconstructed image to output a super-resolution denoised image. The super-resolution denoising module is constructed using a neural network, and the super-resolution image denoising network is trained using the following method before deployment:

[0008] Providing at least one set of original images captured using a structured light illumination mode for a biological structure, wherein each set of original images includes N×M original images, where N and M are integers greater than or equal to 2, N represents the direction in the structured light illumination mode, and M represents the phase of the structured light fringes in the structured light illumination mode;

[0009] Splitting at least one set of original images into at least one set of training input original images and at least one set of training target original images by first downsampling and then upsampling;

[0010] Amplifying the at least one set of training input original images and the at least one set of training target original images to generate a training input original image set and a training target original image set;

[0011] A training network model of a self-supervised deep learning network is established for the super-resolution image denoising network, wherein the training model of the self-supervised deep learning network includes a first feature extraction module, a second feature extraction module, a structure recovery module, and a modulation module, wherein the first feature extraction module, the second feature extraction module, and the structure recovery module are constructed using a neural network, the first feature extraction module and the second feature extraction module are respectively configured to perform feature extraction on an input image, and the modulation module is configured to perform a point multiplication on the input image and stripes with the same direction and phase as the illumination stripes used when shooting in a structured light illumination mode to output a stripe-modulated image,

[0012] In which, when training the network model, the super-resolution reconstruction module, the super-resolution denoising module, the modulation module, and the first feature extraction module constitute a first image processing branch in sequence, and the second feature extraction module constitutes an independent second image processing branch, the training input original images selected from the training input original image set are respectively input into the first image processing branch and the second image processing branch, the output of the first image processing branch and the output of the second image processing branch are added together as the input of the structure recovery module, and the output of the structure recovery module is added together with the original image input into the first image processing branch and the original image input into the second processing branch and averaged as the output during network model training, and the training target original image selected from the training target original image set is used as the target during network model training.

[0013] Optionally, in a method of first downsampling and then upsampling, each of the N×M original images in a set of original images is split into adjacent pixel regions of the same size, at least one pixel is randomly selected from each pixel region and interpolation is used to generate a training input original image corresponding to the original image, and for the remaining pixels in each pixel region, at least one pixel is selected from them and interpolation is used to generate a training target original image corresponding to the original image, so as to generate a set of training input original images and a set of training target original images.

[0014] Optionally, the size of the pixel region is 2×2, and in a method of first downsampling and then upsampling, one pixel is randomly extracted from each pixel region and interpolation is used to generate an input original image for training.

[0015] Optionally, the at least one set of original images captured in a structured light illumination mode for biological structures are low signal-to-noise ratio original images.

[0016] Optionally, the amplification method includes but is not limited to random cropping, random angle rotation or image symmetry operation.

[0017] Optionally, the modulation module is configured to first perform a dot multiplication on the input image and the stripes with the same direction and phase as the illumination stripes used when shooting in the structured light illumination mode, and then convolve it with the system point spread function of the optical imaging system used to shoot the original image to output a stripe-modulated image.

[0018] Optionally, the training input original image set and the training target original image set are normalized.

[0019] Optionally, the loss function of the training network model is the network output image The mean absolute error (MAE) and structural similarity (SSIM) between the target image (X) are composed of two parts, which can be expressed as follows:

[0020]

[0021] Wherein, λ is the weight of the structural similarity loss, for example, the value of λ is greater than or equal to 0.

[0022] Optionally, the neural network includes but is not limited to a convolutional neural network, a deep neural network, a Transformer, and a neural network based on an attention mechanism.

[0023] Optionally, the super-resolution denoising module, the first feature extraction module, the second feature extraction module and / or the structure restoration module can be constructed using the same neural network or different neural networks.

[0024] Optionally, the neural network model used to build the super-resolution denoising module, the first feature extraction module, the second feature extraction module and / or the structure recovery module is a neural network model that can be trained to converge, including but not limited to a U-type neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.

[0025] Optionally, any one of the super-resolution denoising module, the first feature extraction module, the second feature extraction module and the structure recovery module includes a shallow feature extraction submodule, a deep feature extraction submodule and an output submodule connected in series, the shallow feature extraction submodule includes a convolution layer and an activation layer connected in series, the deep feature extraction module includes p residual groups, where p is an integer greater than or equal to 1, the input of the deep feature extraction module is added to the output of the last residual group of the deep feature extraction module, and each residual group includes q residual blocks, where q is greater than or equal to 1. The input of each residual group is added to the output of the last residual block of the residual group. Each residual block includes a convolutional layer and an activation layer connected in series and a channel attention module connected in series downstream. The input of the residual block is added to the output of the channel attention module of the residual block. The channel attention module is composed of a global average pooling layer, a convolutional layer, an activation layer, a convolutional layer, and an activation layer connected in series. It is configured to calculate the weight of each channel of the input image and multiply the weight by each channel of the input image to obtain the output image, and the input of the channel attention module is added to the output of the last activation layer of the channel attention module.

[0026] Optionally, the shallow feature extraction submodule includes one, two or more pairs of convolutional layers and activation layers connected in series; and / or the residual block includes one, two or more pairs of convolutional layers and activation layers connected in series.

[0027] Optionally, in the shallow feature extraction submodule and / or in the residual block and / or in the output submodule, the activation function of the activation layer is a LeakyReLU activation function, which can be expressed as follows:

[0028]

[0029] Here, α is a leakage value, which is greater than or equal to 0 and less than 1, and in particular, α=0.2.

[0030] Optionally, in the channel attention module, the activation function of the upstream activation layer is a LeakyReLU activation function, which can be expressed as follows:

[0031]

[0032] Wherein, α is the leakage value, α is greater than or equal to 0 and less than 1, in particular α = 0.2, and

[0033] The activation function of the downstream activation layer is the Sigmoid activation function, which can be expressed as follows:

[0034]

[0035] According to another aspect of the present application, a computer program product is provided, including a computer program / instruction, wherein the computer program / instruction implements the steps of the aforementioned method when executed by a processor.

[0036] By adopting the above-mentioned technical means of the present application, because there are two unbalanced denoising branches (the first image processing branch and the second image processing branch) when performing network training, in the two branches, the resolution and noise distribution of the input image of the feature extraction module are different from each other, and the training target is the original image associated with the input image (associated due to splitting), so the super-resolution image denoising branch (i.e., the first image processing branch) can learn more high-frequency information, resulting in better noise reduction and elimination of motion artifacts. In addition, the self-supervised training method does not require the special collection of high-quality training data sets, and only requires original images with low signal-to-noise ratio to complete the training of the network model. The denoising effect of the denoising network released after training convergence is more faithful and has a wider application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The following detailed description and the accompanying drawings will provide a more comprehensive understanding of the principles and various aspects of this application. It should be noted that the scales of the various drawings may vary for the purpose of clarity, but this does not affect the understanding of this application. In the accompanying drawings:

[0038] Figure 1Schematically shows a structural diagram of a super-resolution image denoising network according to an embodiment of the present application;

[0039] Figure 2 Schematically shows a model structure diagram of a rationalized deep learning network based on self-supervision according to an embodiment of the present application;

[0040] Figure 3 A non-limiting example of splitting an image is schematically shown;

[0041] Figure 4 Schematically shows an example structure diagram that can be used to build a network model;

[0042] Figure 5 The network structure diagram of a residual block is schematically shown;

[0043] Figure 6 A flowchart schematically illustrates a basic method for training a network according to an embodiment of the present application; and

[0044] Figure 7 The diagram schematically shows the results of super-resolution denoising of a low signal-to-noise ratio super-resolution image using the super-resolution denoising module in the super-resolution image denoising network released after training. DETAILED DESCRIPTION

[0045] In the drawings of this application, features with the same structure or similar functions are represented by the same reference numerals. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0046] Generally, an optical imaging system is used to image a specific biological structure using a structured light illumination mode to obtain at least one set of raw images. For example, the optical imaging system may include an excitation light path and a detection light path, wherein the excitation light path includes an excitation objective lens and other optical components for generating excitation light. The excitation light beam can be emitted through the excitation objective lens in the form of periodic striped structured light to stimulate fluorescence on the biological sample. The detection light path includes a detection objective lens and other optical components for imaging, which are used to receive and detect the stimulated fluorescence. In particular, when performing three-dimensional fluorescence microscopy on a biological sample, especially a living biological sample, multiple layers of fluorescence images are continuously scanned and sampled along the optical axis direction of the detection objective lens, i.e., the axial direction. In this way, each completed scan and sampling, the multiple layers of fluorescence images obtained constitute a fluorescence image stack (also referred to as a "sequence"). Within the scope of the present application, optical imaging systems may include, but are not limited to, two-dimensional structured light systems (2D-SIM), three-dimensional structured light systems (3D-SIM), lattice light sheet structured light systems (LLS-SIM), and grazing incidence illumination structured light systems (GI-SIM).

[0047] The following description of this application only takes the processing of two-dimensional fluorescence original images as an example, but those skilled in the art should be aware that similar technologies can be extended to the processing of three-dimensional fluorescence original images.

[0048] It should be noted that within the scope of this application, the methods and / or method steps and / or neural network models described herein can be programmed as programs / instructions or subroutines that are called and run by a computer. The specific programming methods for the programs / instructions and / or subroutines are not discussed in this application, and those skilled in the art can implement the relevant functions with any well-known programming software and / or commercial software. Therefore, when describing the relevant methods and / or method steps below, this application should be understood that they can also be written as computer programs / instructions to be called and executed by a computer.

[0049] According to one embodiment of the method of the present application, it is first necessary to obtain an original fluorescence image (hereinafter referred to as the original image) taken in a structured light illumination mode for a specific biological sample (such as a biological structure of the mitochondrial outer membrane, endoplasmic reticulum, etc.). In the context of the present application, the term "image" can be understood as a two-dimensional matrix that can be processed by a computer. It should be clear to those skilled in the art that when using an optical imaging system to photograph a biological sample in a structured light illumination mode, according to the requirements of subsequent super-resolution reconstruction, it is necessary to obtain at least one set of original images, where each set of original images includes N×M original images, where N and M are integers greater than or equal to 2, for example, N and M can be 3, N represents the direction in the structured light illumination mode, and M represents the phase of the structured light stripes in the structured light illumination mode. Using a super-resolution reconstruction algorithm (such as the traditional Wiener algorithm), the N×M original images can be reconstructed into one super-resolution image. Here, the original image is preferably an original image with a low signal-to-noise ratio, so that the optical imaging system can easily obtain it for biological samples, especially living biological samples. In the context of this application, the low signal-to-noise ratio original image is not necessarily defined by a clear signal-to-noise ratio range. In fact, the term "low signal-to-noise ratio" means that the signal-to-noise ratio of the input image used in the training of the neural network is equal to or lower than the signal-to-noise ratio of the input image when the corresponding neural network is trained and released for use. For example, the i-th original image in each set of original images can be represented by a two-dimensional matrix [2a×2b] i , where i is a non-zero integer less than N×M, a and b are integers greater than 1, 2a represents the number of pixels in the length direction of the original image, and 2b represents the number of pixels in the width direction of the original image.

[0050] Figure 1 The schematic diagram shows the structure of a super-resolution image denoising network. Using this super-resolution image denoising network, at least one set of original images is processed to obtain at least one super-resolution denoised image. The super-resolution image denoising network generally comprises a super-resolution reconstruction module 100 and a super-resolution denoising module 200. The input of the super-resolution reconstruction module 100 is the at least one set of low-SNR original images mentioned above (each set of original images comprises N×M original images). The super-resolution reconstruction module 100 is implemented using a structured light illumination super-resolution reconstruction algorithm well known to those skilled in the art, and its output is at least one low-SNR super-resolution image. The super-resolution denoising module 200 can be constructed, for example, using a convolutional neural network, a deep neural network, a Transformer, or a neural network based on an attention mechanism, such as a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model. The input of the super-resolution denoising module 200 is at least one low-SNR original image (e.g., the at least one set of low-SNR original images mentioned above), and its output (or prediction) is at least one super-resolution denoised image.

[0051] The inventors of this application have discovered that when training the above-mentioned super-resolution image denoising network, if the low signal-to-noise ratio original image as input is simultaneously input into two independent branches for feature extraction and further training, one branch extracts features from the low signal-to-noise ratio original image, and the other branch extracts features from a super-resolution denoised image based on the low signal-to-noise ratio original image using physical prior information such as illumination stripes. This enables the trained denoising network to achieve higher-quality super-resolution reconstruction and remove motion artifacts. In the context of this application, the terms "feature extraction" or "performing feature extraction" are understood by those skilled in the art with reference to the knowledge familiar to neural network training.

[0052] For the sake of Figure 1 The super-resolution image denoising network shown is trained. Figure 2 The model structure diagram of the rationalized deep learning network based on self-supervision is schematically shown.

[0053] exist Figure 2 For example, a deep learning framework based on PyTorch is used and Python programming language is used to build a rationalized deep learning network model based on self-supervision. In this network model, the super-resolution image denoising network (such as Figure 1 The output of the super-resolution noise reduction image is taken as the input of the modulation module 300. The modulation module 300 can be designed, for example, to perform a dot multiplication on the input super-resolution noise reduction image with stripes of the same direction and phase as the illumination stripes used when the optical imaging system is used to shoot the biological sample in the structured light illumination mode to obtain a stripe-modulated image. Alternatively, the modulation module 300 can be designed, for example, to perform a dot multiplication on the input super-resolution noise reduction image with stripes of the same direction and phase as the illumination stripes used when the optical imaging system is used to shoot the biological sample in the structured light illumination mode, and then convolve with the system point spread function of the optical imaging system used to shoot the original image to obtain a stripe-modulated image. For example, if at least one set of low signal-to-noise ratio original images is subjected to Figure 1 The super-resolution image denoising network shown outputs at least one super-resolution denoised image as input to the modulation module 300, and the output of the modulation module 300 can be an image modulated with the same number of stripes as the at least one group of low signal-to-noise ratio original images.

[0054] For example, if I m,n(r) represents the fringe intensity at coordinate r, wherein m is selected from an integer range greater than or equal to 1 and less than or equal to M, n is selected from an integer range greater than or equal to 1 and less than or equal to N, and r can represent a two-dimensional (length and width direction) coordinate or a three-dimensional coordinate. The above-mentioned fringe (or fringe intensity) can be exemplarily expressed by the following formula (it should be clear to those skilled in the art that other structured illumination fringe formulas are also applicable to the technical solutions of this application):

[0055]

[0056] Among them, k n is the stripe vector in the nth direction, is the fringe phase at the center of the image of the mth phase, MD is the fringe modulation (usually takes the theoretical value 1),

[0057] Therefore, the fringe modulated image F m,n (r) can be obtained using the following formula

[0058]

[0059] Among them, F SR (r) is a super-resolution image after denoising using a known network (for example, a super-resolution image after denoising using the super-resolution denoising module 200 ), and PSF is a system point spread function.

[0060] These fringe-modulated images can then serve as input to a first feature extraction module 410, which is configured to perform feature extraction on the input fringe-modulated images. In other words, the image processing path in the super-resolution denoised image processing (feature extraction) branch during network training is roughly as follows: low signal-to-noise ratio original image -> super-resolution denoised image -> fringe-modulated image -> first feature extraction module 410.

[0061] In addition, at least one group of low signal-to-noise ratio original images serves as the input of the second feature extraction module 420, and the second feature extraction module 420 is configured to extract key features from the input low signal-to-noise ratio original images. That is, the low signal-to-noise ratio original image -> second feature extraction module 420 forms an image processing path in the original image denoising image processing (feature extraction) branch during network training. The outputs of the first feature extraction module 410 and the second feature extraction module 420 are added together as the input of the structure recovery module 430. The structure recovery module 430 is configured to denoise the input image and output at least one group of denoised images. The at least one group of denoised images output by the structure recovery module 430 is summed and averaged with at least one group of low signal-to-noise ratio original images and at least one group of stripe modulated images, and serves as the network output image for the training network. At the same time, the target original image for training obtained in advance is used as the target of the training network, and the loss function is used for back-propagation iterative optimization, preferably using the Adam optimizer for back-propagation iterative optimization. At the same time, the above-mentioned at least one group of low signal-to-noise ratio original images serves as the network input image for the training network. In the present application, the loss function is the network output image. The mean absolute error (MAE) and structural similarity (SSIM) between the target image (X) can be expressed as follows:

[0062]

[0063] Wherein, λ is the weight of the structural similarity loss, for example, the value of λ is greater than or equal to 0, preferably between 0 and 1. During the training process, the initial learning rate can be set to 1×10 -4 . The training batch size of each feature extraction module and structure recovery module is N×n' (n' is a positive integer), and the number of input channels is M. The training batch size of the super-resolution image denoising module is n', and the number of input channels is 1. The number of channels in the middle hidden layer is generally 64. The training batch size of the original image after denoising that the network finally outputs is N×n', and the number of output channels is M. In the context of this application, the summation or averaging of the sums between images shall be based on the cognition generally familiar to those skilled in the art. For example, the summation between images can be understood as the direct summation of the intensities of the sub-pixels of each image, and the summation and averaging can be understood as the direct summation of the intensities of the sub-pixels of each image and then dividing by the number of images to obtain the average.

[0064] In the present application, the above-mentioned super-resolution denoising module 200, the first feature extraction module 410, the second feature extraction module 420, and the structure recovery module 430 can be built using the same network model, such as using a convolutional neural network, a deep neural network, a Transformer, a neural network based on an attention mechanism, or other suitable training convergent neural network models, including but not limited to, for example, a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model. Of course, those skilled in the art should be aware that these four modules can also be built using different network models or at least two of them can be built using different network models.

[0065] For example, in Figure 2 When training the model of the self-supervised rationalized deep learning network shown, first, at least one set of original images, such as at least one set of low signal-to-noise ratio original images, is captured using a structured light illumination mode for a specific biological sample (e.g., biological structures such as the mitochondrial outer membrane and the endoplasmic reticulum). Then, in order to perform self-supervised training of the network, each original image in the at least one set of original images is subjected to image splitting, such as by first downsampling and then upsampling, to generate at least two sets of images (e.g., at least two sets of low signal-to-noise ratio original images). Preferably, the brightness distribution of these images is roughly the same, but the noise is independent of each other. For example, the i-th original image in a set of original images is [2a×2b] i It can be split into a×b adjacent pixel regions, and the size of each pixel region can be 2×2. It should be clear to those skilled in the art that, depending on the number of split pixel regions, the size of the above pixel regions can also be 3×2, 3×3, 4×3, 4×2, 4×1, 2×1, or other specifications, as long as the length and width of the pixel regions can divide the original image [2a×2b] i It should be clear that the above-mentioned adjacent pixel areas mean that there is no overlap between the pixel areas and they cover (or occupy) the entire original image [2a×2b]. i .

[0066] For example, Figure 3 Schematically shows the i-th original image in a set of original images [8×6] i , which is split into 4×3 adjacent pixel regions, each pixel region is 2×2 in size. Then, one pixel is randomly selected from each pixel region to form a downsampled image of size 4×3, and an interpolation method (such as two-dimensional linear or bicubic interpolation) is used to upsample the original image [8×6] for training as the input for network training. i-IN, where the subscript "-IN" means that the image is used as the network training input; and the remaining pixels in each pixel area (for example, 3 pixels) are used to form the corresponding downsampled image, and then upsampled using interpolation (for example, 2D linear or bicubic interpolation) to obtain the training target original image [8×6] used for subsequent network training. i-TAG , where the subscript “- TAG ” means that the image is used as a training target for the network.

[0067] In an alternative embodiment not shown, two pixels may be randomly selected from each pixel region to form a downsampled image of size 4×3, and an interpolation method (e.g., two-dimensional linear or bicubic interpolation) may be used to upsample the image to obtain the training input original image [8×6] which is subsequently used as the network training input. i-IN , where the subscript " -IN " means that the image is used as the input for network training; and the remaining pixels (e.g., 2 pixels) in each pixel area, after forming the corresponding downsampled image, are upsampled using interpolation (e.g., 2D linear or bicubic interpolation) to obtain the training target original image [8×6] used for subsequent network training. i-TAG , where the subscript " -TAG " means that the image is used as a network training target. In an alternative embodiment not shown, downsampling can also be achieved, for example, in the following manner, that is, selecting one pixel from each pixel region, and then randomly selecting different numbers of pixels (for example, one, two, or other numbers of pixels less than or equal to the total number of remaining pixels) from the remaining pixels in the same pixel region, and then upsampling by interpolation to obtain the training input original image or training target original image subsequently used for network training, or the training target original image or training input original image subsequently used for network training. In an alternative embodiment not shown, one or more pixels can be randomly (and non-repeatedly) selected from each pixel region, and then upsampled by interpolation to obtain the training input original image or training target original image subsequently used for network training.

[0068] Therefore, using the above image splitting method, a set of original images can be split into a set of training input original images (comprising N×M training target original images) and a set of training target original images (comprising N×M training target original images). The set of training input original images is super-resolved using a structured light illumination super-resolution reconstruction algorithm well known to those skilled in the art (e.g., see Gustafsson, MG et al., "Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination." Biophys J 94, 4957-4970 (2008)) to obtain a training input original super-resolved image.

[0069] In order to achieve the requirements of self-supervised network training, it is necessary to increase the quantity of the above-mentioned training input original images and training target original images. For example, pixel rearrangement technology (see the inventor's patent disclosure CN116402681A) or image re-degradation technology (see the inventor's patent disclosure CN116721017A or see the author's self-supervised denoising for structured illumination microscopy enables long-term super-resolution live-cell imaging.bioRxiv, 2023-04, and the author's zero-shot learning enables instant denoising and super-resolution in optical fluorescence microscopy.bioRxiv, 2023-02) can be used for amplification. For example, the amplification method may include but is not limited to operations such as random cropping, random angle rotation, and image symmetry. For example, a group of original images obtained by splitting a group of original images can be split into a group of training input original images and a group of training target original images, which are then expanded into L groups of training input original images and L groups of training target original images through an amplification method, where L can be an integer greater than or equal to 2, preferably L is 10,000, 20,000, or 30,000 or larger or smaller. The above-mentioned L groups of training target original images can be used as a training target original image set, and the L groups of training input original images can be used as a training input original image set. It should be clear that the number of groups of training target original images in the training target original image set and the number of groups of training input original images in the training input original image set can be the same or different.

[0070] In the pair Figure 2When the network shown is trained, at least one group of training input original images is selected from the training input original image set as input, and is simultaneously input into a first image processing branch composed of a super-resolution reconstruction module 100, a super-resolution denoising module 200, a modulation module 300 and a first feature extraction module 410, and into an independent second image processing branch composed of a second feature extraction module 420. The output of the first image processing branch and the output of the second image processing branch are added together and used as the input of the structure recovery module 430, and the output of the structure recovery module 430 is added together with the original image input to the first image processing branch and the original image input to the second processing branch and averaged as the output during network model training, and the training target original image selected from the training target original image set is used as the target during network model training.

[0071] The training input original image set and the training target original image set can be normalized, and therefore, when the output of the structure recovery module 430 is added and averaged with the original image input to the first image processing branch and the original image input to the second processing branch, the intensity of each pixel point can be first summed and then the average value is calculated to serve as the output during network model training.

[0072] For example, in Figure 2 When the network shown is trained, in the first image processing branch, L groups of training input original images (e.g., low signal-to-noise ratio images) are super-reconstructed via the super-resolution reconstruction module 100 and then processed via the super-resolution denoising module 200 to generate L training input original super-resolved images, which are then processed via the modulation module 300 to generate at least L groups of fringe modulated images as input to the first feature extraction module 410; in the second image processing branch, the L groups of training input original images (e.g., low signal-to-noise ratio images) are directly used as input to the second feature extraction module 420.

[0073] Figure 4The schematic diagram shows a structure diagram that can be used to build a network model. For example, the super-resolution denoising module 200, the first feature extraction module 410, the second feature extraction module 420, and the structure recovery module 430 can be built using the network model. The network model structure mainly includes a shallow feature extraction submodule, a deep feature extraction submodule and an output submodule connected in series, wherein the output of the shallow feature extraction submodule serves as the input of the deep feature extraction submodule, and the output of the deep feature extraction submodule serves as the input of the output submodule. The shallow feature extraction submodule includes a convolution layer and an activation layer connected in series (for example, the activation function of the activation layer can be a LeakyReLU activation function, or simply a LeakyReLU activation layer). The input image passes through the convolution layer and the activation layer in sequence to extract shallow features, wherein the LeakyReLU activation function can be expressed as follows:

[0074]

[0075] Wherein, α is a leakage value, which is usually set to α=0.2 (for example, it can be greater than or equal to 0 and less than 1). Depending on the computational requirements, a larger number of convolutional layers and activation functions can be set in series in the shallow feature extraction submodule.

[0076] The deep feature extraction module generally includes p residual groups, where p is an integer greater than or equal to 1. The input of the deep feature extraction module is added to the output of the last residual group of the deep feature extraction module to obtain the output image (referred to as a long skip connection). Each residual group includes q residual blocks, where q is an integer greater than or equal to 1. The input of each residual group is added to the output of the last residual block of the residual group to obtain the output image (referred to as a long skip connection). In a preferred embodiment, p and q can each be 4. Figure 5 The network structure diagram of a residual block is shown schematically. Figure 5 As shown, each residual block includes two sets of convolutional layers and activation layers in series (for example, the activation function of the activation layer can be a LeakyReLU activation function) and a channel attention module in series downstream. The input of the residual block is added to the output of the channel attention module of the residual block to obtain the output image (referred to as a long jump connection). The channel attention module is roughly composed of a global average pooling layer, a convolution layer, a LeakyReLU activation layer, a convolution layer, and a Sigmoid activation layer in series. The weight of each channel of the input image is calculated and the weight is multiplied by each channel of the input image to obtain the output image, and the input of the channel attention module is added to the output of the Sigmoid activation layer in series to obtain the output image (referred to as a long jump connection), where the Sigmoid activation function of the Sigmoid activation layer can be expressed as:

[0077]

[0078] The output submodule is generally composed of a convolutional layer, a LeakyReLU activation layer, a channel attention module, a convolutional layer, and a LeakyReLU activation layer in series, which is used to reasonably fuse image features to obtain the final output result. It should be clear that whether in the shallow feature extraction submodule or the deep feature extraction submodule or in the output submodule or in the channel attention module, the number of convolutional layers and corresponding activation layer pairs connected in series can be determined as needed, for example, more or less.

[0079] In adopting Figure 3 In the self-supervised rationalized deep learning network training shown in the figure, because the two unbalanced branches (i.e., the super-resolution denoising branch and the original image denoising branch) work together as training input, the former branch is constrained by physical prior information such as illumination stripes, which can make the network training converge more quickly - this reflects the rationalization of network training, i.e., deep learning, and the released super-resolution image denoising network can better remove noise in super-resolution images, more effectively reduce reconstruction and motion artifact problems in the super-resolution image reconstruction process, and at the same time provide higher image resolution, and achieve more accurate and higher-quality super-resolution image denoising effects under different super-resolution imaging modalities (two-dimensional or three-dimensional super-resolution imaging) and multiple imaging conditions (multiple signal-to-noise ratios, imaging speeds, and imaging times).

[0080] Figure 6 The following schematically shows a flow chart of a basic method for training a network according to an embodiment of the present application. It should be clear that Figure 6 The order of the steps shown in the figures does not restrict the method of the present application, and those skilled in the art can adjust the order of the steps accordingly as needed. Figure 7 The diagram schematically shows the results of super-resolution denoising of a low signal-to-noise ratio super-resolution image using the super-resolution denoising module in the super-resolution image denoising network released after training.

[0081] Although specific embodiments of the present application are described in detail herein, they are provided for illustrative purposes only and should not be construed as limiting the scope of the present application. Furthermore, it will be apparent to those skilled in the art that the various embodiments described herein may be used in combination with one another. Various substitutions, variations, and modifications may be contemplated without departing from the spirit and scope of the present application.

Claims

1. A self-supervised structured light illumination super-resolution microscopy image denoising method comprising: A super-resolution image denoising network is provided. The super-resolution image denoising network generally comprises an upstream super-resolution reconstruction module (100) and a downstream super-resolution denoising module (200). The super-resolution reconstruction module (100) is configured to perform super-resolution reconstruction on an original image captured in a structured light illumination mode using a super-resolution reconstruction algorithm to output a super-resolution reconstructed image. The super-resolution denoising module (200) is configured to perform denoising processing on the super-resolution reconstructed image to output a super-resolution denoised image. The super-resolution denoising module (200) is constructed using a neural network, and the super-resolution image denoising network is trained using the following method before deployment: Providing at least one set of original images captured using a structured light illumination mode for a biological structure, wherein each set of original images includes N×M original images, where N and M are integers greater than or equal to 2, N represents the direction in the structured light illumination mode, and M represents the phase of the structured light fringes in the structured light illumination mode; Splitting at least one set of original images into at least one set of training input original images and at least one set of training target original images by first downsampling and then upsampling; Amplifying the at least one set of training input original images and the at least one set of training target original images to generate a training input original image set and a training target original image set; A training network model of a self-supervised deep learning network is established for the super-resolution image denoising network, wherein the training model of the self-supervised deep learning network includes a first feature extraction module (410), a second feature extraction module (420), a structure recovery module (430), and a modulation module (300), wherein the first feature extraction module (410), the second feature extraction module (420), and the structure recovery module (430) are constructed using a neural network, the first feature extraction module (410) and the second feature extraction module (420) are respectively configured to perform feature extraction on an input image, and the modulation module (300) is configured to perform point multiplication on the input image and stripes with the same direction and phase as the illumination stripes used when shooting in a structured light illumination mode to output a stripe-modulated image, Wherein, when training the network model, the super-resolution reconstruction module (100), the super-resolution denoising module (200), the modulation module (300), and the first feature extraction module (410) constitute a first image processing branch in sequence, and the second feature extraction module (420) constitutes an independent second image processing branch, the training input original image selected from the training input original image set is input into the first image processing branch and the second image processing branch respectively, the output of the first image processing branch and the output of the second image processing branch are added together and used as the input of the structure recovery module (430), and the output of the structure recovery module (430) is added together with the original image input into the first image processing branch and the original image input into the second processing branch and averaged as the output during network model training, and the training target original image selected from the training target original image set is used as the target during network model training.

2. The method according to claim 1, characterized in that In the downsampling-then-upsampling method, each of N×M original images in a set of original images is split into adjacent pixel regions of equal size. At least one pixel is randomly selected from each pixel region and interpolated to generate a training input original image corresponding to the original image. For the remaining pixels in each pixel region, at least one pixel is selected from the remaining pixels and interpolated to generate a training target original image corresponding to the original image, thereby generating a set of training input original images and a set of training target original images.

3. The method according to claim 2, characterized in that The size of the pixel region is 2×2. In a method of downsampling first and then upsampling, one pixel is randomly extracted from each pixel region and interpolation is used to generate an input original image for training.

4. The method according to claim 3, characterized in that The at least one set of original images captured in a structured light illumination mode for biological structures is a low signal-to-noise ratio original image.

5. The method according to claim 4, characterized in that The amplification methods include but are not limited to random cropping, random angle rotation or image symmetry operation.

6. The method according to any one of claims 1 to 5, characterized in that: The modulation module (300) is configured to first perform a point multiplication on the input image and the stripes with the same direction and phase as the illumination stripes used when shooting in a structured light illumination mode, and then perform a convolution with the system point spread function of the optical imaging system used to shoot the original image to output a stripe-modulated image.

7. The method according to any one of claims 1 to 6, characterized in that: The training input original image set and the training target original image set are normalized.

8. The method according to claim 7, characterized in that The loss function of the training network model is composed of the network output image The mean absolute error (MAE) and structural similarity (SSIM) between the target image (X) are composed of two parts, which can be expressed as follows: Among them, λ is the weight of the structural similarity loss, and the value of λ is greater than or equal to 0.

9. The method according to any one of claims 1 to 8, characterized in that: The neural network includes but is not limited to a convolutional neural network, a deep neural network, a Transformer, and a neural network based on an attention mechanism.

10. The method according to claim 9, characterized in that The super-resolution denoising module (200), the first feature extraction module (410), the second feature extraction module (420) and / or the structure recovery module (430) can be constructed using the same neural network or different neural networks.

11. The method according to claim 10, characterized in that The neural network model used to build the super-resolution denoising module (200), the first feature extraction module (410), the second feature extraction module (420) and / or the structure recovery module (430) is a neural network model that can be trained to converge, including but not limited to a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.

12. The method according to any one of claims 1 to 11, characterized in that: Any one of the super-resolution denoising module (200), the first feature extraction module (410), the second feature extraction module (420) and the structure recovery module (430) includes a shallow feature extraction submodule, a deep feature extraction submodule and an output submodule connected in series, the shallow feature extraction submodule includes a convolution layer and an activation layer connected in series, the deep feature extraction module includes p residual groups, where p is an integer greater than or equal to 1, the input of the deep feature extraction module is added to the output of the last residual group of the deep feature extraction module, and each residual group includes q residual blocks, where q is an integer greater than or equal to 1, the input of each residual group is added to the output of the last residual block of the residual group, each residual block includes a convolutional layer and an activation layer connected in series and a channel attention module connected in series downstream, the input of the residual block is added to the output of the channel attention module of the residual block, the channel attention module is composed of a global average pooling layer, a convolutional layer, an activation layer, a convolutional layer, and an activation layer connected in series, and is configured to calculate the weight of each channel of the input image and multiply the weight by each channel of the input image to obtain the output image, and the input of the channel attention module is added to the output of the last activation layer of the channel attention module.

13. The method according to claim 12, characterized in that The shallow feature extraction submodule includes one, two or more pairs of convolutional layers and activation layers connected in series; and / or the residual block includes one, two or more pairs of convolutional layers and activation layers connected in series.

14. The method according to claim 13, characterized in that In the shallow feature extraction submodule and / or in the residual block and / or in the output submodule, the activation function of the activation layer is a LeakyReLU activation function, which can be expressed as follows: Here, α is a leakage value, which is greater than or equal to 0 and less than 1, and in particular, α=0.

2.

15. The method according to claim 14, characterized in that In the channel attention module, the activation function of the upstream activation layer is the LeakyReLU activation function, which can be expressed as follows: Wherein, α is the leakage value, α is greater than or equal to 0 and less than 1, in particular α = 0.2, and The activation function of the downstream activation layer is the Sigmoid activation function, which can be expressed as follows:

16. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images