A fluorescence image denoising and super-resolution method

Abstract

This invention relates to the fields of fluorescence image inpainting and deep learning, and particularly to a fluorescence image denoising and super-resolution method; comprising the following steps: S1, acquiring an image to be processed; S2, constructing a fluorescence image processing model, the fluorescence image processing model being used to perform denoising or super-resolution processing on the image, including an encoder, a decoder, and a convolutional unit, the output of the encoder being connected to the input of the decoder, the output of the decoder being connected to the convolutional unit, and the convolutional unit being processed using the ReLU activation function; the encoder includes multiple encoding sub-blocks, each encoding sub-block including multiple residual blocks, each residual block including an instance normalization layer, a ReLU activation layer, and a convolutional layer; S3, inputting the image to be processed acquired in step S1 into the fluorescence image processing model constructed in step S2 to obtain a denoised and super-resolution image; thereby optimizing the performance of the denoising or super-resolution task.

Description

Technical Field

[0001] This invention relates to the fields of fluorescence image restoration and deep learning technology, and in particular to a fluorescence image denoising and super-resolution method. Background Technology

[0002] In fluorescence imaging, excessively high laser power or prolonged exposure can damage samples, cause photobleaching, and produce additional side effects, while lower power or shorter exposure times reduce the image signal-to-noise ratio. Deep learning-based computational methods offer a feasible approach to addressing this challenge by learning task-specific denoising models from limited available data. Higher resolution enables finer observation of biological processes; however, achieving higher resolution typically requires expensive instrumentation, such as structured illumination microscopy (SIM) systems, and the diffraction limit sets a fundamental upper limit on the achievable resolution of conventional optical microscopes. Similar to denoising techniques, deep learning-based computational methods can utilize existing data to learn the mapping relationship from low-resolution to high-resolution images, providing a promising solution for improving image resolution.

[0003] The current state-of-the-art method, UniFMIR, is based on the Transformer architecture and integrates multiple image inpainting tasks to build its foundational model, achieving significant performance improvements over traditional CARE methods. However, due to limitations such as scarce training data or excessively high image resolution, denoising models are typically trained on small image patches and then applied to full-resolution images during inference. Research indicates that the Transformer architecture may experience performance degradation when scaling to higher inference resolutions compared to convolutional neural network-based architectures. Furthermore, existing methods primarily focus on improving performance. While increasing the number of model parameters or adopting more advanced architectures (such as the Transformer model) can yield certain benefits, larger-scale models also place higher demands on hardware and training time. Comprehensive comparative analyses of these trade-offs are still relatively scarce in current research.

[0004] Therefore, there is an urgent need to provide a fluorescence image denoising and super-resolution method that optimizes the performance of denoising and super-resolution tasks compared to existing technologies. Summary of the Invention

[0005] This invention addresses the technical problems existing in the prior art by providing a method for fluorescence image denoising and super-resolution.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A fluorescence image denoising and super-resolution method includes the following steps: S1. Obtain the image to be processed; S2. Construct a fluorescence image processing model, which is used to denoise or perform super-resolution processing on an image. The model includes an encoder, a decoder, and a convolutional unit. The output of the encoder is connected to the input of the decoder, and the output of the decoder is connected to the convolutional unit. The output of the convolutional unit is processed using the ReLU activation function. The encoder includes multiple coding sub-blocks, each coding sub-block includes multiple residual blocks, and each residual block includes an instance normalization layer, a ReLU activation layer and a convolutional layer; S3. Input the image to be processed obtained in step S1 into the fluorescence image processing model constructed in step S2 to obtain the denoised or super-resolution image.

[0007] Furthermore, the encoder includes multiple encoding sub-blocks, and the decoder includes multiple decoding sub-blocks. The encoding sub-blocks and decoding sub-blocks are configured in a one-to-one correspondence, and the corresponding encoding sub-blocks and decoding sub-blocks are connected through a connection layer.

[0008] Furthermore, each coded subblock comprises two residual blocks, and each residual block comprises two instance normalization layers, two ReLU activation layers, and two convolutional layers.

[0009] Furthermore, adjacent coded sub-blocks are connected through max-pooling units.

[0010] Furthermore, each decoded sub-block consists of four convolutional blocks, and adjacent decoded sub-blocks are connected through upsampling layers.

[0011] Furthermore, step S3 specifically includes the following steps: S31. The image to be processed is input into the first encoding sub-block, and image processing is performed sequentially through each encoding sub-block. Image processing is also performed between adjacent encoding sub-blocks through a max pooling unit. S32. The image output from the last coded sub-block is input into the first decoded sub-block for image processing. Data transmission is performed between adjacent decoded sub-blocks through the upsampling unit. S33. The image output from the last decoded sub-block is input to the convolution unit, then processed by the ReLU activation function, and finally output as a denoised or super-resolution image.

[0012] Furthermore, in step S31, each residual block in each coded sub-block undergoes image processing using the following formula: ; in This represents the input of the current residual block. `In` represents `nn.InstanceNorm2d` from the Python `torch` package, `ReLU` represents `nn.ReLU` from the Python `torch` package, and `Conv` represents `nn.Conv2d` from the Python `torch` package. This indicates the output after processing the current residual block; Each coded sub-block undergoes image processing using the following formula: ); in Res represents the input to the current encoded sub-block, and Res represents the residual block described above. This indicates the output after processing the current coded sub-block.

[0013] Furthermore, in step S32, each convolutional block in each decoding sub-block undergoes image processing using the following formula: ; in This represents the input to the current convolutional block. `In` represents `nn.InstanceNorm2d` from the Python `torch` package, `ReLU` represents `nn.ReLU` from the Python `torch` package, and `Conv` represents `nn.Conv2d` from the Python `torch` package. This represents the output after processing the current convolutional block; The calculation formula for each decoded sub-block is as follows: ; in This represents the input to the current decoded sub-block, Co represents the processing method for each convolutional block, and y represents the data of the encoded sub-block corresponding to the current decoded sub-block after processing by the max pooling unit. This represents the output after processing by the current convolutional block.

[0014] Furthermore, except for the last encoded sub-block, the outputs of the other encoded sub-blocks are fed into the max-pooling unit for processing, and then horizontally transmitted to the corresponding decoded sub-block for addition with the output of the previous decoded sub-block and the upsampled output. The max-pooling unit uses nn.MaxPool2d from the Python torch package for image processing. The upsampling layer uses nn.Upsample(scale_factor=2, mode='nearest') from the Python torch package for processing.

[0015] Furthermore, in step S33, the convolutional unit specifically performs image processing using the following formula: ; in This represents the input to the convolutional unit. `In` represents `nn.InstanceNorm2d` from the Python `torch` package, `ReLU` represents `nn.ReLU` from the Python `torch` package, and `Conv` represents `nn.Conv2d` from the Python `torch` package. This represents the output after processing by the convolution unit.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention can train a good image restoration model based on a small number of available images, and then apply the restoration model to downstream denoising or super-resolution tasks to demonstrate its performance, thereby optimizing the performance of denoising and super-resolution tasks. Attached Figure Description

[0017] Figure 1 This is a structural diagram of the fluorescence image processing model of the present invention.

[0018] Figure 2 This is a schematic diagram of the experimental results of the fluorescence image processing model of the present invention on five denoising datasets.

[0019] Figure 3 This is a schematic diagram of the ablation experiment results of the fluorescence image processing model of this invention on a denoised dataset.

[0020] Figure 4 This is a schematic diagram of the experimental results of the fluorescence image processing model of this invention on six super-resolution datasets.

[0021] Figure 5 This is a schematic diagram of the ablation experiment results of the fluorescence image processing model of this invention on a super-resolution dataset. Detailed Implementation

[0022] The technical solution of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0023] This invention provides a method for fluorescence image denoising and super-resolution, comprising the following steps: S1. Obtain the image to be processed.

[0024] S2. Construct a fluorescence image processing model. This model is used for image denoising or super-resolution processing, such as... Figure 1As shown, the fluorescence image processing model adopts a PIR architecture, specifically including an encoder, a decoder, a connection layer, and a convolutional unit. The encoder includes four levels of encoding sub-blocks, and the decoder includes four levels of decoding sub-blocks. The encoding sub-blocks and decoding sub-blocks are set up in a one-to-one correspondence. The corresponding encoding sub-blocks and decoding sub-blocks are connected through connection layers. The first encoding sub-block receives the input image to be processed, and the last decoding sub-block is connected to the convolutional unit. The output of the convolutional unit is processed by a ReLU activation function to enhance the model's ability to perform nonlinear transformations on anomalous signals.

[0025] Each coded subblock consists of two residual blocks, each residual block consists of two instance normalization layers, two ReLU activation layers and two convolutional layers, and adjacent coded subblocks are connected by max pooling units; each decoded subblock consists of four convolutional blocks, and adjacent decoded subblocks are connected by upsampling layers.

[0026] The four coded sub-blocks are designated as the first coded sub-block, the second coded sub-block, the third coded sub-block, and the fourth coded sub-block, respectively. The four decoding sub-blocks are designated as the first decoding sub-block, the second decoding sub-block, the third decoding sub-block, and the fourth decoding sub-block, respectively. The first coded sub-block corresponds to the fourth decoding sub-block, the second coded sub-block corresponds to the third decoding sub-block, the third coded sub-block corresponds to the second decoding sub-block, and the fourth coded sub-block corresponds to the first decoding sub-block. The feature dimensions of the first coded sub-block, the second coded sub-block, the third coded sub-block, and the fourth coded sub-block are set to 64, 128, 256, and 512, respectively. The feature dimensions of the first decoding sub-block, the second decoding sub-block, the third decoding sub-block, and the fourth decoding sub-block are set to 512, 256, 128, and 64, respectively.

[0027] S3. Input the image to be processed obtained in S1 into the fluorescence image processing model constructed in step S2 for processing, and output the denoised or super-resolution image. Specifically, this includes the following steps: S31. The image to be processed is input through the first coding sub-block and sequentially enters each coding sub-block for image processing. Adjacent coding sub-blocks are also processed through a max pooling unit.

[0028] Each residual block in each coded sub-block is image processed using the following formula: ; in This represents the input of the current residual block. `In` represents `nn.InstanceNorm2d` from the Python `torch` package, `ReLU` represents `nn.ReLU` from the Python `torch` package, and `Conv` represents `nn.Conv2d` from the Python `torch` package. This indicates the output after the current residual block has been processed.

[0029] Each coded sub-block undergoes image processing using the following formula: ); in Res represents the input to the current encoded sub-block, and Res represents the residual block described above. This indicates the output after processing the current coded sub-block.

[0030] Except for the fourth coded sub-block, the outputs of the other coded sub-blocks are fed into the max pooling unit for processing, and then horizontally transmitted to the corresponding decoded sub-blocks.

[0031] The max pooling unit used is nn.MaxPool2d from the Python torch package.

[0032] S32. The image output from the fourth encoding sub-block is input into the first decoding sub-block for image processing. Data transmission is performed between adjacent decoding sub-blocks through the upsampling unit.

[0033] Each convolutional block in each decoding sub-block undergoes image processing using the following formula: ; in This represents the input to the current convolutional block. `In` represents `nn.InstanceNorm2d` from the Python `torch` package, `ReLU` represents `nn.ReLU` from the Python `torch` package, and `Conv` represents `nn.Conv2d` from the Python `torch` package. This represents the output after processing by the current convolutional block.

[0034] The calculation formula for each decoded sub-block is as follows: ; in This represents the input to the current decoded sub-block, Co represents the processing method for each convolutional block, and y represents the data of the encoded sub-block corresponding to the current decoded sub-block after processing by the max pooling unit. This represents the output after processing by the current convolutional block.

[0035] The upsampling layer uses nn.Upsample(scale_factor=2, mode='nearest') from the Python torch package for processing.

[0036] S33. The image output from the fourth decoding sub-block is input into the convolution unit. The output of the convolution unit is processed by the ReLU activation function, and then the denoised and super-resolution image is output.

[0037] The convolutional unit performs image processing using the following formula: ; in This represents the input to the convolutional unit. `In` represents `nn.InstanceNorm2d` from the Python `torch` package, `ReLU` represents `nn.ReLU` from the Python `torch` package, and `Conv` represents `nn.Conv2d` from the Python `torch` package. This represents the output after processing by the convolution unit.

[0038] like Figure 2 As shown, the model is evaluated using four metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Normalized Root Mean Square Error (NRMSE), and Time. It can be seen that the proposed PIR architecture achieves state-of-the-art performance compared to previous CARE and UniFMIR methods on five denoising task datasets (SynMT, Flywing, Tribolium, EcoliHNS, and EcoliMreb), while also reducing training speed by at least an order of magnitude.

[0039] Figure 2 The five denoising task datasets used in this study were obtained from the following sources: the SynMT dataset was obtained from https: / / github.com / WeisongZhao / SN2N / tree / main / examples / denoising2D / data, the Flywing dataset was obtained from https: / / edmond.mpg.de / dataset.xhtml?persistentId=doi:10.17617 / 3.FDFZOF, the Tribolium dataset was obtained from https: / / edmond.mpg.de / dataset.xhtml?persistentId=doi:10.17617 / 3.FDFZOF, and the EcoliMreb dataset was obtained from https: / / zenodo.org / records / 6460867.

[0040] The CARE method is detailed at http: / / www.nature.com / articles / s41592-018-0216-7.

[0041] The UniFMIR method is detailed at https: / / www.nature.com / articles / s41592-024-02244-3.

[0042] like Figure 3 As shown, different combinations of using instance normalization versus batch normalization and whether or not the final ReLU activation function is used are compared. PIR(B&N) represents the classic architecture (without the final ReLU activation function), using batch normalization; PIR(I&N) represents instance normalization but without the final ReLU activation function; and PIR(B&R) represents batch normalization but with the final ReLU activation function. The results show that the improved PIR architecture significantly improves various metrics and exhibits the best robustness compared to the classic architecture and other combinations.

[0043] like Figure 4 As shown, the model is evaluated using four metrics: PSNR, SSIM, NRMSE, and time. It can be seen that our proposed PIR architecture achieves performance comparable to previous state-of-the-art methods on six super-resolution task datasets (Saureus, Ecoli, ER, MTs, Factin, and CCPs), and exhibits significant advantages on some datasets, while also reducing training speed by up to nearly an order of magnitude.

[0044] Figure 4 The six super-resolution task datasets used (Saureus, Ecoli, ER, MTs, Factin, and CCPs) were obtained from https: / / doi.org / 10.5281 / zenodo.8401470.

[0045] like Figure 5 As shown, different combinations were also examined. The results show that the PIR architecture used in this invention achieves optimal performance and exhibits the best robustness compared to the classic architecture and other combinations.

[0046] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions made by those skilled in the art to the technical solution of the present invention do not depart from the essence and scope of the technical solution of the present invention.

Claims

1. A fluorescence image denoising and super-resolution method, characterized in that, Includes the following steps: S1. Obtain the image to be processed; S2. Construct a fluorescence image processing model, which is used to denoise or perform super-resolution processing on an image. The model includes an encoder, a decoder, and a convolutional unit. The output of the encoder is connected to the input of the decoder, and the output of the decoder is connected to the convolutional unit. The output of the convolutional unit is processed using the ReLU activation function. The encoder includes multiple coding sub-blocks, each coding sub-block includes multiple residual blocks, and each residual block includes an instance normalization layer, a ReLU activation layer and a convolutional layer; S3. Input the image to be processed obtained in step S1 into the fluorescence image processing model constructed in step S2 to obtain the denoised or super-resolution image.

2. The fluorescence image denoising and super-resolution method according to claim 1, characterized in that, The decoder includes multiple decoding sub-blocks, and the encoding sub-blocks are configured in a one-to-one correspondence with the decoding sub-blocks. The corresponding encoding sub-blocks and decoding sub-blocks are connected through a connection layer.

3. The fluorescence image denoising and super-resolution method according to claim 2, characterized in that, Each encoded subblock consists of two residual blocks, and each residual block includes two instance normalization layers, two ReLU activation layers, and two convolutional layers.

4. The fluorescence image denoising and super-resolution method according to claim 3, characterized in that, Adjacent coded sub-blocks are connected through max pooling units.

5. The fluorescence image denoising and super-resolution method according to claim 2, characterized in that, Each decoded sub-block consists of four convolutional blocks, and adjacent decoded sub-blocks are connected through upsampling layers.

6. The fluorescence image denoising and super-resolution method according to claim 2, characterized in that, Step S3 specifically includes the following steps: S31. The image to be processed is input into the first encoding sub-block, and then processed sequentially through each encoding sub-block. Image processing is also performed between adjacent encoding sub-blocks through a max pooling unit. S32. The image output from the last coded sub-block is input into the first decoded sub-block for image processing. Data transmission is performed between adjacent decoded sub-blocks through the upsampling unit. S33. The image output from the last decoded sub-block is input to the convolution unit, then processed by the ReLU activation function, and finally output as a denoised or super-resolution image.

7. The fluorescence image denoising and super-resolution method according to claim 6, characterized in that, In step S31, each residual block in each coded sub-block undergoes image processing using the following formula: ； in This represents the input of the current residual block. `In` represents `nn.InstanceNorm2d` from the Python `torch` package, `ReLU` represents `nn.ReLU` from the Python `torch` package, and `Conv` represents `nn.Conv2d` from the Python `torch` package. This indicates the output after processing the current residual block; Each coded sub-block undergoes image processing using the following formula: )； in Represents the input of the current encoded sub-block, and Res represents the residual block described above. This indicates the output after processing the current coded sub-block.

8. The fluorescence image denoising and super-resolution method according to claim 7, characterized in that, In step S32, each convolutional block in each decoding sub-block undergoes image processing using the following formula: ； in This represents the input to the current convolutional block. `In` represents `nn.InstanceNorm2d` from the Python `torch` package, `ReLU` represents `nn.ReLU` from the Python `torch` package, and `Conv` represents `nn.Conv2d` from the Python `torch` package. This represents the output after processing the current convolutional block; The calculation formula for each decoded sub-block is as follows: ； in This represents the input to the current decoded sub-block, Co represents the processing method for each convolutional block, and y represents the data of the encoded sub-block corresponding to the current decoded sub-block after processing by the max pooling unit. This represents the output after processing by the current convolutional block.

9. The fluorescence image denoising and super-resolution method according to claim 8, characterized in that, Except for the last encoded sub-block, the outputs of other encoded sub-blocks are fed into the max-pooling unit for processing. The processed data is then horizontally transmitted to the corresponding decoded sub-block and added to the output of the previous decoded sub-block and the upsampled output. The max-pooling unit uses nn.MaxPool2d from the Python torch package for image processing. The upsampling layer uses nn.Upsample(scale_factor=2, mode='nearest') from the Python torch package for processing.

10. The fluorescence image denoising and super-resolution method according to claim 9, characterized in that, In step S33, the convolutional unit performs image processing using the following formula: ； in This represents the input to the convolutional unit. `In` represents `nn.InstanceNorm2d` from the Python `torch` package, `ReLU` represents `nn.ReLU` from the Python `torch` package, and `Conv` represents `nn.Conv2d` from the Python `torch` package. This represents the output after processing by the convolution unit.

Images