Denoising method based on R2U-Net and adaptive noise adding module

Abstract

The invention discloses a denoising method based on R2U-Net and an adaptive noise addition model, and the method comprises the steps: obtaining an original noise image from a fluorescence microscope, carrying out the preprocessing, and reducing the noise level of the original noise image through median filtering; building a noise adding module to add noise to the image, wherein the noise adding comprises regional noise adding and global noise adding; the method comprises the following steps: establishing an R2U-Net network, and introducing a residual learning and recursion unit on the basis of U-Net; the R2U-Net network is trained, a new noise image is generated through self-supervised learning and serves as input of the R2U-Net network, the R2U-Net network is used for predicting a denoising result and optimizing the network, and the optimized R2U-Net network is used for denoising an image to be processed. According to the method, the noise in the fluorescent microscopic image can be dynamically and accurately removed without a large amount of annotation data, the detail information of the image is reserved, and the image quality is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision image processing, and specifically relates to a denoising method based on R2U-Net and an adaptive noise addition module. Background Art

[0002] Fluorescence microscopy imaging (FMI), as an important biomedical imaging technology, is widely used in fields such as cell structure observation and tissue analysis. The image acquisition of FMI is usually interfered by noise. Especially under high-noise background and weak-signal conditions, it seriously affects the quality of the image and the accuracy of analysis. Noise usually shows the discreteness of photon counting, mainly including the combination of Poisson noise and Gaussian noise. Since the signal in the image is weak, the noise is particularly significant in areas with higher brightness. Therefore, denoising processing has become a key link to improve the image quality.

[0003] Traditional denoising methods include filtering-based algorithms, model-based methods, and learning-based denoising techniques. Most of these methods are based on statistical modeling or classical signal processing techniques, such as Gaussian filtering, mean filtering, wavelet transform, etc. Although they can effectively remove noise in some cases, they often lead to the loss of image details. For complex noise environments, such as the combined noise (such as the mixture of Poisson and Gaussian noise) in FMI, traditional methods are difficult to obtain good denoising effects.

[0004] Currently, image denoising methods based on deep learning have been widely applied in many fields. In particular, convolutional neural networks (CNNs) and their variants (such as U-Net, DIP, etc.) are used for image denoising. Especially methods such as Noiser2Noise and Noise2Void adopt a self-supervised learning approach, build a model to learn the characteristics of noise, and automatically recover a more realistic image from the noisy image.

[0005] The core idea of these methods is to add noise and perform multiple trainings, and use the network to learn the statistical characteristics of the noise to achieve the denoising task. Methods such as Noiser2Noise usually rely on explicit assumptions about the noise type and modeling of the noise distribution, which makes them perform well under some simple noise conditions, but face challenges under complex noise (such as the combined noise that appears in FMI images). In addition, these methods usually require a large amount of labeled data for training, and the choice of the noise model directly affects the denoising effect and computational efficiency.

[0006] In the field of medical image denoising, deep learning methods, such as the network architecture based on U-Net, have been successfully applied to various image denoising tasks. The symmetric structure of U-Net enables it to efficiently capture the detailed information of images, so it has been widely used in medical image segmentation and denoising tasks. However, a major challenge faced by U-Net and similar convolutional neural network models in denoising tasks is the need for a large amount of labeled data for training, which is expensive and time-consuming for the processing of medical images and fluorescence microscopy images. To overcome this problem, some self-supervised learning methods have emerged in recent years, which can effectively train denoising networks without labeled data. For example, through self-supervised learning methods, the network can extract useful features from unlabeled noisy images and perform denoising. A typical representative of such methods is the method based on the generative adversarial network (GAN), which learns image denoising through the adversarial process of training a generative network and a discriminative network.

[0007] Although the image denoising technology based on self-supervised learning can be trained without labeled data, most methods are still insufficient when dealing with complex fluorescence microscopy images. Especially in the case of high-noise backgrounds, traditional self-supervised learning methods are difficult to accurately restore image details. In addition, existing denoising methods usually rely on fixed noise models and are difficult to adapt to the changes in noise characteristics in different images. Therefore, existing denoising technologies still have problems such as insufficient noise models and low denoising accuracy. Summary of the Invention

[0008] Object of the Invention: Aiming at the problems in the background technology, the present invention provides a denoising method based on R2U-Net and an adaptive noise addition model to solve the problems of low denoising accuracy and poor noise suppression effect in the prior art due to complex noise distribution and lack of labeled data. By using deep learning and adaptive noise removal technology, it is possible to dynamically and accurately remove the noise in fluorescence microscopy images without a large amount of labeled data, while retaining the detailed information of the images and improving the image quality.

[0009] Technical Solution: The present invention provides a denoising method based on R2U-Net and an adaptive noise addition model, including the following steps:

[0010] Step 1: Obtain the original noisy image from a fluorescence microscope and perform preprocessing. Reduce the noise level of the original noisy image through median filtering and obtain an image with less noise as the input for subsequent noise addition and the target image for network training;

[0011] Step 2: Build a noise addition module to add noise to the image in Step 1, including regional noise addition and global noise addition; the regional noise addition adaptively adjusts the noise intensity according to the brightness regions of the image, and determines the noise intensity of each region by calculating the mean of the pixel intensities of each region; the global noise addition adds Poisson noise to the entire image;

[0012] Step 3: Build an R2U-Net network, which introduces residual modules and recurrent units on the basis of U-Net;

[0013] Step 4: Train the R2U-Net network. During the training process, the input of the R2U-Net network comes from the image with added noise in Step 2. A new noise image is generated through self-supervised learning. The new noise image is used as the input of the R2U-Net network. The R2U-Net network is used to predict the denoising result and compare it with the target image to further optimize the network;

[0014] Step 5: After the training is completed, use the optimized R2U-Net network to denoise the image to be processed. The original image is preprocessed and normalized. The processed image is input into the trained R2U-Net model, and the denoised image is output.

[0015] Furthermore, the preprocessing in Step 1 includes normalizing the original noise image to ensure that all pixel values are in the range of [0, 1], and performing a filtering operation with a 3x3 window size on the image using median filtering. The processed image serves as both the input image and the target image for training.

[0016] Furthermore, the preprocessed image is further processed by Channel Amplification to increase the number of training samples.

[0017] Furthermore, the noise addition module in Step 2 is specifically as follows:

[0018] Divide the image into multiple small regions, generate different Poisson noises and Gaussian noises for each region based on the regional average pixel intensity, and add Poisson noise to the entire image to simulate the noise background within the full image range;

[0019] Regional noise addition: First, generate a regional mask M[k], which represents the mean of the pixel intensities in the k-th region of the image. Then, calculate the Poisson noise distribution factor λ p [k] and the Gaussian noise standard deviation σ g [k]:

[0020] σ g [k] = kg ×M[k]

[0021]

[0022] where k p and k g are respectively the Gaussian mapping factor and the Poisson mapping factor optimized for a specific microscope. For each pixel (i, j) in the k-th region, Poisson noise x p and Gaussian noise x g are added to obtain the total noise:

[0023] x i,j = x g,i,j + x p,i,j

[0024] where x p,i,j ~ P(λ p [k] × x i,j ) represents Poisson noise; represents Gaussian noise; Global noise addition: To simulate the background noise of the entire image, Poisson noise is directly added to the image:

[0025] x i,j = x p,i,j

[0026] where x p,i,j ~ P(λ p × x i,j ), and λ p is the Poisson noise distribution factor optimized for a specific microscope.

[0027] Furthermore, the R2U-Net network in step 3 is specifically as follows:

[0028] The encoder part adopts the downsampling path of U-Net to gradually extract multi-level features of the image; each convolutional block includes recurrent convolution and max pooling; recurrent convolution adds a recurrent mechanism to the convolutional layer to recursively process the features multiple times, and max pooling gradually reduces the resolution of the feature map while retaining the main features;

[0029] The bottleneck layer is located between the encoder and the decoder and is used to further extract semantic information from the high-level features, including the Recurrent Convolution Unit (RCU) to further enhance the features through recurrent operations;

[0030] Then the image enters the decoder part, which is a symmetric structure corresponding to the encoder, and is used to gradually restore the spatial resolution of the feature map. Each decoder includes an upsampling operation, a skip connection, and recurrent convolution;

[0031] The last layer of the output layer is a convolutional layer, which is used to map the decoded feature map to the target segmentation categories.

[0032] Furthermore, during the training of the R2U-Net network, the mean squared error (MSE) loss function is used, and the Adam optimizer is used for training. The learning rate is set to 0.001, and a learning rate scheduler is used. The training process for each epoch includes forward propagation, loss calculation, gradient update, and parameter optimization.

[0033] Beneficial effects:

[0034] 1. By combining self-supervised learning with the R2U-Net network, the present invention optimizes denoising by simulating noise samples and training models, reducing the need for real annotated data. Without real annotated data, it can significantly improve the denoising performance by generating noise samples and self-optimizing the model. Compared with traditional denoising methods, the present invention can effectively remove different types of noise, especially when dealing with complex noise distributions (such as Poisson noise and Gaussian noise in fluorescence microscopy images), showing stronger adaptability. According to experimental data, compared with using classical denoising algorithms (such as BM3D, DIP, etc.), the image quality after denoising by this method has been significantly improved. This has important application value in the denoising of medical images and microscopy images, especially in the case of a lack of annotated data.

[0035] 2. By designing the regional noise addition and global noise addition modules, the present invention simulates the noise distribution in different regions of real images (i.e., stronger noise in bright regions and weaker noise in dark regions), effectively overcoming the deficiency of traditional methods that cannot accurately model complex noise types. Through the design of the region-adaptive noise model, the present invention can automatically adjust the noise intensity according to the light intensity information in different regions of the image, thereby more accurately simulating real noise and improving the denoising effect. This method shows higher accuracy and effectiveness in the denoising of medical images or microscopy images. Brief description of the drawings

[0036] Figure 1 It is a flowchart of the present invention based on the R2U-Net and the adaptive noise addition model.

[0037] Figure 2 It is a structure diagram of the RU-NET of the present invention.

[0038] Figure 3 It is a structure diagram of the residual network of the present invention.

[0039] Figure 4 It is an overall structure diagram of the present invention based on the R2U-Net and the adaptive noise addition model.

[0040] Figure 5This is the structural diagram of the adaptive noise addition model of the present invention.

[0041] Figure 6 This is the effect diagram detected on the dataset of the embodiment of the present invention. Specific implementation manners

[0042] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and cannot be used to limit the protection scope of the present invention.

[0043] This embodiment demonstrates the application of combining the R2U-Net network and self-supervised learning technology in fluorescence microscope image denoising. Specifically, through steps such as image preprocessing, noise addition, and model training, a better denoising effect is achieved. The following details the implementation steps as Figure 1 、Method feasibility and experimental results.

[0044] Step 1, obtain the original noisy image from the fluorescence microscope. This image may contain Poisson noise, Gaussian noise, and other possible stray noises. Apply the Mediam Filter (median filter) with a size of 3x3 to preprocess the original image to generate a "less noisy" image, which is used as the input of the model. The preprocessed image will go through Channel Amplification (channel expansion) to improve the generalization ability of the network by increasing the number of training samples.

[0045] Given a filtered image I filtered ∈R H×W×C , where H is the image height, W is the image width, and C is the number of channels. Repeat each channel λ times to generate a new enhanced image I amp , whose size is H×W×(λ×C).

[0046] I amp (h,w,i k )=I filtered (h,w,i)

[0047] where k = 1, 2,..., C represents the original channels of the image, i k = 1, 2,..., λ represents the repeated channel index, and λ is the expansion factor, which is used to determine the number of times each channel is repeated. The input image is stored in the image folder, where there are noise maps and label maps. The noise map is used as the input image, and the label map is compared with the final denoised output image to verify PSNR and SSIM.

[0048] Step 2, build a noise addition module. The present invention designs a region-based adaptive noise model such as Figure 5, which combines Poisson noise and Gaussian noise to adapt to the noise distribution in different regions of fluorescence microscope images. It is specifically divided into two categories: region-wise noise addition and overall noise addition.

[0049] Region-wise noise addition adaptively adjusts the noise intensity according to the brightness regions of the image. By calculating the mean of the pixel intensities in each region, the noise intensity of each region is determined. First, a region mask M[k] is generated, representing the mean of the pixel intensities in the k-th region of the image. Then, according to the mask M[k], the Poisson noise distribution factor λ p [k] and the Gaussian noise standard deviation σ g [k] are calculated as follows:

[0050] σ g [k] = k g ×M[k]

[0051]

[0052] where: k p and k g are the Gaussian mapping factor and Poisson mapping factor optimized for a specific microscope, respectively. For each pixel (i, j) in the k-th region, Poisson noise x p and Gaussian noise x g are added to obtain the total noise:

[0053] x i,j = x g,i,j + x p,i,j

[0054] where x p,i,j ~P(λ p [k]×x i,j ) represents Poisson noise; represents Gaussian noise.

[0055] Overall noise addition To simulate the background noise of the entire image, Poisson noise is directly added to the image.

[0056] x i,j = x p,i,j

[0057] where x p,i,j ~P(λ p ×x i,j ), and λ p is the Poisson noise distribution factor optimized for a specific microscope.

[0058] The image is divided into multiple small regions. Noise is added to each region according to the intensity of the pixels, and then Poisson noise is added to the entire image. The goal is to generate a noisy image that is close to the noise in a real microscope image for self-supervised learning training. This step simulates the real microscope noise and can help the network learn a more accurate denoising mapping relationship.

[0059] Step 3: Build the RU-NET network model required for this experiment as Figure 2 , The R2U-Net network model introduces a residual module and a recurrent unit on the basis of U-Net. The encoder part adopts the downsampling path of the classic U-Net to gradually extract multi-level features of the image. Each convolutional block contains recurrent convolution and max pooling. Recurrent convolution adds a recurrent mechanism to the convolutional layer to recursively process the features multiple times. Max pooling gradually reduces the resolution of the feature map while retaining the main features. The bottleneck layer is located between the encoder and the decoder and is used to further extract semantic information from the high-level features. It contains a recurrent convolutional unit (RCU) that further enhances the features through recurrent operations. Then the image enters the decoder part, which corresponds to the symmetric structure of the encoder and is used to gradually restore the spatial resolution of the feature map. Each decoder includes an upsampling operation, a skip connection (concatenation or addition), and a recurrent convolution. The output layer is the last convolutional layer, which is used to map the decoded feature map to the target segmentation category.

[0060] In this experiment, adding a residual model can solve the problem of gradient disappearance and improve the image denoising performance as Figure 3 . The residual module usually contains two convolutional layers, and each convolutional layer is followed by an activation function (ReLU). The key is the skip connection, which directly adds the input to the output to achieve a shortcut path. Figure 3 The right part is the result of the convolution, plus the original input on the left. The recurrent unit is the reuse of the same convolutional layer at multiple time steps, and each iteration fuses the current input with the previous output. That is Figure 3 the circular arrow in

[0061] The improved R2U-Net network model is used for training. The input of this network model architecture is the noisy image from Step 2. The target output is the denoised image (i.e., the target image) learned by the network, without the need for additional manually labeled data. The training objective is to use the clean image generated by median filtering as the network training target and minimize the difference between the denoised image and the target image. The training method is to adopt a self-supervised learning method, and the network automatically optimizes the denoising process according to the input noisy image. The training process of the R2U-NET model uses the noisy image as the input and adjusts the network parameters by optimizing the loss function (such as mean square error MSE). The median-filtered image is used as the target, and the network adjusts the difference between the predicted image and the target image through self-supervised learning. Since the network automatically optimizes the denoising process according to self-supervised learning, it can effectively learn the noise removal strategy in the absence of manually labeled data.

[0062] Step 4, after completing the training, use the trained network for inference of the denoised image. In this stage, the input image passes through the R2U-Net model and ImageAverage to output the denoised image. The overall experimental steps are as Figure 4 , integrating the previous four steps. The sample size (SS) is set to 75, the number of training epochs (EPI) for each image is 150, and the amplification factor (AMP) is set to 2. The image output by the network will have less noise and clearer details, improving the image quality.

[0063] By comparing with traditional denoising methods (such as BM3D, DIP, etc.), the quality of the denoised image is evaluated. The method of the present invention is verified on the Fluorescence Microscopy Denoising (FMD) dataset, which includes noisy images with different noise intensities and their corresponding ground truth images.

[0064] Through the above steps for experimental training, including image preprocessing, noise addition, model training, and inference, the results are shown in Table 1:

[0065] Table 1

[0066] Confocal

[0067] Method PSNR SSIM BM3D 31.2783 0.8197 DIP 33.9367 0.9022 N2S 34.5141 0.9240 OUR 40.2536 0.9653

[0068] This result is obtained on the pictures of Confocal (laser confocal microscope). In terms of the denoising effect, the R2U-Net model combined with self-supervised learning can maintain a high PSNR and SSIM in the case of high noise intensity. By comparing the denoising effects of different algorithms, it can be concluded that the method of the present invention shows stronger robustness and higher denoising accuracy in complex noise situations. Figure 5This is the result graph obtained from this experiment. It can be seen that the finally output image is very similar to the ground truth, and the denoising effect is obvious.

Claims

1. A denoising method based on R2U-Net and adaptive noise addition model, characterized in that: The steps include: Step 1: Obtain the original noisy image from the fluorescence microscope and perform preprocessing to reduce the noise level of the original noisy image by median filtering, and obtain an image with less noise as the input for subsequent noise addition and the target image for network training; Step 2: Build a noise adding module to add noise to the image in step 1, including regional noise addition and global noise addition; The regional noise addition adaptively adjusts the noise intensity according to the brightness area of ​​the image, and determines the noise intensity of each area by calculating the mean of the pixel intensity of each area; the global noise addition adds Poisson noise to the entire image; Step 3: Build the R2U-Net network, which introduces residual modules and recursive units based on U-Net; Step 4: Train the R2U-Net network. During the training process, the input of the R2U-Net network comes from the image with noise added in step 2. A new noise image is generated through self-supervised learning. The new noise image is used as the input of the R2U-Net network. The denoising result is predicted by the R2U-Net network and compared with the target image to further optimize the network. Step 5: After training, use the optimized R2U-Net network to denoise the image to be processed. Preprocess and normalize the original image. Input the processed image into the trained R2U-Net model to output the denoised image.

2. The denoising method based on R2U-Net and adaptive noise addition model according to claim 1, characterized in that: The preprocessing in step 1 includes normalizing the original noise image to ensure that all pixel values ​​are in the interval [0, 1], using median filtering to filter the image with a window size of 3x3, and the processed image is used as both an input image and a target image for training.

3. The denoising method based on R2U-Net and adaptive noise addition model according to claim 2, characterized in that: The preprocessed images are then subjected to channel expansion ChannelAmplification to increase the number of training samples.

4. The denoising method based on R2U-Net and adaptive noise addition model according to claim 1, characterized in that: The noise adding module in step 2 is specifically as follows: The image is divided into multiple small areas, and different Poisson noise and Gaussian noise are generated for each area based on the average pixel intensity of the area, and Poisson noise is added to the entire image to simulate the noise background in the entire image; Regional noise addition: First, generate a regional mask M[k], which represents the mean pixel intensity of the kth region in the image. Then, calculate the Poisson noise distribution factor λ of each region based on the mask M[k]. p [k] and Gaussian noise standard deviation σ g [k]: s g [k]=k g ×M[k] Among them, k p and k g are Gaussian mapping factors and Poisson mapping factors optimized for a specific microscope. For each pixel (i, j) in the kth region, Poisson noise x is added. p and Gaussian noise x g , and the total noise is obtained: x i,j =x g,i,j +x p,i,j Among them, x p,i,j ~P(λ p [k]×x i,j ) represents Poisson noise; represents Gaussian noise; Global noise addition: To simulate the background noise of the entire image, Poisson noise is added directly to the image: x i,j =x p,i,j Among them, x p,i,j ~P(λ p × i,j ), λ p is the Poisson noise distribution factor optimized for a specific microscope.

5. The denoising method based on R2U-Net and adaptive noise addition model according to claim 1, characterized in that: The R2U-Net network in step 3 is specifically: The encoder part uses the downsampling path of U-Net to gradually extract the multi-level features of the image; each convolution block includes circular convolution and maximum pooling; circular convolution adds a loop mechanism to the convolution layer to perform multiple recursive processing on the features, and maximum pooling gradually reduces the resolution of the feature map while retaining the main features; The bottleneck layer is located between the encoder and the decoder, and is used to further extract semantic information from high-level features. It includes the recurrent convolution unit (RCU), which further enhances the features through recurrent operations. The image then enters the decoder part, which corresponds to the symmetrical structure of the encoder and is used to gradually restore the spatial resolution of the feature map. Each decoder includes upsampling operations, skip connections, and circular convolutions. The last layer of the output layer is a convolutional layer, which is used to map the decoded feature map to the target segmentation category.

6. The denoising method based on R2U-Net and adaptive noise addition model according to claim 5, characterized in that: During training, the R2U-Net network uses the mean square error (MSE) loss function and the Adam optimizer for training. The learning rate is set to 0.001 and the learning rate scheduler is used. The training process of each epoch includes forward propagation, loss calculation, gradient update, and parameter optimization.

Images