Double-branch self-supervision image denoising method for real scene

Abstract

The invention discloses a real scene-oriented double-branch self-supervised image denoising method, which comprises the following steps of: decomposing a noise image into a low-frequency sub-band and a high-frequency sub-band, processing noise in the high-frequency sub-band by adaptively adjusting a soft threshold and a hard threshold, reconstructing the noise image, and extracting frequency domain denoising features through a residual block network; the noise image is masked, the masked image is input into the U-Net backbone network, and the U-Net backbone network outputs spatial domain denoising features; and fusing the frequency domain denoising features and the spatial domain denoising features, and inputting the fused features into a convolutional network to obtain a denoised noise image. A U-Net structure is adopted in a spatial domain branch, the detail retention capability is enhanced through multi-scale feature extraction and dynamic mask generation, meanwhile, a frequency domain branch is introduced, and noise suppression is carried out through Haar wavelet transform, so that the limitation that U-Net only depends on spatial domain processing is made up.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of integrated circuits, and more particularly, the present application relates to a real scene-oriented double-branch self-supervised image denoising method. BACKGROUND

[0002] The denoising problem is essentially a typical inverse problem, which aims to recover a clear image as close as possible to the real signal from a noisy image. This task not only directly affects the visual quality of the image, but also plays a crucial role in downstream tasks such as target detection and image segmentation. However, in real-world scenarios, due to the influence of various different capture condition devices, images contain complex and diverse noise, and the noise level and distribution are unknown and difficult to quantify, making image denoising extremely challenging.

[0003] In recent years, algorithms based on U-Net structure have performed well in medical image segmentation, image super-resolution reconstruction, and image deblurring tasks, and have become a research hotspot in the field of computer vision.

[0004] However, the traditional U-Net structure is prone to over-reliance on spatial domain features when dealing with complex noise (such as non-Gaussian or mixed noise), resulting in poor denoising effect. SUMMARY

[0005] The present application provides a real scene-oriented double-branch self-supervised image denoising method to improve the above problems.

[0006] The present application is implemented as follows: a real scene-oriented double-branch self-supervised image denoising method, characterized in that the method is as follows:

[0007] (1) The noise image is decomposed into a low-frequency subband and a high-frequency subband, the noise in the high-frequency subband is processed by adaptively adjusting the soft threshold and the hard threshold, the noise image is reconstructed, and the frequency domain denoising features are extracted through the residual block network;

[0008] (2) The noise image is masked, and the input U-Net backbone network after masking is output, and the spatial domain denoising features are output by the U-Net backbone network;

[0009] (3) The frequency domain denoising features and the spatial domain denoising features are fused and input into a two-layer convolutional network to obtain the denoised noise image.

[0010] Further, the noise process of the high-frequency subband S is as follows:

[0011]

[0012] Where S' represents the denoised high-frequency subband, sigma(·) is the Sigmoid function, std Sis the standard deviation of the high-frequency subband S, M hard is a hard threshold, and Ts represents a soft threshold, and the determination method of the soft threshold Ts is as follows:

[0013] Ts = median(|S|) + a std(|S|);

[0014] Wherein, a is an adjustable noise coefficient, median(|S|) represents the median of the high-frequency subband amplitude |S|, and std(|S|) represents the standard deviation of the high-frequency subband amplitude |S|.

[0015] Further, the hard threshold M hard The determination process is as follows:

[0016]

[0017] Wherein, Th is an amplitude threshold, Th = mean(|S|) + std(|S|).

[0018] Further, the mask generation process of the noise image is as follows:

[0019] Extract the feature maps of different scales of the noise image, generate the masks of each feature map, and generate the comprehensive mask of the noise image based on the masks of all feature maps.

[0020] Further, the mask generation process of the i-th scale feature map Feature i The specific process is as follows:

[0021] A 3x3 convolution kernel is used to map the multi-channel feature map Feature i Into a single-channel response map Response i .

[0022] The single-channel response map Response i Is normalized to the probability mask M i Of the feature map Feature i Through the Sigmoid function.

[0023] Further, the generation process of the comprehensive mask is as follows:

[0024] All scales of the probability mask are up-sampled to the resolution of the noise image by bilinear interpolation, the average value of all scales of the probability mask is calculated, and the comprehensive mask is generated.

[0025] Further, the denoised noise image I d Is expressed as:

[0026] I d = Conv2d(LeakyReLU(Conv2d(Concat(Ffreq ,F spatial ))));

[0027] wherein, Concat(F freq ,F spatial ) represents aligning and splicing the frequency domain denoising feature F freq and the spatial domain denoising feature F spatial , Conv2d represents a two-dimensional convolution network of a 3*3 convolution kernel, and LeakyReLU represents an activation function.

[0028] Further, the total loss function is:

[0029]

[0030] wherein, is a structural similarity loss, is a loss for measuring the absolute error between the denoised image and the pseudo-clean image, I pred (i) and I target (i) respectively represent the i-th pixel value of the denoised image and the pseudo-clean image, N is the total number of pixels in the image, and λ ssim is a weight coefficient.

[0031] Further, the generation process of the pseudo-clean image is specifically as follows:

[0032] The noise image I n is traversed using a sliding window with a size of s*s, the local mean and the local variance in each sliding window are calculated, the standard deviation is normalized, and a binary mask M(x,y) is generated through a threshold τ;

[0033] The low-texture region with M(x,y)=0 is divided into non-overlapping blocks with a size of k*k, and the pixels in each block are dynamically randomized to form a pseudo-clean image.

[0034] The U-Net structure is adopted in the spatial domain branch, the detail retention capability is enhanced through multi-scale feature extraction and dynamic mask generation, the frequency domain branch is introduced to suppress noise through Haar wavelet transform, and the limitation of U-Net relying only on spatial domain processing is made up. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a flowchart of the dual-branch self-supervised image denoising method for real scenes provided by the embodiment of the application;

[0036] Figure 2Denoising results of different iterations in the training process provided by the embodiments of the present application, wherein (a) is 0 times, (b) is 100 times, (c) is 200 times, (d) is 400 times, (e) is 400 times, and (f) is 800 times;

[0037] Figure 3 Comparison chart of denoising effects of different image denoising methods provided by the embodiments of the present application on different data sets, wherein (a) is Noisy, (b) is DIP, (c) is Self2Self, (d) is NN+, (e) is MASH, (f) is AMSNet, (g) is the denoising method provided by the present application, and (h) is Ground-truth.

[0038] Figure 4 Comparison chart of denoising effects of different image denoising methods provided by the embodiments of the present application on self-collected data sets, wherein (a) is Noisy, (b) is DIP, (c) is Self2Self, (d) is NN+, (e) is MASH, (f) is AMSNet, and (g) is the denoising method provided by the present application.

[0039] Figure 5 Qualitative comparison chart of denoising effects of several different module combinations of the denoising method provided by the embodiments of the present application on the PolyU data set, wherein (a) is Noisy, (b) is BL, (c) is BL+FB, (d) is BL+MSB, (e) is BL+FB+MSB, and (f) is Ground-truth.

[0040] Figure 6 Comparison chart of denoising effects of the multi-scale mask generation module provided by the embodiments of the present application under different configurations, wherein (a) is the noisy image Noisy, (b) is the no mask configuration No-Mask, (c) is the single-scale mask configuration Single-Scale Mask, (d) is the dual-scale mask configuration Dual-Scale Mask, (e) is the full multi-scale mask configuration FullMulti-Scale Mask, and (f) is the clean image Ground-truth. DETAILED DESCRIPTION

[0041] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings, and the descriptions of the embodiments will help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solutions of the present application.

[0042] This invention proposes a self-supervised image denoising method with a dual-branch architecture combining frequency and spatial domains. First, frequency domain analysis is used to accurately suppress high-frequency noise. Then, in the spatial domain, multi-scale feature extraction and dynamic mask generation are used to effectively model spatial noise. Finally, a cross-domain feature fusion module is introduced to achieve collaborative noise suppression. This method significantly improves the model's adaptability to real-world scene noise without relying on paired clean-noise data.

[0043] Figure 1 A flowchart of a dual-branch self-supervised image denoising method for real-world scenes provided in an embodiment of the present invention, the method comprising the following steps:

[0044] (1) Frequency domain processing branch: The noisy image is decomposed into low-frequency sub-band and high-frequency sub-band. The noise in the high-frequency sub-band is processed by adaptively adjusting the soft threshold and hard threshold, the noisy image is reconstructed, and then the frequency domain denoising features are extracted by the residual block network.

[0045] The frequency domain processing branch decomposes the image into multiple frequency sub-bands, suppressing noise components in the high-frequency sub-bands while preserving the low-frequency structural information of the image. The spatial multi-scale branch extracts spatial domain features through multi-scale convolution and generates local adaptive masks to selectively enhance signal regions and suppress noise regions. Finally, the feature fusion module integrates the outputs of the frequency and spatial domain branches to generate a high-quality denoised image. The entire image denoising process can be represented as:

[0046] I d =F fusion (F freq (I n ),F spatial (I n ));

[0047] in, This represents the input noisy image. The output image is described by the denoised image, where C represents the number of channels, and H and W represent the height and width of the image, respectively. freq F spatial Let F represent the processing functions for the frequency domain branch and the spatial domain branch, respectively. fusion This is the feature fusion module.

[0048] Considering that noise in real-world scenes is primarily contained in high-frequency features, effective separation of low-frequency background information and high-frequency detail information in an image allows for targeted noise removal in the high-frequency subband, thereby improving denoising performance. The frequency domain processing branch is designed to enhance the suppression of high-frequency noise through frequency domain analysis, while preserving the low-frequency background information and key structural features of the image. The detailed network structure of this branch is as follows: Figure 2The frequency domain processing branch first introduces the Haar wavelet transform to decompose the input noisy acoustic image into multiple frequency subbands, denoted as follows:

[0049] DWT(I n )={LL,LH,HL,HH};

[0050] where LL is the low-frequency subband containing background information such as large-scale color changes, and LH, HL, and HH represent high-frequency subbands in horizontal, vertical, and diagonal directions, respectively, containing texture and edge information, all with a size of R^{C×H / 2×W / 2}.

[0051] To suppress noise in the high-frequency subbands, considering the characteristics of small amplitude and wide distribution of noise in the high-frequency subbands, the present application proposes an adaptive threshold processing method to process noise in the high-frequency components by combining soft threshold and hard threshold, where the soft threshold smoothes weak noise through the Sigmoid function to reduce artifacts, and the hard threshold forces attenuation of extreme noise values to enhance noise suppression effect. For each high-frequency subband S, S∈{LH, HL, HH}, the processing process is represented as:

[0052]

[0053] where S' represents the denoised high-frequency subband, σ(·) is the Sigmoid function used to implement smooth threshold processing to avoid artifacts in the processed image; std S is the standard deviation of the high-frequency subband S, M hard is the hard threshold used to suppress extreme noise values, Ts represents the soft threshold, which is dynamically determined according to the statistical characteristics of the high-frequency subband S, and is specifically as follows:

[0054] Ts=median(|S|)+α·std(|S|);

[0055] Th=mean(|S|)+std(|S|);

[0056] where α is an adjustable noise coefficient, with an empirical value of 0.05, median(|S|) represents the median of the high-frequency subband amplitude |S|, and std(|S|) represents the standard deviation of the high-frequency subband amplitude |S|; for high-frequency coefficients with amplitude |S| exceeding the amplitude threshold Th, the hard threshold M hard is represented as follows:

[0057]

[0058] By this strategy of combining soft threshold with hard threshold, the noise of the image can be effectively suppressed, while the important edge information of the image can be reserved. Finally, the inverse discrete wavelet transform is performed on each subband after noise suppression, so that the reconstruction of the denoised image is effectively realized.

[0059] I f = IDWT (LL, LH', HL', HH');

[0060] In addition, in order to further maintain the global smoothness of the image in the denoising process, the frequency domain branch also introduces a residual learning strategy:

[0061] F freq (I n ) = I f + G (I f );

[0062] Where G(·) is a residual block network, which contains multiple convolutional layers with batch normalization and LeakyReLU activation functions.

[0063] The frequency domain processing branch focuses on the accurate suppression of high-frequency noise in the frequency domain, and through adaptive threshold strategy and residual learning, the denoising performance in complex noise scenes is improved, while the limitations of traditional frequency domain methods (such as BM3D) in high-frequency detail loss or artifact generation are avoided.

[0064] The present application proposes an adaptive noise threshold function, which dynamically adjusts the threshold by combining soft threshold (based on Sigmoid function) and hard threshold (based on subband statistical characteristics), effectively alleviating the detail loss and artifact problem of hard threshold and soft threshold function. Unlike traditional global threshold estimation, the method in this paper adaptively determines the threshold according to the mean, median and standard deviation of each subband, so as to better adapt to complex noise distribution and improve the robustness. In particular, the Haar wavelet transform is selected as the wavelet basis function in this paper. Haar wavelet has the characteristics of simplicity, symmetry and orthogonality, and its calculation complexity is low, which is suitable for real-time image processing tasks.

[0065] (2) Multi-scale spatial branch: mask the noisy image, input the masked image into the U-Net backbone network, and the U-Net backbone network outputs spatial domain denoising features;

[0066] Considering the problem that traditional denoising methods often lose image details when dealing with high-frequency noise, this paper designs a dual-branch architecture combining frequency and spatial domains. In the frequency domain branch, the frequency components of the image are decomposed by discrete wavelet transform (DWT), and the soft thresholding technique is used to accurately suppress and reconstruct high-frequency noise. The spatial domain branch realizes adaptive processing of spatial noise through multi-scale feature extraction and dynamic mask generation technology. This dual-branch design uses the feature interaction module for cross-domain information fusion, fully utilizing the sensitivity of frequency domain to noise frequency distribution and the modeling ability of spatial domain to spatial structure, which can not only preserve the integrity of high-frequency details but also avoid texture blurring problems in spatial domain processing.

[0067] The multi-scale spatial branch aims to capture diversified features in the spatial domain of the image and selectively suppress noise areas through multi-scale feature extraction and dynamic mask generation. The core lies in using multi-scale convolution to extract local and global features and generating a local adaptive mask to guide the denoising process. The model structure of the spatial domain branch is as follows Figure 3 , which contains four main modules: multi-scale feature extraction module, mask generation module, feature alignment and fusion module, and U-Net backbone network.

[0068] Multi-scale feature extraction module: The multi-scale feature extraction module mainly consists of a series of convolutional layers and down-sampling layers, which are used to capture local features of different scales in the image, providing multi-level feature representation for subsequent mask generation and denoising processing.

[0069] Specifically, the multi-scale feature extraction module first takes the image to be denoised as input , and through a series of convolution and down-sampling operations, it constructs multi-scale feature representations, achieving complete capture from local details to global structure. For the input image I n , the processing flow of multi-scale feature extraction is as follows:

[0070] Initial feature extraction: for each scale i, i = 1, 2, 3, first use a 3*3 convolution kernel with a stride of 1 and padding of 1. The convolution kernel increases the number of output channels to enhance the expression ability of the features, such as expanding the input channel C (e.g., C = 3) to 32 channels at the first scale (i = 0). Then, through batch normalization and LeakyReLU activation function (slope 0.1), the feature stability and non-linear expression are enhanced. This operation keeps the spatial resolution unchanged before and after convolution, and the output feature map dimension of the first scale is 32xHxW.

[0071]

[0072] Down-sampling operation: 2*2 max-pooling layer with stride 2 is adopted to halve the spatial resolution of feature maps, such as HxW to H / 2xW / 2. The dimension of feature maps after the 1st scale pooling is 32xH / 2xW / 2.

[0073]

[0074] Feature dimension adjustment: 3*3 convolutional layer is used to expand the number of channels to a scale-dependent number. For the i-th scale, the number of channels is 32*2 i , which means that the number of feature channels increases exponentially as the scale increases. The i-th scale feature map Feature i is represented as follows:

[0075]

[0076] After the extraction of feature map Feature i , the i-th scale feature map Feature i is used as the input to extract the i+1-th scale feature map Feature i+1 , that is, during the extraction of feature map Feature i+1 , the feature map Feature i is sequentially subjected to initial feature extraction, down-sampling operation and feature dimension adjustment.

[0077] The mask generation module generates an adaptive attention mask for the i-th scale feature map Feature i through a nonlinear mapping to guide the selective suppression of noise regions by the denoising network. The core is to generate a weight mask related to the noise distribution by learning the local statistical properties of the feature map. The specific process is as follows:

[0078] For the i-th scale feature map Feature , where C i = 32*2 i , H i = H / 2 i+1 , W i = W / 2 i+1 , the mask generation module performs the following operations:

[0079] a. A 3x3 convolution kernel is used to map multi-channel features to a single-channel response map:

[0080]

[0081] where Conv 3×3 represents a 3x3 convolution operation with learnable kernel parameters, aiming to capture the local difference patterns between noise and signal in the feature map.

[0082] b. Normalize the response map to a probability mask by Sigmoid function:

[0083]

[0084] where σ(·) constrains the value range to [0,1], M i represents the pixel-level denoising confidence, 1 represents a high-confidence clean pixel, and 0 represents a noise pixel that needs to be strongly suppressed.

[0085] c. To unify the mask resolution of different scales, each mask M i is upsampled to the resolution HxW of the input image by bilinear interpolation, and the average value is taken to generate a comprehensive mask

[0086]

[0087] where, represents a bilinear upsampling operation, S is the total number of scales, S=3, and the comprehensive mask M acts on the input noise image I by element-wise multiplication n to generate the masked image I Masked . Masked The image I spatial is input into the U-Net backbone network, and the U-Net backbone network outputs the spatial domain denoising feature F n .

[0088]

[0089] where represents channel-wise multiplication, and this operation can weaken the gradient propagation in noise-dominant regions to avoid noise interference in feature reconstruction; and high mask values preserve the feature response in signal-dominant regions and maintain texture authenticity.

[0090] The present application generates a dynamic multi-scale attention mask to adaptively constrain the gradient in noise regions, which effectively improves the model's ability to retain details at different scales and lays a foundation for high-fidelity image reconstruction.

[0091] Cross-domain feature fusion module: the frequency domain denoising feature and the spatial domain denoising feature are fused and input into a convolutional network to obtain a denoised noise image;

[0092] After obtaining the frequency domain denoising feature F freq and the spatial domain denoising feature F spatial , alignment and splicing are performed, and in order to further extract and integrate cross-domain features, the spliced features are processed by a convolutional network to generate the final denoising result, which is represented as follows: ​

[0093] I d = Conv2d(LeakyReLU(Conv2d(Concat(F freq ,F spatial ))), 64);

[0094] where Concat(F freq ,F spatial ) denotes the alignment and splicing of the frequency domain denoising feature F freq and the spatial domain denoising feature F spatial , Conv2d denotes a two-dimensional convolution network with a 3*3 convolution kernel, and LeakyReLU denotes an activation function.

[0095] To address the pain points of complex noise distribution and lack of paired training data in real scenarios, existing self-supervised methods often cause feature learning bias due to noise correlation interference. The present application proposes a double self-supervised strategy of dynamic random shuffling and sliding standard deviation estimation. On the one hand, random block pixel shuffling is performed on the input image to break the spatial correlation of noise and force the network to learn robust image content representation. On the other hand, a sliding window standard deviation estimation module is designed to dynamically calculate the noise intensity distribution of the local area and incorporate it as a priori constraint into the feature reconstruction process. This self-supervised method not only breaks through the limitations of paired data dependence, but also enhances the model's generalization ability to unknown noise distribution through dynamic perception of noise statistical characteristics.

[0096] Sliding standard deviation estimation: dynamically estimate the local noise intensity distribution through a sliding window, and incorporate it as a priori constraint into feature reconstruction to improve the model's generalization ability to unknown noise distribution. Given an input noise image I n , first calculate the local mean and variance using a sliding window of size s x s (s is set to 2). The calculation formula is as follows:

[0097]

[0098] where W(x,y) denotes the sliding window centered at (x,y), I n (i,j) denotes the pixel value at position (i,j) in image I n , μ(x,y) denotes the average value of pixels within the sliding window W(x,y), and σ 2 (x,y) denotes the pixel variance within the sliding window W(x,y).

[0099] After normalizing the standard deviation σ(x,y), a binary mask M(x,y) is generated by threshold τ (set to 0.5):

[0100]

[0101] Mask M(x,y) divides the image into high-texture regions (M=1, high noise intensity) and low-texture regions (M=0, low noise intensity), guiding the subsequent random shuffle and feature reconstruction process.

[0102] Dynamic random shuffle strategy: The dynamic random shuffle strategy generates pseudo training pairs by performing local pixel reorganization on the input noise image I n , to simulate the feature distribution of clean images and achieve self-supervised learning.

[0103] By randomly shuffling the pixels in blocks, the spatial correlation of noise is broken, forcing the network to learn robust image content representation and reducing noise interference. Given the input noise image I n , the image I n is divided into non-overlapping blocks of size k x k (k is the shuffle factor, set to 4). For each block block, a random permutation index π is generated, and the pixels are rearranged by the following operation:

[0104] I reordered = I n (π(i,j)),(i,j)∈block;

[0105] where π(i,j) is the probability distribution of the i-th row and j-th column pixel in the block block, and π(i,j) is uniformly distributed in the interval [0,1] and ensures that the permutation is performed within the block. To avoid damaging the details of high-texture regions, the shuffle operation is only performed in low-texture regions, and the final output is the shuffled image I Shuffled , which is in the form of:

[0106] I Shuffled =M·I n +(1-M)·I reordered ;

[0107] where M is a binary mask and · represents element-wise multiplication. This selective shuffle preserves the details of high-texture regions while introducing noise-independent perturbations, enhancing the network's robust representation of image content. The shuffled image serves as a "pseudo-clean" image in self-supervised learning, forming a pseudo-training pair with the original noise image. The pseudo-clean image destroys the spatial correlation of noise through the shuffle operation while preserving the overall statistical properties of the image, providing an approximate supervision signal for the model.

[0108] To achieve high-quality denoising, the invention proposes a composite perceptual loss function consisting of L1 loss and structural similarity (SSIM) loss The constitution is respectively from the pixel level reconstruction accuracy and the perception consistency two dimensions optimization model's denoising performance, ensures that the denoised image is balanced between detail restoration and visual naturalness.The total loss function is:

[0109]

[0110] Wherein, I pred (i) and I target (i) respectively represent the i-th pixel value of the denoised image and the pseudo-clean image, N is the total number of pixels in the image, The loss measures the absolute error between pixels, which can more intuitively optimize the detail restoration of the image, and λ ssim Is a weight coefficient for balancing the contribution of the two parts, and the noise image is denoised by the double-branch self-supervised image denoising method proposed in the application to form a denoised image.

[0111] The structural similarity (SSIM) loss ensures that the denoising result retains the complete structural information of the image:

[0112]

[0113] Wherein, SSIM (I pred ,I target ) represents the structural similarity between the denoised image I pred And the pseudo-clean image I target .

[0114] The double-branch self-supervised image denoising method for real scenes proposed in the application has the following beneficial technical effects:

[0115] (1) By accurately suppressing high-frequency noise in the frequency domain and fusing global and local multi-scale features in the spatial domain, the spatial noise is effectively modeled and processed, so that the noise is effectively removed.

[0116] (2) A self-supervised image denoising learning method is proposed, which discards the dependence on paired clean-noise image data, and effectively extracts features from a single noise image through dynamic random disturbance and sliding standard deviation estimation strategies, thereby enhancing the robustness of the model and the generalization in real natural scene images with complex noise distribution.

[0117] Dataset setup: To verify the effectiveness of the proposed image denoising method, we conduct experimental evaluation on multiple publicly available different types of real noise datasets (SIDD, FMDD, PolyU, and CC) and a self-collected real scene noise image set. The SIDD dataset contains natural sRGB images from smartphones, and we use the validation set and benchmark dataset of SIDD for evaluation, each containing 1280 image patches of size 3x256x256. The PolyU and CC datasets contain 100 and 15 natural images from different commercial camera brands, respectively, each image of size 512x512. The FMDD dataset contains fluorescence microscope images, each image of size 512x512.

[0118] To further verify the generalization ability of DBDenoiser in real complex scenes, we randomly collect a set of high-noise image datasets in daily life scenes using a smartphone, which contains 100 images covering low-light conditions (such as night shooting) and complex texture scenes (such as cloth, leaves, etc.). These images are taken in different environments, with diverse and challenging noise characteristics. Compared with public datasets, this dataset is closer to unpredictable noise patterns in the real world, and can fully test the robustness and detail preservation ability of the model under non-ideal conditions.

[0119] Training strategy and experimental parameters: To ensure that the model can fully learn the complex characteristics of the image denoising task, this study designs a set of refined training strategies, including the selection of optimization objectives, dynamic learning rate adjustment, early stopping strategy, and frequency domain enhancement technology. The model uses a self-supervised learning strategy, taking noisy images as input and predicting denoised images without clean images as supervision signals. The Adam optimizer is used for training, with an initial learning rate of 0.0004, and the cosine annealing strategy is used for learning rate adjustment, with a minimum learning rate of 1e-6. The Batch Size is set to 8 during training. To stabilize the gradient update process and prevent gradient explosion, the gradient clipping strategy is used with a threshold of 1.0. At the same time, the early stopping mechanism is introduced, which stops training when the validation loss does not improve for 50 consecutive epochs. Pseudo-clean images (shuffled images) are used as targets to drive the model to learn noise-independent image content. The noise suppression coefficient α of the frequency domain branch is set to 0.05, and the mask generation module of the spatial branch uses three scales (scales = 3) of feature fusion.

[0120] Comparison method and evaluation index: To verify the effectiveness of the proposed method, we conduct a full experimental comparison with several state-of-the-art image denoising algorithms, including BM3D, DIP, self2self], NN+, ScoreDVI, MASH, and AMSNet. The optimal source code published by the authors is used for image denoising testing.

[0121] The evaluation indexes use PSNR (peak signal-to-noise ratio) and SSIM (structural similarity) to quantify the denoising effect from the two dimensions of pixel-level reconstruction accuracy and perceptual consistency, respectively. All results are averaged by three independent experiments to eliminate the influence of randomness.

[0122] Table 1 shows the denoising quantitative results of the denoising method provided by the present application and the mainstream denoising algorithm on multiple public data sets, wherein the larger the PSNR and SSIM values represent the better denoising performance. In each group of data sets, the optimal and suboptimal PSNR / SSIM values are marked in bold and underlined, respectively. From the quantitative results in Table 1, it can be seen that the dual-branch denoising network proposed in this paper performs well on different test sets, especially on the FMDD, CC and PolyU data sets, with PSNR and SSIM reaching 34.54 dB / 0.915, 37.48 dB / 0.956 and 39.28 dB / 0.972, respectively. Compared with other methods, the method in this paper has achieved significant improvement on these data sets.

[0123] On the FMDD data set, the dual-branch self-supervised denoising network proposed in the present application has achieved significant performance improvement. Specifically, the PSNR and SSIM of DBDenoiser on this data set are 34.54 dB / 0.915, which is 0.83 dB and 0.033 higher than that of the MASH method (33.71 dB / 0.882). This shows that the proposed method not only effectively suppresses noise when dealing with real scene noise, but also maintains the structure and details of the image. The noise of FMDD images is usually particularly obvious in the high-frequency region. The DBDenoiser proposed in the present application combines the dual-branch design of frequency domain and spatial domain. Specifically, the frequency domain branch decomposes the image into multiple frequency subbands through discrete wavelet transform, which can accurately suppress high-frequency noise. For the high-frequency noise in the FMDD data set, the frequency domain branch can effectively remove the noise without losing the details of the image, especially in the edges and texture parts of the image. The spatial domain branch uses multi-scale feature extraction and dynamic mask generation technology to adaptively process the local noise in the FMDD data set. Since the noise distribution in the FMDD image has strong locality in the spatial domain, the spatial branch can extract local and global features of the image through multi-scale convolution, thereby better capturing image details and suppressing noise. This joint design of spatial and frequency domains makes the algorithm in this paper have stronger robustness and better detail preservation ability in the denoising task of real scenes.

[0124] Table 1. Quantitative comparison of different single-image denoising methods (PSNR (dB) / SSIM)

[0125]

[0126] To visually demonstrate the generation and updating process of the dynamic mask, Figure 2 The multiscale masks and corresponding denoising results are shown at iterations 0, 100, 200, 400, 600, and 800, respectively, during the training process. The mask values gradually converge from a random distribution to a stable state reflecting the noise distribution, indicating that the model learns the noise characteristics of the feature map and dynamically adjusts the mask to focus on the noise area, thereby improving the denoising accuracy.

[0127] Figure 3 The input noisy image, reference clean image, and denoising results of the method and benchmark method are shown. In particular, on the SIDD dataset and CC dataset, the DBDenoiser proposed in the present application significantly improves the denoising effect, especially in the recovery of details and high-frequency parts, and is significantly better than MASH, DIP, Self2Self, and other methods. Specifically, DIP and Self2Self are prone to lose details during denoising, especially in the edges and texture parts of the image, and the noise removal effect is relatively poor. In contrast, the method of the present application can better preserve details, especially in high-frequency regions with rich details, and the denoising effect is more delicate. This is mainly due to the multiscale feature extraction and dynamic mask generation mechanism introduced in the spatial domain branch, which can adaptively adjust the noise suppression area at different scales, thereby improving the detail preservation capability of the image.

[0128] Observing the denoising performance on the SIDD dataset and CC dataset, it is found that in some complex noise environments, the MASH method may produce artifacts or blurring during denoising, while the method of the present application avoids the loss of details caused by excessive smoothing while removing noise, especially in the edges and texture parts of the image. This is mainly due to the adaptive threshold processing strategy combined in the frequency domain branch, which soft-thresholds weak noise and hard-thresholds abnormal values, both of which cooperate to avoid the ringing artifacts commonly seen in traditional wavelet methods, and can accurately suppress high-frequency noise and preserve important structural information. Self2Self in the CC dataset relies too much on spatial domain features, resulting in block-like residual noise in some areas; while the multiscale mask generation module of DBDenoiser dynamically allocates small, medium, and large scale mask weights to suppress point noise, cluster artifacts, and global uneven illumination, respectively. This shows that the multiscale mask design can effectively model the heterogeneous distribution of noise.

[0129] MASH in processing cell membrane edge, due to the dependence of single scale mask, leading to high frequency texture fracture, the frequency domain branch of DBDenoiser through the input image is decomposed into multi-scale frequency components by Haar wavelet transform, and adopts adaptive soft and hard threshold fusion strategy, low amplitude noise is suppressed, avoid cell edge fracture. Based on the median and standard deviation of subband, the extreme noise value (such as abnormal bright spot in poisson noise) is forced to be truncated. The spatial domain branch of DBDenoiser generates an adaptive multi-scale mask: small scale mask can locate the tiny noise particles (such as fluorescently labeled protein clusters), medium scale mask can suppress the large area noise block between cells, and large scale mask can eliminate the global artifacts of imaging system (such as uneven illumination), through sliding window standard deviation estimation, dynamic perception of local noise intensity, avoid the overfitting problem caused by preset mask ratio.

[0130] Since the self-collected noise image dataset is derived from the actual environment, the corresponding clean image cannot be obtained, so the quantitative evaluation index value cannot be calculated, therefore, the denoising effect is evaluated only by visual observation, focusing on the clarity of the image, the detail retention, the noise residual degree and whether there are over-smoothing or artifact phenomena. The test on the self-collected dataset aims to verify the robustness of the model in processing unknown noise types and complex scenes. During the test, the model predicts each noisy image multiple times and takes the average result. The qualitative comparison of the method proposed in the present application and other denoising methods on the self-collected dataset is shown in Figure 4 .

[0131] From the visual effect, different methods perform significantly different in processing complex noise scenes. DIP and NN+ methods still have many noise points in the denoising process, the overall denoising effect is general, and they cannot effectively suppress high-intensity correlated noise or non-Gaussian noise. The Self2Self method appears color loss in some images, leading to unnatural color and poor detail performance in the denoised image. The MASH method tends to over-smooth, significantly losing the texture and edge details in the image. Although the AMSNet method has certain effect in noise suppression, it is prone to chessboard effect in high-frequency regions, affecting the visual quality of the image. In contrast, the method proposed in the present application effectively suppresses high-frequency noise through the frequency domain branch, and preserves the main structure of the image. At the same time, the spatial domain branch successfully recovers the details in the low-frequency region, such as texture and edge, through multi-scale feature extraction. In the image containing non-Gaussian noise or mixed noise, our method significantly reduces noise residue, while avoiding obvious artifacts, over-smoothing or chessboard effect, achieving the best balance between noise removal and detail preservation. Overall, the test results on the self-collected dataset show that the method proposed in the present application has strong robustness and adaptability in unseen real scenes, further verifying its potential in practical applications.

[0132] To comprehensively evaluate the contribution of each core component in the proposed DBDenoiser model, ablation experiments were conducted on the PolyU dataset and the SIDD validation dataset. The role of the frequency branch module, the multi-scale spatial branch module, and the multi-scale mask generation module was analyzed. The experiments were quantitatively evaluated by peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), and qualitatively analyzed by visual effects.

[0133] Frequency branch and multi-scale spatial branch ablation experiment: First, the contribution of the frequency branch and the multi-scale spatial branch to the denoising performance was tested. The following four configurations were set: (1) only the backbone network (Baseline, BL), using the traditional U-Net structure; (2) the backbone network combined with the frequency branch (BL+FB); (3) the backbone network combined with the multi-scale spatial branch (BL+MSB); (4) the complete model (BL+FB+MSB). The experiment was conducted on the PolyU dataset, and the results are shown in Table 2. The experimental results show that the frequency branch and the multi-scale spatial branch significantly improve the denoising effect, and the mask regularization further improves the generalization ability of the model.

[0134] Table 2 Quantitative comparison of denoising performance of different modules and their combinations on the PolyU dataset

[0135]

[0136] Among them, √ indicates the use of the corresponding branch, and the optimal value of PSNR / SSIM is highlighted in bold.

[0137] At the same time, the visual effect analysis, Figure 5The input noise image in the ablation experiment, the denoising result, and the denoising effect under different module combinations are shown. From the visualization results, it can be seen that when using the traditional U-Net structure alone as the baseline, the denoising effect is poor, especially in the high-frequency area, there is obvious noise residue, the recovery ability of the edge and detail part is weak, and there is obvious blur in the image. After enabling the frequency domain branch, the denoising effect is improved, especially in the high-frequency area. The frequency domain branch accurately suppresses high-frequency noise through wavelet transform, but it still has limitations in detail recovery in the spatial domain, and the details are still missing, resulting in a denoised image that is not clear enough. When the multi-scale spatial branch is enabled, the spatial noise of the image is well handled, and the detail recovery effect in the low-frequency area is significantly improved. However, when using the spatial domain branch alone, the suppression ability for high-frequency noise is weak, resulting in obvious noise residue in the denoised image. After enabling the complete model of the frequency domain and spatial domain double branches, the denoising effect is best, and the edges and textures are well recovered. Compared with other configurations, the clarity and detail retention ability of the image are significantly improved, noise residue and artifacts are minimized, and the denoising result is more natural and realistic.

[0138] Multi-scale mask generation module ablation experiment: To further evaluate the role of the multi-scale mask generation module, we designed the following four configurations on the SIDD validation dataset: (1) No mask (No-Mask): Remove the multi-scale mask generation module, directly use the feature extraction module of the spatial domain branch for denoising, and the input image does not use any mask; (2) Single-scale mask (Single-Scale Mask): Only enable the first scale mask generator; (3) Dual-scale mask (Dual-Scale Mask): Enable the first and second scale mask generators; (4) Full multi-scale mask (Full Multi-Scale Mask): Enable all three scale mask generators as the benchmark. The experiment uses the same training settings as the main experiment, denoises each image in the SIDD validation set, calculates the PSNR and SSIM, and saves the denoised image for visual comparison. The quantitative results are shown in Table 3, which shows that the full multi-scale mask configuration (Full Multi-Scale Mask) is significantly better than other configurations in terms of PSNR and SSIM, verifying the key role of the multi-scale mask generation module in enhancing denoising performance. The no mask configuration (No-Mask) performs the worst, indicating that mask generation is crucial for dynamic adjustment of spatial domain features.

[0139] Table 3 Quantitative results of multi-scale mask generation module ablation experiment on SIDD validation dataset

[0140]

[0141] The optimal values of PSNR / SSIM are highlighted in bold.

[0142] The qualitative analysis results are shown in Figure 6 The de-noising effects of the multi-scale mask generation module under different configurations are evaluated through visual comparison on the SIDD validation dataset. Figure 6 The comparison of noisy images (Noisy), no-mask configuration (No-Mask), single-scale mask configuration (Single-Scale Mask), dual-scale mask configuration (Dual-Scale Mask), full multi-scale mask configuration (Full Multi-Scale Mask), and clean images (Ground-truth) is shown. From the visual results, the differences in noise suppression, detail preservation, and color restoration of each configuration can be clearly observed. The images of the no-mask configuration have obvious color deviation compared to the clean images, showing tone distortion and insufficient contrast, which indicates that spatial domain feature extraction without mask guidance is difficult to effectively handle complex noise patterns. The single-scale and dual-scale mask configurations have improved in detail recovery and noise suppression, and the color deviation has been alleviated to some extent. The full multi-scale mask configuration exhibits the best de-noising effect, with significant improvement in clarity of edges and texture areas, minimal noise residue and artifacts, and overall natural and realistic image quality close to the visual quality of the clean image. These results show that the multi-scale mask generation module significantly enhances the model's adaptability to complex noise scenes by dynamically adjusting the feature weights of different scales, especially in color restoration and detail preservation.

[0143] The present application proposes an image de-noising method based on joint processing of frequency domain and spatial domain, which combines dynamic mask selection and multi-scale feature fusion, successfully dealing with the de-noising task under different noise patterns. We use a loss function with perceptual loss and mask regularization, and design an optimized training strategy, so that the model can perform well in various noise environments. Through experimental verification on multiple datasets, the model proposed in the present application not only achieves the best de-noising performance on the standard datasets SIDD and FMDD, but also demonstrates strong de-noising ability on the PolyU and CC datasets in actual noise environments, especially when dealing with highly correlated noise.

[0144] The present application is described by way of example, and it is obvious that the specific implementation of the present application is not limited by the above method. Any non-essential improvement or direct application of the concept and technical solution of the present application to other occasions without modification is within the scope of protection of the present application.

Claims

1. A real scene oriented dual branch self-supervised image denoising method, characterized in that, The method is specifically as follows: (1) The noise image is decomposed into a low-frequency subband and a high-frequency subband, the noise in the high-frequency subband is processed by adaptively adjusting a soft threshold and a hard threshold, the noise image is reconstructed, and a frequency domain denoising feature is extracted through a residual block network; (2) The noise image is masked, the input after the masking is input into a U-Net backbone network, and a spatial domain denoising feature is output by the U-Net backbone network; (3) The frequency domain denoising feature and the spatial domain denoising feature are fused and input into a convolution network to obtain a denoised noise image.

2. The real scene oriented dual-branch self-supervised image denoising method of claim 1, wherein, The noise process of the high-frequency subband S is specifically as follows: wherein S' represents the high-frequency subband after denoising, σ(·) is a Sigmoid function, std S is a standard deviation of the high-frequency subband S, M hard is a hard threshold value, and Ts represents a soft threshold value. The soft threshold value Ts is determined as follows: Ts = median(|S|) + a std(|S|); Wherein, a is an adjustable noise coefficient, median(|S|) represents the median of the amplitude |S| of the high-frequency subband, and std(|S|) represents the standard deviation of the amplitude |S| of the high-frequency subband.

3. The real scene oriented dual-branch self-supervised image denoising method of claim 2, wherein, Hard thresholding M hard The determination process is as follows: Wherein, Th is an amplitude threshold, Th = mean(|S|) + std(|S|).

4. The real scene oriented dual-branch self-supervised image denoising method of claim 1, wherein, The mask generation process of the noise image is specifically as follows: The feature maps of different scales of the noise image are extracted, the masks of the feature maps are generated, and the comprehensive mask of the noise image is generated based on the masks of all the feature maps.

5. The real scene oriented dual-branch self-supervised image denoising method of claim 4, wherein, Feature map of the i-th scale i The mask generation process is as follows: The multi-channel feature map Feature is mapped into a single-channel response map Response by using a 3x3 convolution kernel i i ;​ The single channel response map Response is normalized by a Sigmoid function to a probability mask M i of the feature map Feature i i .​ 6. The real scene oriented dual-branch self-supervised image denoising method of claim 5, wherein, The generation process of the comprehensive mask is specifically as follows: All the probability masks of different scales are up-sampled to the resolution of the noise image through bilinear interpolation, the average value of the probability masks of all the scales is calculated, and the comprehensive mask is generated.

7. The real-scene oriented dual-branch self-supervised image denoising method of claim 1, wherein, de-noised noisy image I d is represented as follows: I d = Conv2d(LeakyReLU(Conv2d(Concat(F freq , F spatial )), 64, 3, 1, 1) ); wherein, Concat(F freq ,F spatial ) represents aligning and splicing the frequency domain denoising feature F freq and the spatial domain denoising feature F spatial , Conv2d represents a two-dimensional convolution network of a 3*3 convolution kernel, and LeakyReLU represents an activation function.

8. The real-scene oriented dual-branch self-supervised image denoising method of claim 1, wherein, The total loss function is: wherein, is a structural similarity loss, is a loss measuring the absolute error between the denoised image and the pseudo-clean image, I pred (i) and I target (i) represent the i-th pixel value of the denoised image and the pseudo-clean image, respectively, N is the total number of pixels in the image, λ ssim is a weight coefficient.

9. The real scene oriented dual-branch self-supervised image denoising method of claim 8, wherein, The generation process of the pseudo-clean image is specifically as follows: traversing the noisy image I using a sliding window of size s x s n calculating the local mean and local variance within each sliding window, normalizing the standard deviation, and generating a binary mask M(x, y) by thresholding τ The low-texture region with M(x, y) = 0 is divided into non-overlapping blocks with a size of k x k, the pixels in each block are dynamically randomized, and the pseudo-clean image is formed.

Images