U-Net3 + image denoising method in combination with noise adaptive robust attention mechanism, electronic equipment and computer readable storage medium

Abstract

The invention belongs to the technical field of digital image processing, and particularly relates to a U-Net3 + image denoising method combined with a noise adaptive robust attention mechanism, electronic equipment and a computer readable storage medium. The method comprises the following steps: S1, preprocessing a noisy image, and inputting the preprocessed noisy image into a trained image denoising network model for prediction to obtain a predicted noise component of the noisy image; and S2, carrying out pixel-by-pixel subtraction operation on the preprocessed image with noise and the predicted noise component to obtain a denoised image. According to the invention, the noise adaptive robust attention module is introduced into the bottleneck layer of the U-Net3 + architecture, so that the recognition capability of a complex noise mode and the holding capability of image details are effectively improved. By optimizing the integration of a noise adaptive robust attention mechanism and a U-Net3 + architecture, global and local noise modes are more effectively identified, so that the image denoising performance is remarkably improved, and excellent detail preservation and artifact suppression can be realized especially in a complex noise scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital image processing, and particularly relates to a U-Net3+ image denoising method combining a noise adaptive robust attention mechanism, an electronic device and a computer readable storage medium. BACKGROUND

[0002] During the acquisition, transmission and storage of digital images, they inevitably suffer from various noises, such as Gaussian noise and salt and pepper noise. These noises can seriously degrade the visual quality of images and the performance of subsequent advanced vision tasks, such as object detection and image segmentation. Therefore, image denoising, as a fundamental and important research direction in the field of image processing, has always been a focus of attention.

[0003] In recent years, deep learning-based image denoising methods have made significant progress. Convolutional neural networks (CNN), especially U-Net and its variants, have shown excellent performance in image denoising tasks due to their powerful feature extraction and multi-scale information fusion capabilities. U-Net3+, as an important improvement of U-Net, introduces full-scale skip connections, enabling each layer of the decoder to aggregate feature maps from all scales of the encoder and deeper levels of the decoder, thus more effectively capturing contextual information and preserving image details.

[0004] To further improve the performance of deep learning models, attention mechanisms are widely introduced to enable the model to adaptively focus on important parts of the input features. However, existing attention mechanisms may have the following problems when applied to image denoising with complex networks such as U-Net3+: 1) The calculation method of attention weights may not optimally adapt to the recognition needs of global and local noise patterns in the denoising task, easily leading to detail loss or introducing new artifacts; 2) The integration method of attention modules and U-Net3+ may not fully leverage the synergistic effect of the two, leading to waste of computational resources or limited improvement in denoising performance, especially in fine texture restoration and artifact suppression.

[0005] Therefore, in view of the limitations of existing attention mechanisms in noise pattern recognition and detail preservation in U-Net3+ image denoising applications, it is necessary to propose a U-Net3+ image denoising method combining a noise adaptive robust attention mechanism. SUMMARY

[0006] In view of the problems and deficiencies in the prior art, the purpose of the present application is to provide a U-Net3+ image denoising method combining a noise adaptive robust attention mechanism, an electronic device and a computer readable storage medium.

[0007] In order to achieve the above object, the present application adopts the following technical solutions:

[0008] The first aspect of the present application provides a U-Net3+ image denoising method combined with a noise adaptive robust attention mechanism, comprising the following steps:

[0009] S1: preprocessing the noisy image, inputting the preprocessed noisy image into the trained image denoising network model for prediction to obtain the predicted noise component of the noisy image;

[0010] S2: performing a pixel-by-pixel subtraction operation on the preprocessed noisy image and the predicted noise component to obtain a denoised image;

[0011] The image denoising network model is constructed by inserting a bottleneck processing module between the encoder module and the decoder module of the U-Net3+ network; the encoder module is used for multi-scale feature extraction of the preprocessed noisy image, generating multi-level encoding features and outputting, and the multi-level encoding features include the original bottleneck features output by the deepest layer of the encoding path of the encoder module; the bottleneck processing module includes a denoising adaptive robust attention module and a feature fusion unit, the denoising adaptive robust attention module is used for adaptive learning of noise patterns and calibration of attention weights on the original bottleneck features output by the encoder module, to obtain enhanced bottleneck features and output; the feature fusion unit is used for fusing the enhanced bottleneck features with the original bottleneck features output by the encoder module to obtain fused bottleneck features; the decoder module is based on the full-scale skip connection mechanism of the U-Net3+ network, and performs upsampling and reconstruction on the fused bottleneck features output by the bottleneck processing module and the multi-level encoding features output by the encoder module, to predict and output the noise component of the input noisy image.

[0012] Further, the denoising adaptive robust attention module includes a first linear transformation layer, a first Softmax normalization layer along a specific dimension, a second element sum-based normalization layer along the specific dimension, and a second linear transformation layer arranged in sequence.

[0013] Further, the specific operation process of the denoising adaptive robust attention module for processing the original bottleneck features is as follows:

[0014] i. Dimension adaptation input: the original bottleneck features hd5 original Through the dimension rearrangement operation and the view transformation operation, it is converted into a sequence form X in , with a shape of BxNxC bn , where N=H bn xW bn represents the number of tokens after flattening the spatial dimensions;

[0015] ii. first linear transformation (L mk ): mapping the serialized input features to s external units, resulting in first intermediate representations where A1 = L mk (X in );

[0016] iii. first normalization: applying a Softmax function to A1 along the token dimension N (for each external unit s), resulting in first normalized attention maps where A2 = SOftmax dim＝N (A1), such that for each batch b and each memory unit s,

[0017] iv. second normalization: normalizing A2 again along the token dimension N (for each external unit s), first computing the sum of elements of A2 along dimension N, S sumN (and lower bound clipping ∈ min to guarantee numerical stability, resulting in S' sumN ), then dividing A2 by S' sumN , resulting in second normalized attention maps where (A3) b,n,s = (A2) b,n,s / (S' sumN ) b,s ;

[0018] v. second linear transformation (L mv ): mapping the double-normalized attention representations A3 back to the original feature dimension C, resulting in the output of the denoising adaptive robust attention module where X out = L mv (A3); vi. dimension restoration and fusion output: restoring the original spatial dimension of the output of the denoising adaptive robust attention module X out by inverse view transformation and dimension rearrangement operations, resulting in attention-enhanced bottleneck features hd5 ea .

[0019] Further, the feature fusion unit fuses the enhanced bottleneck features and the original bottleneck features output by the encoder module by element-wise addition.

[0020] Further, the encoder module is composed of five sequentially connected encoding units, and a down-sampling module is arranged between any two encoding units; each encoding unit includes two convolutional layers, and a batch normalization layer and a nonlinear activation function ReLU layer are connected after each convolutional layer.

[0021] Further, the decoding module comprises four decoding units corresponding to the first four encoding unit levels in the encoder module, the inputs of the decoding units comprise the fused bottleneck feature, the encoding features output by the encoding unit corresponding to the decoding unit level, all the encoding features in the multi-level encoding features output by the encoder module and having a spatial resolution greater than the current decoding unit level, and the decoding features output by the last decoding unit; the processing procedure of the decoding unit on the input image is that: the decoding unit performs channel dimension splicing processing on the input fused bottleneck feature, the encoding features output by the encoding unit corresponding to the decoding unit level, all the encoding features in the multi-level encoding features output by the encoder module and having a spatial resolution greater than the current decoding unit level, and all the decoding features in the multi-level decoding features output by the decoder module and having a spatial resolution less than the current decoding unit level, to obtain a wide feature map, and then performs convolution processing on the wide feature map by using a convolution block to obtain decoding features and output the decoding features.

[0022] Further, after obtaining the denoised image in step S2, the denoised image needs to be post-processed, and the specific operation of the post-processing is: truncating the denoised image to a pixel value range of [0, 1] to obtain a final denoised image.

[0023] Further, in step S1, the pre-processing mode comprises size adjustment and pixel value normalization, which is consistent with the training phase.

[0024] The second aspect of the present application provides a training method of an image denoising network model, comprising the following steps:

[0025] (1) constructing a training sample set and a verification sample set, the training sample set is a pair of noisy / clean image blocks extracted from large-format original images in the SIDD dataset, and is constructed by random cropping, data enhancement and normalization processing on the extracted image blocks; the verification sample set is obtained by loading complete original images from the SIDD dataset and performing random cropping and normalization processing on the original images;

[0026] (2) using the training sample set to train the pre-constructed image denoising network model, updating the network parameters of the image denoising network model, and obtaining a trained image denoising network model;

[0027] (3) using the verification sample set to verify the trained image denoising network model, evaluating the performance of the trained image denoising network model, selecting the optimal trained image denoising network model, and completing the training of the image denoising network model; wherein the image denoising network model is the image denoising network model of the first aspect described above.

[0028] Further, the specific operation of step (2) is: inputting the training sample in the training sample set into the image denoising network model for processing to obtain the predicted noise component of the noisy image in the training sample and output, and calculating the fidelity loss between the predicted noise component and the real noise component; then performing a pixel-by-pixel subtraction operation on the noisy image and the predicted noise component to obtain a denoised image, calculating the structure preservation loss, perception loss and edge preservation loss of the denoised image and the clean image, and constructing a loss function based on the fidelity loss, the structure preservation loss, the perception loss and the edge preservation loss; according to the constructed loss function, the parameters of the image denoising network model are adjusted by using the back propagation algorithm and the optimizer to make the loss function converge, and a trained image denoising network model is obtained; wherein the calculation formula of the loss function is as follows:

[0029] L total = λ1L fidelity + λ2L structure + λ3L perceptual + λ4L edge

[0030] In the formula, L fidelity is the fidelity loss, L structure is the structure preservation loss, L perceptual is the perception loss, L edge is the edge preservation loss, and λ i is the weight.

[0031] The third aspect of the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to realize the U-Net3+ image denoising method combining the noise adaptive robust attention mechanism according to the first aspect described above, or the training method according to the second aspect described above.

[0032] The fourth aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a computer processor to realize the U-Net3+ image denoising method combining the noise adaptive robust attention mechanism according to the first aspect described above, or the training method according to the second aspect described above.

[0033] Compared with the prior art, the present application has the following advantages:

[0034] (1) The application effectively improves the recognition ability of complex noise patterns and the ability to maintain image details by introducing a noise adaptive robust attention module in the bottleneck layer of the U-Net3+ architecture. By optimizing the integration of the noise adaptive robust attention mechanism and the U-Net3+ architecture, the global and local noise patterns are more effectively recognized, thereby significantly improving the image denoising performance, especially in complex noise scenarios, which can achieve excellent detail preservation and artifact suppression.

[0035] (2) The model can more effectively capture and utilize the complex dependency between features and enhance the understanding of global context information through the unique adaptive and robust calibration strategy in the proposed noise adaptive robust attention module. Especially in distinguishing noise from real image details. When this attention module is integrated into the bottleneck layer of the U-Net3+ architecture, the network's representation ability in the deepest layer of features can be significantly improved. This enhanced representation is effectively transmitted to each stage of the decoder through the full-scale skip connection of U-Net3+, thereby exhibiting superior performance in image denoising tasks, especially in preserving fine textures, suppressing artifacts, and handling complex noise patterns. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a schematic diagram of the image denoising network model of the application;

[0037] Figure 2 is a schematic diagram of the internal structure of the denoising adaptive robust attention module of the application;

[0038] Figure 3 is a schematic diagram of the feature fusion unit of the application. DETAILED DESCRIPTION

[0039] To make the purpose, technical scheme and advantages of the application clearer and more apparent, the following will further detail the application by combining the embodiments with the drawings. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0040] Example 1:

[0041] The embodiment provides a U-Net3+ image denoising method combined with a noise adaptive robust attention mechanism, and the specific steps are as follows:

[0042] S1: Preprocess the noisy image, input the preprocessed noisy image into the trained image denoising network model for prediction, and obtain the predicted noise component of the noisy image;

[0043] In step S1, the step of pre-processing the noisy image is: pre-processing the noisy image to be processed, such as size adjustment, pixel value normalization to the range [0, 1], etc., consistent with the training stage.

[0044] In step S1, the image denoising network model is constructed by inserting a bottleneck processing module between the encoder module and the decoder module of the U-Net3+ network; the encoder module is used for multi-scale feature extraction of the pre-processed noisy image, generating multi-level encoding features and outputting, the multi-level encoding features include the original bottleneck features output by the deepest layer of the encoding path of the encoder module; the bottleneck processing module includes a denoising adaptive robust attention module and a feature fusion unit, the denoising adaptive robust attention module is used for adaptive learning of noise mode and calibration of attention weight of the original bottleneck features output by the encoder module, to obtain enhanced bottleneck features and output; the feature fusion unit is used for fusing the enhanced bottleneck features with the original bottleneck features output by the encoder module to obtain fused bottleneck features; the decoder module is based on the full-scale skip connection mechanism of the U-Net3+ network, and the fused bottleneck features output by the bottleneck processing module and the multi-level encoding features output by the encoder module are up-sampled and reconstructed to predict the noise component of the input noisy image and output.

[0045] The denoising adaptive robust attention module comprises a first linear transformation layer, a first Softmax normalization layer along a specific dimension, a second element sum-based normalization layer along the specific dimension, and a second linear transformation layer arranged in sequence. The encoder module is composed of five sequentially connected encoding units, and a down-sampling module is arranged between any two encoding units. The decoding module comprises four decoding units corresponding to the first four encoding units in the encoder module.

[0046] Specifically, the image denoising network model schematic diagram is as shown in the figure Figure 1 Based on the U-Net3+ architecture, the image denoising network model comprises an encoder module, a bottleneck processing module integrated with a noise adaptive robust attention module, and a decoder module adopting full-scale skip connection.

[0047] The specific descriptions of the encoder module, the bottleneck processing module, and the decoder module are as follows:

[0048] 1. The encoder module is designed to gradually transform the input noisy image into a low spatial resolution and high channel dimension feature space to capture rich contextual information.

[0049] (1) Structural composition: in this embodiment, the encoder module is composed of five sequentially connected main encoding units (labeled as E1, E2, E3, E4, and E5).

[0050] (2) Intra-encoding unit: Each intra-encoding unit preferably contains a standard convolutional block. The convolutional block can be composed of two 3x3 convolutional layers in series (corresponding to a unetConv2 module at the implementation level). Each convolutional layer can be optionally followed by a BatchNorm2d layer to stabilize the training process and accelerate convergence, and a ReLU layer to introduce non-linear representation capabilities.

[0051] (3) Down-sampling mechanism: Except for the last encoding unit E5 (whose output is the bottleneck layer feature), the output of each encoding unit (E1 to E4) will undergo a down-sampling operation before being passed to the next encoding unit. In this embodiment, a 2x2 MaxPool2d layer is preferably used for down-sampling. Down-sampling not only gradually reduces the spatial size of the feature map, but also effectively increases the receptive field of the subsequent convolutional layer.

[0052] (4) Feature output: After processing by the encoder module, a series of hierarchical feature maps are generated:

[0053] h1: First-level encoding feature generated by the first encoding unit E1.

[0054] h2: Second-level encoding feature generated by the second encoding unit E2.

[0055] h3: Third-level encoding feature generated by the third encoding unit E3.

[0056] h4: Fourth-level encoding feature generated by the fourth encoding unit E4.

[0057] hd5 original : Original bottleneck feature generated by the fifth encoding unit E5, representing the deepest output of the encoding path. The number of channels of these feature maps can be pre-set, for example, filters = [C0, C1, C2, C3, C4], where the channel numbers can be [64, 128, 256, 512, 1024] respectively, corresponding to the features from h1 to hd5 original

[0058] 2. The core responsibility of the bottleneck processing module is to optimize the depth and enhance the information of the original bottleneck feature hd5 original output by the encoder. This module contains the following two key components:

[0059] (1) Denoising adaptive robust attention module, as shown in Figure 2 , this module receives the original bottleneck feature hd5 original ​As input. Its internal mechanism realizes the deep optimization of feature information by adaptively learning the noise characteristics and robustly calibrating the attention weights. The denoising adaptive robust attention module includes a first linear transformation, a first Softmax normalization along a certain dimension (for adaptively identifying important regions), a second element-wise normalization along the certain dimension (for robustly smoothing the attention weights to prevent extreme values from interfering), and a second linear transformation. After processing by this module, the output generates an attention-enhanced enhanced bottleneck feature hd5 ea .

[0060] Specifically, the specific operation process of the denoising adaptive robust attention module for processing the original bottleneck feature is as follows:

[0061] i. Dimension adaptation input: the original bottleneck feature hd5 original is converted into a sequence form X in with a shape of BxNxC bn through a dimension rearrangement operation and a view transformation operation bn , where N = H bn x W mk represents the number of tokens after flattening the spatial dimension;

[0062] ii. First linear transformation (L mk ): map the serialized input feature in to s external units to generate a first intermediate representation A1 where A1 = L dim＝N (X );

[0063] iii. First normalization: apply a Softmax function to A1 along the token dimension N (for each external unit s) to generate a first normalized attention map A2 where A2 = Softmax sumN (A1), such that for each batch b and each memory unit s,

[0064] iv. Second normalization: normalize A2 again along the token dimension N (for each external unit s) to first calculate the element sum S min of A2 along the dimension N sumN (with lower limit truncation ∈ min to ensure numerical stability, obtaining S' sumN ), and then divide A2 by S' sumN to generate a second normalized attention map A3 where (A3) b,n,s = (A2) b,n,s / (S' sumN ) b,s ;

[0065] v. Second linear transformation (L) mv The attention representation A3, after double normalization, is mapped back to the original feature dimension C to obtain the output of the denoised adaptive robust attention module. Where X out =L mv (A3); vi. Dimension Restoration and Fusion Output: The output X of the denoising adaptive robust attention module is... out By performing inverse view transformation and dimension rearrangement operations, the original spatial dimensions are restored, resulting in the attention-enhanced bottleneck feature hd5. ea .

[0066] (2) Feature fusion unit, such as Figure 3 As shown, this unit is responsible for processing the enhanced bottleneck features hd5 obtained by the denoising adaptive robust attention module. ea Compared to the original bottleneck features (HD5) before being input into the bottleneck processing module original Effective fusion was achieved to obtain the fusion bottleneck characteristics (hd5). fused In this embodiment, an element-wise addition fusion strategy is preferably adopted: hd5 fused =hd5 original +hd5 ea This residual connection-based fusion approach helps to fully preserve important information from the original feature stream while incorporating contextual enhancements from the attention mechanism. The fused bottleneck feature is HD5. fused It will serve as a key input in the decoding process.

[0067] 3. The goal of the decoder module is to fuse bottleneck features based on the output of the bottleneck processing module (HD5). fused By combining the multi-scale features (h1, h2, h3, h4) extracted at different levels by the encoder module, and through the unique full-scale skip connection mechanism of U-Net3+, the spatial resolution and fine details of the image are gradually restored, and the noise pattern is finally accurately predicted.

[0068] The decoder module contains four decoding units (labeled D4, D3, D2, and D1). The levels of the decoding units D4, D3, D2, and D1 correspond to the levels of the encoding units E4, E3, E2, and E1 in the encoder, respectively.

[0069] The key of the U-Net3+ architecture lies in the unique input constitution of each unit of the decoder. The input of any decoding unit Dk includes the fused bottleneck feature, the encoding feature output by the encoding unit corresponding to the level of the decoding unit, all encoding features with a spatial resolution greater than the current decoding unit level in the multi-level encoding features output by the encoder module, and the decoding feature output by the previous decoding unit. The processing process of the decoding unit for the input image is as follows: the decoding unit performs channel dimension splicing processing on the input fused bottleneck feature, the encoding feature output by the encoding unit corresponding to the level of the decoding unit, all encoding features with a spatial resolution greater than the current decoding unit level in the multi-level encoding features output by the encoder module, and the decoding feature output by the previous decoding unit, to obtain a wide feature map (the total number of channels of the wide feature map formed after splicing is UpChannels = CatChannels x 5). The wide feature map will undergo one or more convolution layers (in this example, a convolution block containing a 3x3 convolution, an optional batch normalization BN, and a ReLU) for deep information interaction and fusion, and finally obtain the decoding feature of the decoding unit and output it. Taking the decoding unit D4 as an example, the input of the decoding unit D4 is: the fused bottleneck feature (hd5 fused ), the encoding feature h4, and all encoding features with a spatial resolution greater than the current decoding unit level in the multi-level encoding features output by the encoder module (i.e., h1, h2, h3). Among them, all encoding features with a spatial resolution greater than the current decoding unit level in the multi-level encoding features output by the encoder module (i.e., h1, h2, h3) will undergo necessary downsampling operations and a convolution block (containing a convolution layer, an optional batch normalization layer, and a ReLU activation function) before being input into the decoding unit D4. The purpose of this preprocessing is to match the spatial size and the number of channels of the target output of the current decoding unit. The fused bottleneck feature (hd5 fused ) will undergo necessary upsampling operations (for example, bilinear interpolation nn.Upsample) and a convolution block before being input into the decoding unit D4, to match the target spatial size and the number of channels.

[0070] Final output layer: the decoding unit D1 of the top layer of the decoder module (i.e., the spatial resolution is the largest, which is the same as the input image) outputs the final decoding feature hd1. The feature hd1 will undergo a final output convolution layer (for example, outconv1) again, and the convolution layer usually adopts a 3x3 convolution kernel, and the number of output channels is set to be the same as the number of channels of the input noisy image 101 (in this example, the RGB image is 3). The output of this convolution layer is the predicted noise component.

[0071] S2: performing a pixel-by-pixel subtraction operation on the preprocessed noisy image and the predicted noise component to obtain a denoised image. Further, the denoised image can be post-processed by truncating the pixel value range of the denoised image to [0, 1] to ensure the effectiveness and visual effect of the output image, thereby obtaining a final denoised image.

[0072] Embodiment 2

[0073] The embodiment provides a training method of an image denoising network model, and the specific steps are as follows:

[0074] (1) Constructing a training sample set and a verification sample set: the embodiment adopts a public SIDD dataset. The training sample set is preprocessed in the following manner: first, extracting pairs of 512x512 pixel noisy / clean image blocks from the SIDD dataset large-format original image, 10 pairs per scene; second, performing 256x256 pixel random cropping on the extracted image blocks; third, applying random horizontal / vertical flipping and rotation for data augmentation; and finally, converting the image data into a PyTorch tensor and normalizing it to the [0, 1] range. The verification sample set is preprocessed in the following manner: loading the complete original image from the SIDD dataset and performing 512x512 pixel random cropping and normalization;

[0075] (2) Training the pre-constructed image denoising network model using the training sample set, updating the network parameters of the image denoising network model, and obtaining a trained image denoising network model;

[0076] (3) Verifying the trained image denoising network model using the verification sample set, evaluating the performance of the trained image denoising network model, selecting the optimal trained image denoising network model, and completing the training of the image denoising network model; wherein the image denoising network model is any of the image denoising network models in Embodiment 1.

[0077] In step (2), the specific operation of training the pre-constructed image denoising network model is as follows: input the training samples in the training sample set into the image denoising network model for processing, obtain the predicted noise component of the noisy image in the training samples and output it, calculate the fidelity loss between the predicted noise component and the real noise component (obtained by subtracting the clean image from the noisy image); then perform pixel-by-pixel subtraction between the noisy image and the predicted noise component to obtain the denoised image, calculate the structure preservation loss, perceptual loss and edge preservation loss between the denoised image and the clean image, construct a loss function based on the fidelity loss, structure preservation loss, perceptual loss and edge preservation loss; according to the constructed loss function, use the backpropagation algorithm and optimizer to adjust the parameters of the image denoising network model so that the loss function converges, and obtain the trained image denoising network model; wherein, the calculation formula of the loss function is as follows:

[0078] L total =λ1L fidelity +λ2L structure +λ3L perceptual +λ4L edge

[0079] In the formula, L fidelity To preserve fidelity, L structure For structural retention loss, L perceptual To perceive loss, L edge For edge-preserving loss, λ i As weight.

[0080] The specific descriptions of fidelity loss, structure preservation loss, perception loss, and edge preservation loss are as follows:

[0081] Fidelity loss (L) fidelity This example uses the L1 Charbonnier loss function to calculate the pixel-level difference between the noise component predicted by the model and the actual noise component (obtained by subtracting the clean reference image from the noisy image). The Charbonnier loss is a smoothed version of the L1 loss and is less sensitive to outliers.

[0082] Structural retention loss (L) structure This example uses MSSSIM (Multi-Scale Structural Similarity Index) loss. MSSSIM can better measure the similarity of images in terms of structural information, encouraging the denoised image to maintain structural consistency with the clean reference image.

[0083] Perceived loss (L) perceptualThis example uses LPIPS (Learned Perceptual Image Patch Similarity) loss. LPIPS uses a pre-trained deep neural network to extract features and calculates the distance in the feature space, thus better measuring the similarity between the denoised image and the clean image from the perspective of human perception.

[0084] Edge-preserving loss (L edge ): This example designs a custom edge loss function (EdgeLoss). This loss function compares the gradient maps of the denoised image and the clean image (this example uses the Sobel operator to extract edges) to punish the blurring of edges or loss of details that may occur during the denoising process.

[0085] Specifically, the training method of the image denoising network model comprises:

[0086] 1. Adopt a reasonable weight initialization strategy:

[0087] (1) The weights of the standard convolution layers in the encoder module and the decoder module (the convolution operations implemented by the unetConv2 module in this example) are initialized using Kaiming normal distribution. This method is particularly suitable for use with ReLU and its variants of activation functions.

[0088] (2) The weights of the two linear transformation layers (L mk and L mv ) in the denoising adaptive robust attention module are preferably initialized using a normal distribution with a mean of 0 and a small standard deviation (for example, 0.001 or 0.02), and the bias term is usually initialized to 0.

[0089] (3) The scaling factor (weight) of the batch normalization layer (BatchNorm2d) is usually initialized to 1, and the offset (bias) is usually initialized to 0.

[0090] 2. Use an efficient adaptive moment estimation optimizer, this example uses the Adam (Adaptive Moment Estimation) optimizer, and the initial learning rate is set to 1×10^(-4). The cosine annealing learning rate scheduling strategy is used to dynamically adjust the learning rate during the training process. The training process supports checkpoint recovery, training and validation are performed every period, and the best model is saved according to the index on the validation set.

[0091] Verification experiment of the present application:

[0092] To verify the effectiveness and superiority of the image denoising method based on the U-Net3+ architecture and the noise adaptive robust attention mechanism (in the experiment, "UNET3+_NARA") proposed in the present application, a series of comparative experiments were conducted on the published SIDD dataset. The evaluation indicators mainly include average loss (AVG LOSS, the lower the better), peak signal-to-noise ratio (AVG PSNR, unit dB, the higher the better), structural similarity index (AVG SSIM, the higher the better), and learning perceptual image block similarity (AVG LPIPS, the lower the better).

[0093] The specific experimental method is as follows:

[0094] The noisy image was preprocessed, and the preprocessed noisy image was input into different image denoising models for denoising processing. In this experiment, the image denoising models included the following: (1) standard U-Net model ("U-Net"): as a basic U-shaped network comparison baseline; (2) U-Net model integrated with attention ("U-Net_NARA"): the denoising adaptive robust attention module of the present application was integrated in the bottleneck layer of the standard U-Net, to verify the effect of the attention mechanism itself; (3) standard U-Net3+ model ("UNET3+"): as an advanced U-shaped network comparison baseline, to verify the performance of the architecture based on the present application; (4) image denoising network model proposed in the present application ("UNET3+_NARA"). The performance of each model on the SIDD dataset is shown in Table 1.

[0095] Table 1: Comparison table of model performance on SIDD dataset.

[0096] Model Average loss Peak signal-to-noise ratio (dB) Structural similarity index Learning perceptual image patch similarity U-Net 0.0219 38.83 0.9395 0.0549 U-Net_NARA 0.0199 38.83 0.9444 0.0365 UNET3+ 0.0196 39 0.9472 0.0443 UNET3+_NARA 0.0177 39.96 0.955 0.0353

[0097] As can be seen from Table 1, the method of the present application (UNET3+_NARA) performs best in all key performance indicators. Compared with the standard U-Net3+, the method of the present application has significant improvements in PSNR, SSIM and LPIPS indicators, indicating that it has obvious improvements in detail preservation, structural similarity and perceptual quality, fully proving the excellent performance and superiority of the method of the present application in the image denoising task. The significant improvement of the method of the present application: compared with all other comparative methods, the UNET3+_NARA model of the present application has obvious advantages in all indicators. For example, compared with the standard U-Net3+, the PSNR is significantly improved, and the LPIPS is significantly reduced, indicating that the denoised image is not only better in objective indicators, but also closer to the real image in perceptual quality, better in detail preservation, and less in artifacts.

[0098] Example 3:

[0099] The embodiment provides an electronic device, including a memory and a processor, the memory stores a computer program, and the processor implements the U-Net3+ image denoising method with a noise-adaptive robust attention mechanism as described in the above embodiment 1 or the training method of the image denoising network model as described in the above embodiment 2 when the computer program is executed.

[0100] The hardware of the electronic device in the embodiment further includes a GPU, a display buffer memory, a RAMD / A converter and a heat sink cooperating with the processor; the GPU is responsible for processing the graphic display of the electronic device, provides image rendering and acceleration functions, and accelerates large-scale data-intensive tasks by using the parallel computing advantage.

[0101] Further, the U-Net3+ image denoising method with a noise-adaptive robust attention mechanism as described in the above embodiment 1 or the training method of the image denoising network model as described in the above embodiment 2 can be implemented as a computer software program. For example, the embodiment includes a computer program product including a computer program carried on a computer readable medium, and the computer program includes program codes for executing the method. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a detachable medium. When the computer program is executed by the processor, the above functions defined in the method of the application are executed.

[0102] Embodiment 4:

[0103] The embodiment provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the U-Net3+ image denoising method with a noise-adaptive robust attention mechanism as described in the above embodiment 1 or the training method of the image denoising network model as described in the above embodiment 2.

[0104] The computer readable medium described in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal that propagates in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take many forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer the program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted in any suitable form, including but not limited to, wireless, wire line, optical, RF, or any suitable combination thereof.

[0105] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object oriented programming languages, such as python, C++, as well as conventional procedural programming languages or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0106] The computer readable storage medium of the embodiment can be accelerated by hardware such as a GPU, and the parallel computing advantage of the GPU is used to accelerate the U-Net3+ image denoising method with the adaptive robust attention mechanism combined with noise as described in Embodiment 1 above, or the training method of the image denoising network model as described in Embodiment 2 above.

[0107] The above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the protection scope of the present application. Based on the idea of the present application, those skilled in the art can modify or equivalently replace the technical solutions of the present application without departing from the essence and scope of the technical solutions of the present application.

Claims

1. A U-Net3+ image denoising method combining a noise-adaptive robust attention mechanism, characterized in that, Includes the following steps: S1: Preprocess the noisy image, input the preprocessed noisy image into the trained image denoising network model for prediction, and obtain the predicted noise component of the noisy image. S2: Subtract the predicted noise component from the preprocessed noisy image pixel by pixel to obtain the denoised image; The image denoising network model is constructed by inserting a bottleneck processing module between the encoder and decoder modules of the U-Net3+ network. The encoder module is used to extract multi-scale features from the preprocessed noisy image, generate multi-level encoded features, and output them. The multi-level encoded features include the original bottleneck features output from the deepest layer of the encoder module's encoding path. The bottleneck processing module includes a denoising adaptive robust attention module and a feature fusion unit. The denoising adaptive robust attention module is used to adaptively learn noise patterns and calibrate attention weights on the original bottleneck features output by the encoder module to obtain enhanced bottleneck features and output them. The feature fusion unit is used to fuse the enhanced bottleneck feature with the original bottleneck feature output by the encoder module to obtain the fused bottleneck feature; the decoder module, based on the full-scale skip connection mechanism of the U-Net3+ network, upsamples and reconstructs the fused bottleneck feature output by the bottleneck processing module and the multi-level coding feature output by the encoder module, predicts the noise component of the input noisy image and outputs it.

2. The U-Net3+ image denoising method combining noise adaptive robust attention mechanism according to claim 1, characterized in that, The denoising adaptive robust attention module includes a first linear transformation layer, a first Softmax normalization layer, a second element-sum-based normalization layer, and a second linear transformation layer arranged sequentially.

3. The U-Net3+ image denoising method combining noise adaptive robust attention mechanism according to claim 1, characterized in that, The feature fusion unit fuses the enhanced bottleneck features with the original bottleneck features output by the encoder module by adding elements one by one.

4. The U-Net3+ image denoising method combining noise adaptive robust attention mechanism according to claim 1, characterized in that, The encoder module consists of five sequentially connected encoding units, with a downsampling module between any two encoding units; each encoding unit includes two convolutional layers, and each convolutional layer is followed by a batch normalization layer and a nonlinear activation function ReLU layer.

5. The U-Net3+ image denoising method combining noise adaptive robust attention mechanism according to claim 1, characterized in that, The decoding module includes four decoding units corresponding to the first four encoding unit levels in the encoder module. The input of each decoding unit includes the fusion bottleneck feature, the encoding feature output by the encoding unit corresponding to the decoding unit level, all encoding features with spatial resolution greater than the current decoding unit level in the multi-level encoding features output by the encoder module, and the decoding feature output by the previous decoding unit. The processing procedure of the decoding unit for the input image is as follows: the decoding unit performs channel-dimensional concatenation processing on the input fusion bottleneck feature, the encoding feature output by the encoding unit corresponding to the decoding unit level, all encoding features with spatial resolution greater than the current decoding unit level in the multi-level encoding features output by the encoder module, and all decoding features with spatial resolution less than the current decoding unit level in the multi-level encoding features output by the decoder module to obtain a wide feature map. Then, a convolution block is used to perform convolution processing on the wide feature map to obtain the decoded features and output them.

6. The U-Net3+ image denoising method combining noise adaptive robust attention mechanism according to claim 1, characterized in that, After obtaining the denoised image in step S2, the denoised image needs to be post-processed. The specific operation of the post-processing is to truncate the denoised image to the pixel value range of [0,1] to obtain the final denoised image.

7. A training method for an image denoising network model, characterized in that, Includes the following steps: (1) Construct a training sample set and a validation sample set. The training sample set is constructed by extracting pairs of noisy / clean image patches from the large-format original images of the SIDD dataset and performing random cropping, data augmentation and normalization on the extracted image patches. The verification sample set is obtained by loading complete original images from the SIDD dataset and performing random cropping and normalization on the original images; (2) The pre-constructed image denoising network model is trained using the training sample set, and the network parameters of the image denoising network model are updated to obtain the trained image denoising network model. (3) The trained image denoising network model is validated using the validation sample set, the performance of the trained image denoising network model is evaluated, the optimal trained image denoising network model is selected, and the training of the image denoising network model is completed; wherein, the image denoising network model is any one of the image denoising network models described in claims 1-6.

8. The training method for the image denoising network model according to claim 7, characterized in that, The specific operation of step (2) is as follows: input the training samples in the training sample set into the image denoising network model for processing, obtain the predicted noise component of the noisy image in the training sample and output it, and calculate the fidelity loss between the predicted noise component and the real noise component. Then, the noisy image is subtracted pixel-by-pixel from the predicted noise component to obtain the denoised image. The structure-preserving loss, perceptual loss, and edge-preserving loss are calculated between the denoised image and the clean image. A loss function is constructed based on these losses. According to the constructed loss function, the parameters of the image denoising network model are adjusted using backpropagation and an optimizer until the loss function converges, resulting in the trained image denoising network model. The formula for calculating the loss function is as follows: L total =λ1L fidelity +λ2L structure +λ3L perceptual +λ4L edge In the formula, L fidelity To preserve fidelity, L structure For structural retention loss, L perceptual To perceive loss, L edge For edge-preserving loss, λ i For weights.

9. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the U-Net3+ image denoising method with noise adaptive robust attention mechanism as described in any one of claims 1-6, or the training method as described in claim 7 or 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computer processor, implements the U-Net3+ image denoising method combining a noise-adaptive robust attention mechanism as described in any one of claims 1-6, or the training method as described in claim 7 or 8.

Images