Joint denoising training method combining noise-free image and noise image

Abstract

The invention discloses a joint denoising training method combining a noiseless image and a noise image. The method comprises the following steps: S1, training an image denoising model: training the image denoising model by using the noiseless image and the corresponding noise image as training data; s2, data enhancement: performing multiple enhancement operations on the input image, wherein the enhancement operations comprise at least one of rotation, overturning, cutting and color change; s3, in a model training process, optimizing the network by using a mixed loss function; s4, implementing a dynamic optimization strategy in the training process: dynamically adjusting the learning rate of the model according to the current denoising effect so as to further improve the convergence speed and denoising performance of the model; and S5, updating the parameters by adopting a self-adaptive optimizer in the model optimization process. Based on the method provided by the invention, the general denoising performance of the image denoising model can be further improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and relates to a joint denoising training method combining a noise-free image and a noise image. Background Art

[0002] Image denoising is a basic step in image processing and the basis of advanced computer vision tasks such as object detection and image segmentation. It aims to recover high-quality images from noisy images. During the acquisition and transmission process, images are often corrupted by noise interference, which seriously degrades the visual quality of the acquired images. During the acquisition and processing of images, various types of noise may exist, including Gaussian noise, Poisson noise, JPEG compression noise, etc., making image denoising a very challenging task. Since the image degradation process is very complex and it is difficult to accurately characterize the noise information, restoring a clean image is a difficult task. In the field of computer vision, image denoising has been a research hotspot since the 1990s.

[0003] Most traditional denoising methods are algorithm-based. With the development of deep learning, convolutional neural network (CNN) methods have also become the mainstream in the field of image denoising, such as DnCNN, which has achieved great performance improvements in image denoising. Nevertheless, there are still some limitations to methods based on convolutional neural networks. Since the convolution kernel is usually fixed, it means that using the same convolution kernel to process different image regions may not be the best choice. In addition, the convolution kernel can be regarded as a small window for capturing local information, which means that global information is easily lost when trying to model the relationship between distant pixels.

[0004] On the other hand, the introduction of VIT pioneered the introduction of Transformer into the field of computer vision. Due to its ability to effectively capture global contextual information in images and adaptively focus on pixels at different positions, in more and more computer vision tasks, including image segmentation, object detection, restoration, and super-resolution, methods based on Transformer have surpassed methods based on convolutional neural networks (CNNs) and reached the state-of-the-art level. However, training the Transformer model requires a large number of parameters, especially when processing high-resolution data such as images, and the boundary pixels of the image are difficult to be fully utilized in the Transformer, which may lead to inaccurate feature extraction near the boundary. Therefore, how to apply Transformer, especially in low-level computer vision tasks, remains a challenge. Summary of the invention

[0005] The object of the present invention is to provide a joint denoising training method combining noise-free images and noisy images, and to use a hybrid loss function to further improve the denoising performance of the image denoising model.

[0006] A joint denoising training method combining a noise-free image and a noisy image, characterized by comprising the following steps:

[0007] S1. Training of image denoising model:

[0008] Use the noise-free images and the corresponding noisy images as training data to train the image denoising model;

[0009] S2. Data enhancement:

[0010] Perform multiple enhancement operations on the input image, including at least one of rotation, flipping, cropping, and color change, and then input the enhanced input image into the image denoising model for training to increase the robustness and generalization ability of the image denoising model;

[0011] S3. During the model training process, the network is optimized using a hybrid loss function;

[0012] The hybrid loss function consists of two parts: the first loss function is composed of the pixel-level loss of the denoised noisy image and the real clean image; the second loss function is composed of the pixel-level loss of the denoised noise-free image and the real clean image;

[0013] S4. Implement dynamic optimization strategies during training:

[0014] According to the current denoising effect, the learning rate of the model is dynamically adjusted to further improve the convergence speed and denoising performance of the model;

[0015] S5. Adaptive optimizer is used to update parameters during model optimization:

[0016] After each training iteration, the model performance is monitored by the validation set performance indicators (such as peak signal-to-noise ratio PSNR); when the validation set performance indicators tend to stabilize after multiple iterations and no longer improve significantly, or when the preset training rounds are reached, the training process is terminated.

[0017] In step S1, the training process uses the noise-free image and the noisy image data with random Gaussian noise added as input, and uses the corresponding real clean image as a label to optimize the image denoising model through supervised learning.

[0018] In step S1, the image denoising model is a graph named SUNet (Swin Transformer UNet)

[0019] Like denoising models;

[0020] SUNet is an image denoising model that combines the Swin Transformer with the UNet architecture. The model consists of three modules: shallow feature extraction module, UNet feature extraction module, and image reconstruction module.

[0021] (1) Shallow feature extraction module:

[0022] This module adjusts the feature dimension of the input image through a 3×3 convolutional layer and extracts the low-level features of the image; this process can be expressed by the following formula:

[0023] F shallow =M SFE (I)

[0024] in, represents the image to be denoised, H and W represent the height and width of the image respectively, and 3 represents the number of channels of the image (i.e., RGB channels); M SFE (·) represents the shallow feature extraction layer, which is the process of extracting features from the image through convolution operation;

[0025] Represents shallow features, and C represents the number of channels of shallow features (set to 96 in the present invention. 96 channels refer to the depth of the feature map generated by the convolution operation. The selection of 96 channels aims to balance the computational complexity and feature expression ability, ensuring that sufficient low-level features can be effectively extracted for subsequent processing.).

[0026] (2) UNet feature extraction module:

[0027] This module adopts the encoder-decoder structure as the main body;

[0028] The encoder gradually reduces the spatial resolution of the image through multiple Swin Transformer blocks and downsampling layers (such as PatchMerging layers) and extracts deep features; the decoder gradually restores the spatial resolution of the image through multiple Swin Transformer blocks and upsampling layers, while combining skip connections to retain shallow feature information.

[0029] Swin Transformer is a transformer model based on sliding window attention mechanism, which can effectively capture local and global feature information in images. Each Swin Transformer block consists of 8 Swin Transformer layers, each of which performs self-attention calculation in a local range through the sliding window mechanism, and gradually increases the receptive field using a hierarchical structure to capture multi-scale contextual information.

[0030] Will Input to UNet feature extraction M VFE In (·), the process of extracting high-level and multi-scale deep features can be expressed as:

[0031] F deep =M VFE (F shallow )

[0032] Among them, deep features Consistent with the shallow feature dimension, C is the number of channels (96 in the present invention).

[0033] (3) Image reconstruction module

[0034] This module uses a 3×3 convolutional layer M R (·), generate the denoised image based on the deep features; the final image reconstruction process is expressed by the following formula:

[0035]

[0036] in, Represents the final output of the model, H and W represent the height and width of the image respectively, and 3 represents the number of channels of the image (i.e., RGB channels). R (·) represents the image reconstruction layer, which is the convolution operation from the deep feature F deep Reconstruct the denoised image;

[0037] It is worth mentioning that the convolutional layer in the shallow feature extraction module is used to extract shallow features (i.e., low-level features of the image), while the convolutional layer in the reconstruction module is used to generate the final denoised image from deep features. Although the two convolutional layers have the same structure, both are 3×3 convolutional layers, their input and output dimensions are different. The former processes low-level features, while the latter generates the final denoised image. The convolution operation and implementation of the two are similar, but their functions are different. The former is designed to extract features, while the latter is used to reconstruct images.

[0038] In step S2, the data enhancement operation includes image rotation angle, contrast adjustment, brightness change and random cropping, and the enhanced image is used to further improve the generalization ability of the model.

[0039] In step S3, the hybrid loss function is composed of the first loss function and the second loss function:

[0040] First loss function: Input the noisy image into the denoised image generated by the image denoising model, and then calculate the first loss value based on the pixel value of the real sample image:

[0041]

[0042] in, is the denoised image output by the model, I $％ea( is a real clean image; ‖·‖ refers to the L1 norm operation, that is, calculating the absolute error of the image at each pixel position, and adding all these errors to obtain the image loss value;

[0043] Second loss function: Input the clean image into the denoised image generated by the model, and then calculate the second loss based on the pixel value of the real sample image:

[0044]

[0045] in, is the clean image after denoising, I $％ea( For a real clean image;

[0046] The expression of the mixed loss function is:

[0047]

[0048] Among them, λ1 and λ ρ is a hyperparameter used to control the weights of the two loss functions. ρ Both are set to 1.

[0049] In step S4, the dynamic optimization strategy refers to adjusting the learning rate according to the denoising effect of the model during training to avoid overfitting and further improve the stability of the model;

[0050] First, the warm-up strategy is adopted; in the warm-up phase, the learning rate is increased from η . (an initially small value) gradually increases until the preset initial learning rate η is reached 0a12345 , and keep this value until the end of the warm-up phase. The learning rate adjustment formula is:

[0051]

[0052] Among them, η6 is the learning rate of the current training round t step, η wa9:3；p represents the target learning rate at the end of the warm-up phase, which is set to 2e-4 in the present invention, and η. is the learning rate at the beginning of the warm-up phase, usually set to 1e-6; T 0a12345 is the total number of training rounds in the warm-up phase, which is set to 3 in the present invention.

[0053] After the warm-up phase, the learning rate enters the cosine annealing phase, and the learning rate gradually decreases in the manner of a cosine function as the training progresses; the learning rate adjustment formula for the cosine annealing strategy is:

[0054]

[0055] Among them, η6 is the learning rate of the current training round t step, η 2i( is the minimum learning rate during training (generally set to 1e-6), η wa9:3；p is the target learning rate at the end of the warm-up phase; T HIHa％ is the total number of training rounds, T 0a12345 is the total number of training rounds in the warm-up phase, which are set to 200 and 3 respectively in the present invention. The cosine annealing strategy will increase the learning rate from η wa9:3；p Start and gradually decrease to η 2i( , helping the optimizer to adjust parameters more finely when approaching the optimal solution.

[0056] In step S5, the adaptive optimizer is an Adam optimizer, which uses a dynamic adjustment of the learning rate and a weight decay strategy to further improve the training efficiency and denoising effect of the model. The update formula of the Adam optimizer is as follows:

[0057]

[0058] Among them, θ6 is the model parameter of the current training round t step, θ6 is the parameter that needs to be optimized in the network (such as the convolution kernel weight in the shallow feature extraction module and the image reconstruction module, the weight matrix and bias term of the SwinTransformer self-attention layer in the UNet feature extraction module, etc.), η6 is the current learning rate, and are the first-order and second-order moment estimates of the gradient, ε is a very small constant to prevent division by zero errors, and ε is usually taken as 1e-8.

[0059] First moment estimate and the second moment estimate The update rules are as follows:

[0060] m6=β1m 631 +(1-β1)g6

[0061] v6=β ρ v 631 +(1-β ρ )g6 ρ

[0062]

[0063] Among them, g6 is the gradient of the loss function relative to the model parameters when the training round is t; this gradient reflects the direction of change and adjustment amplitude of the model parameters in the current round, and is an important basis for the optimization algorithm (Adam optimizer) to update the model parameters. Specifically, as the training rounds progress, the model parameters (such as the convolution kernel weights in the shallow feature extraction module and the image reconstruction module, the weight matrix and bias term of the SwinTransformer self-attention layer in the UNet feature extraction module, etc.) will be dynamically adjusted to gradually minimize the loss function. The gradient of the loss function relative to the model parameters is calculated by the back-propagation algorithm, which is used to indicate how each parameter is adjusted in the current round to reduce the training error and thus improve the model performance. β1 and β c is the decay rate hyperparameter of the first-order and second-order moment estimates, usually taken as β1 = 0.9 and β ρ =0.999.

[0064] Through the back-propagation algorithm, the optimizer is used to continuously update the model parameters and iteratively optimize the model until the best denoising effect is achieved. In addition, model regularization techniques (such as L2 regularization, Dropout, etc.) can be used during the training process to further prevent overfitting and make the model more robust and reliable on the test data.

[0065] Beneficial effects:

[0066] The present invention combines a joint denoising training method of noise-free images and noise images. Specifically, it includes the following steps: S1. Training of an image denoising model, using noise-free images and corresponding noise images as training data to train the image denoising model; S2. Data enhancement, performing multiple enhancement operations on the input image, and inputting the enhanced image into the model for training to increase the robustness and generalization ability of the model; S3. Using a mixed loss function, by adding the pixel-level loss of the denoised noise-free image and the real clean image to the calculation process of the mixed loss function, the denoising effect and detail restoration ability of the model are improved; S4. Dynamic optimization, adjusting the learning rate according to the denoising effect of the model during the training process to improve the denoising effect; S5. Model optimization, using an adaptive optimizer to update the model parameters, iterative optimization, until the best denoising effect is achieved. Based on the existing image denoising model, the present invention considers that the model may eliminate the noise that does not exist in the image, and proposes a joint denoising training method combining noise-free image and noisy image, so that the proposed image denoising model can ensure the reasonable convergence position of the output image in the image solution space, thereby obtaining a denoised image that is as close to the real sample image as possible.

[0067] Compared with the prior art, the present invention has the following advantages:

[0068] The purpose of the present invention is to provide a joint denoising training method combining noise-free images and noisy images. Compared with the prior art, the present invention fully considers the influence of noise-free images on the image denoising effect, and effectively avoids the model from removing noise that does not originally exist in the image during the denoising process by adding the pixel-level loss of the denoised noise-free image and the real clean image to the calculation process of the loss function, thereby reducing the suppression of irrelevant noise. This method can further improve the denoising performance of the image denoising model, enhance the robustness of the model to real noise, and thus improve the quality of the denoised image.

[0069] Based on the method proposed in the present invention, the general denoising performance of the image denoising model can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the overall process of a joint denoising training method combining noise-free images and noisy images provided by the present invention.

[0071] Figure 2 is a flow chart of the method of the present invention.

[0072] Figure 3 It is a partial visual display of the denoising result of the present invention. DETAILED DESCRIPTION

[0073] The present invention will be further described below in conjunction with specific embodiments and accompanying drawings:

[0074] Example 1

[0075] Task Definition

[0076] Image denoising aims to restore a clean image X as close to the original as possible from a noisy image Y. Specifically, given a noisy image Y, the task goal is to output a denoised image xQ = D(Y) through a denoising model D(·) so that the denoised image as similar as possible to the original clean image X.

[0077] The object of the present invention is to provide a joint denoising method combining a noise-free image and a noisy image to further improve the visual effect of image quality.

[0078] Implementation steps

[0079] like Figures 1-2 As shown, the present invention provides a joint denoising method combining a noise-free image and a noisy image, comprising the following steps:

[0080] Step 1: Training of image denoising model:

[0081] The image denoising model is trained using the noise-free image and the corresponding noisy image as training data. The denoising model is an image denoising model called SUNet (Swin Transformer UNet) proposed in the literature Chi-Mao Fan, Tsung-Jung Liu, and Kuan-Hsien Liu. Sunet: swin transformer unet for image denoising. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS), pages 2333–2337. IEEE, 2022. Specifically, SUNet is an image denoising model that combines Swin Transformer with UNet architecture. The model consists of three modules: shallow feature extraction module, UNet feature extraction module, and image reconstruction module.

[0082] Shallow feature extraction module. This module adjusts the feature dimension of the input image through a 3×3 convolution layer and extracts the low-level features of the image. The process can be expressed by the following formula:

[0083] F shallow =M SFE (I)

[0084] in, represents the image to be denoised, H and W represent the height and width of the image respectively, and 3 represents the number of channels of the image (i.e., RGB channels); represents shallow features, and C represents the number of channels of shallow features (set to 96 in the present invention).

[0085] UNet feature extraction module. This module is mainly based on the encoder-decoder structure. The encoder gradually reduces the image spatial resolution through multiple Swin Transformer blocks and downsampling layers (such as Patch Merging layer) and extracts deep features; the decoder gradually restores the image spatial resolution through multiple Swin Transformer blocks and upsampling layers, and combines skip connections to retain shallow feature information.

[0086] Swin Transformer is a transformer model based on sliding window attention mechanism, which can effectively capture local and global feature information in images. Each Swin Transformer block consists of 8 Swin Transformer layers, each of which performs self-attention calculation in a local range through the sliding window mechanism, and gradually increases the receptive field using a hierarchical structure to capture multi-scale contextual information.

[0087] Will Input to UNet feature extraction M VFE In (·), the process of extracting high-level and multi-scale deep features can be expressed as:

[0088] F deep =M VFE (F shallow )

[0089] in, Consistent with the shallow feature dimension, C is the number of channels (96 in the present invention).

[0090] Image reconstruction module. This module uses a 3×3 convolutional layer to generate a denoised image from deep features. The final image reconstruction process is expressed by the following formula:

[0091]

[0092] in, Represents the final output of the model, H and W represent the height and width of the image respectively, and 3 represents the number of channels of the image (i.e., RGB channels).

[0093] It is worth mentioning that the convolution layer in the shallow feature extraction is used to extract shallow features (i.e., low-level features of the image), while the convolution layer in the reconstruction module is used to generate the final denoised image from the deep features. Although the two convolution layers have the same structure, both are 3×3 convolution layers, their input and output dimensions are different. The former processes low-level features, while the latter generates the final denoised image. The convolution operation and implementation of the two are similar, but their functions are different. The former is designed to extract features, while the latter is used to reconstruct images.

[0094] Step 2. Data enhancement: perform multiple enhancement operations on the input image:

[0095] During the training process, low-quality images (noise images) and corresponding high-quality images (clean images) are first padded to ensure that the image size is consistent with the model input requirements and to avoid information loss at the edge of the image. Next, a random cropping operation is performed to crop images of different regions, which increases the diversity of training samples and enables the model to learn denoising features at different image locations. Furthermore, the diversity of training data is enhanced through random geometric transformations, such as horizontal flipping and rotation, so that the model can adapt to images of different directions and angles. On this basis, in order to simulate the Gaussian noise environment in reality, noise of different intensities from σ = 5 to σ = 50 is added to the real clean images to form a noisy image set, ensuring that the model can process images of various noise intensities.

[0096] Step 3. During model training, use the mixed loss function to optimize the network:

[0097] First, a first loss value is determined according to the pixel value of the denoised image generated by inputting the noisy image into the model, the pixel value of the real clean image, and the first loss function. The first loss function is exemplarily a mean absolute error loss function, which can be expressed as:

[0098]

[0099] in, is the denoised image output by the model, I $％ea( A real clean image.

[0100] Then, the second loss value is determined according to the pixel value of the denoised image generated by inputting the noise-free image into the model, the pixel value of the real sample image, and the second loss function. The second loss function is exemplarily a mean absolute error loss function, which can be expressed as:

[0101]

[0102] in, is the noise-free image after denoising, I $％ea( A real clean image.

[0103] Next, the loss function value may be determined according to the first loss value and the second loss value. In an embodiment of the present invention, the first loss value and the second loss value are summed to obtain the loss function value:

[0104]

[0105] Among them, λ1 and λ ρ are hyperparameters (in the embodiment, both are set to 1) used to control the weights of the two parts of the loss function.

[0106] Step 4: Dynamic optimization, dynamically adjust the learning rate of the model according to the denoising effect of the model during training:

[0107] In order to improve the stability in the early stage of training and gradually increase the learning rate, a warm-up strategy was adopted in the early stage of training to gradually increase the learning rate to a predetermined value; in the later stage of training, a cosine annealing strategy was adopted to gradually reduce the learning rate to smooth the convergence process of the model and avoid overfitting.

[0108] Specifically, the warm-up phase strategy is first adopted. In the warm-up phase, the learning rate gradually increases from an initial small value until it reaches the preset initial learning rate η 0a12345 , and keep this value until the end of the warm-up phase. The learning rate adjustment formula is:

[0109]

[0110] Among them, η6 is the learning rate of the current training round t step, η wa9:3；p represents the target learning rate at the end of the warm-up phase, which is set to 2e-4 in the present invention, and η . is the learning rate at the beginning of the warm-up phase, usually set to 1e-6. 0a12345 is the total number of training rounds in the warm-up phase, which is set to 3 in the present invention.

[0111] After the warm-up phase, the learning rate enters the cosine annealing phase, and the learning rate gradually decreases in the manner of a cosine function as the training progresses. The learning rate adjustment formula for the cosine annealing strategy is:

[0112]

[0113] Among them, η6 is the learning rate of the current training round t step, η 2i( is the minimum learning rate during training, generally set to 1e-6, η wa9:3；p is the target learning rate at the end of the warm-up phase; T HIHa％ is the total number of training rounds, T 0a12345 is the total number of training rounds in the warm-up phase, which are set to 200 and 3 respectively in the present invention. The cosine annealing strategy will increase the learning rate from η wa9:3；p Start and gradually decrease to η 2i( , helping the optimizer to adjust parameters more finely when approaching the optimal solution.

[0114] S5. Model optimization, using adaptive optimizer to update model parameters:

[0115] First, select an optimizer based on the Adam optimization algorithm, which is a widely used adaptive optimization method that can automatically adjust the step size of parameter updates during model training, thereby improving training efficiency and convergence performance. Specifically, the update formula of the Adam optimizer is as follows:

[0116]

[0117] Among them, θ6 is the model parameter of the current training round t step, η6 is the current learning rate, and is the first-order and second-order moment estimate of the gradient, ε is a very small constant to prevent division by zero errors, usually taken as 1e-8.

[0118] First moment estimate and the second moment estimate The update rules are as follows:

[0119] m6=β1m 631 +(1-β6)g6

[0120] v6=β ρ v 631 +(1-β ρ )g6 ρ

[0121]

[0122] Among them, g6 is the gradient of the current training round t, β1 and β ρ is the decay rate hyperparameter of the first-order and second-order moment estimates, usually taken as β1 = 0.9 and β ρ =0.999.

[0123] In each round of training, the optimizer uses the back-propagation algorithm to calculate the gradient based on the model's loss function and updates the model's weight parameters. As training progresses, the learning rate is gradually adjusted to achieve more refined optimization and further improve the image denoising effect. Through these optimization processes, the model can gradually approach the optimal state and ultimately output the optimal image denoising model.

[0124] Experimental Results

[0125] 1. Dataset

[0126] To evaluate the effectiveness of the method, the denoising model was trained end-to-end on the DIV2K dataset, and the image denoising performance was evaluated on two commonly used Kodak and BSD datasets.

[0127] DIV2K dataset: This is a popular dataset for image processing research. The dataset contains high-resolution images from various fields, with a resolution of usually 2K (2048x1080 pixels) or higher. The images in the dataset cover a variety of different scenes, including natural scenery, urban scenery, indoor photos, etc. This diversity makes DIV2K suitable for a variety of image processing tasks.

[0128] Kodak dataset: This dataset was released by Eastman Kodak in 1994 and contains 24 standard test images for evaluating image processing and image compression algorithms. These images cover different scenes and themes, including people, natural scenery, and indoor scenes, and the resolution of each image is 768×512 pixels.

[0129] BSD dataset: This dataset is a commonly used dataset for image segmentation and computer vision tasks. It was first created by the Computer Vision Research Group at the University of California, Berkeley (UC Berkeley) to provide a wide range of natural scene images for research and evaluation of image segmentation and vision-related tasks.

[0130] 2. Evaluation indicators

[0131] Evaluation indicators in the field of image denoising are used to quantitatively evaluate the performance of denoising algorithms. The following are two common image denoising evaluation indicators and their calculation formulas. PSNR is one of the indicators to measure image quality and is used to indicate the quality of an image. It is the ratio of the maximum possible peak value of the original image to the MSE (mean square error). The higher the better.

[0132]

[0133] Where MAX is the maximum pixel value of the image, and the unit of PSNR is dB.

[0134] SSIM measures the structural similarity between two images, including brightness, contrast, and structure. The value range of SSIM is between -1 and 1, and the closer to 1, the higher the image quality.

[0135]

[0136] Among them, μ represents the mean, σ represents the standard deviation, and σ xy is the covariance of x and y, c1 and c2 are constants to maintain stability, usually (0.01×L) 2 and (0.03×L) 2 , where L is the dynamic range of pixel values.

[0137] 3. Performance comparison

[0138] In this section, the performance of the proposed model, the baseline model, and the classic image denoising algorithm are compared on two datasets:

[0139] CBM3D[Kostadin Dabov,Alessandro Foi,Vladimir Katkovnik,and Karen Egiazarian.Image denoising by sparse 3-d transform-domain collaborative filtering.IEEE Transactions on image processing,16(8):2080–2095,2007.]: An image denoising method based on block matching and three-dimensional transform filtering, which performs collaborative denoising by similar image blocks.

[0140] DNCNN[Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian de-noiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, 26(7): 3142–3155, 2017.]: An end-to-end image denoising method based on convolutional neural networks. It efficiently extracts noise features through residual learning and batch normalization, and achieves robust denoising for multiple noise levels.

[0141] FFDNet[Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward a fast and flexible solution for cnn-based image denoising. IEEE Transactions on ImageProcessing, 27(9): 4608–4622, 2018]: An efficient and flexible image denoising method that introduces an adjustable noise level map and a multi-scale architecture to balance denoising performance and processing speed. It is suitable for denoising tasks with different noise intensities and image resolutions.

[0142] IRCNN[Eirikur Agustsson and Radu Timofte.Ntire 2017challenge onsingle image super-resolution:Dataset and study.In Proceedings of the IEEE conference on computer vision and pattern recognition workshops,pages 126–135,2017]: An iteratively optimized image denoising method that combines deep convolutional networks and image priors to gradually reduce noise in the form of residual learning to achieve efficient image restoration.

[0143] SUNet(Baseline)[Chi-Mao Fan,Tsung-Jung Liu,and Kuan-Hsien Liu.Sunet:swin transformer unet for image denoising.In 2022IEEE International Symposiumon Circuits and Systems(ISCAS),pages 2333–2337.IEEE,2022.]: A method for image denoising using SwinTransformer combined with UNet architecture.

[0144] Table 1 Performance comparison of different methods on the CBSD68 dataset

[0145]

[0146] Table 2 Performance comparison of different methods on Kodak24 dataset

[0147]

[0148]

[0149] In the CBSD68 dataset, when the noise level is σ=15, the PSNR of the present invention is improved by 0.33dB compared with the SUNet baseline model. When the noise level is σ=50, the present invention still maintains its superiority, with a PSNR of 28.00dB and an SSIM of 0.8012, which are 0.24dB and 0.0087 higher than the baseline model, respectively. In addition, the experimental results on the Kodak24 dataset further verify the effectiveness of the method of the present invention. At all noise levels, the present invention outperforms the traditional denoising model, the CNN method and the SUNet baseline model, fully demonstrating the advantages of image denoising training based on the hybrid loss function.

[0150] 4. Qualitative analysis

[0151] The visualization of some denoising results of the present invention on the CBSD test set shows the denoising results after denoising, such as Figure 3 It can be found that the present invention can effectively suppress artifacts and show sharper image edges.

[0152] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A joint denoising training method combining noise-free images and noisy images, characterized in that: The following steps are involved: S1. Training of image denoising model: Use the noise-free images and the corresponding noisy images as training data to train the image denoising model; S2. Data enhancement: Perform multiple enhancement operations on the input image, including at least one of rotation, flipping, cropping, and color change, and then input the enhanced input image into the image denoising model for training to increase the robustness and generalization ability of the image denoising model; S3. During the model training process, the network is optimized using a hybrid loss function; The hybrid loss function consists of two parts: the first loss function is composed of the pixel-level loss of the denoised noisy image and the real clean image; the second loss function is composed of the pixel-level loss of the denoised noise-free image and the real clean image; S4. Implement dynamic optimization strategies during training: According to the current denoising effect, the learning rate of the model is dynamically adjusted to further improve the convergence speed and denoising performance of the model; S5. Adaptive optimizer is used to update parameters during model optimization: After each training iteration, the model performance is monitored through the validation set performance indicators; when the validation set performance indicators tend to stabilize after multiple iterations and no longer improve significantly, or when the preset training rounds are reached, the training process is terminated.

2. A joint denoising training method combining noise-free images and noisy images according to claim 1, characterized in that: In step S1, the training process uses the noise-free image and the noisy image data with random Gaussian noise added as input, and uses the corresponding real clean image as a label to optimize the image denoising model through supervised learning.

3. The joint denoising training method combining noise-free images and noisy images according to claim 1, characterized in that: In step S1, the image denoising model is an image denoising model named SUNet; SUNet is an image denoising model that combines the Swin Transformer with the UNet architecture. The model consists of three modules: shallow feature extraction module, UNet feature extraction module, and image reconstruction module. (1) Shallow feature extraction module: This module adjusts the feature dimension of the input image through a 3×3 convolutional layer and extracts the low-level features of the image; this process can be expressed by the following formula: F shallow =M SFE (I) in, represents the image to be denoised, H and W represent the height and width of the image respectively, and 3 represents the number of channels of the image (i.e., RGB channels); M SFE (·) represents the shallow feature extraction layer, which is the process of extracting features from the image through convolution operation; represents shallow features, and C represents the number of channels of shallow features; (2) UNet feature extraction module: This module adopts an encoder-decoder structure as the main body; the encoder gradually reduces the spatial resolution of the image through multiple Swin Transformer blocks and downsampling layers and extracts deep features; the decoder gradually restores the spatial resolution of the image through multiple Swin Transformer blocks and upsampling layers, while combining jump connections to retain shallow feature information; (3) Image reconstruction module This module uses a 3×3 convolutional layer M R (·), generate the denoised image based on the deep features; the final image reconstruction process is expressed by the following formula: in, Represents the final output of the model, H and W represent the height and width of the image respectively, and 3 represents the number of channels of the image; M R (·) represents the image reconstruction layer, which is the convolution operation from the deep feature F deep Reconstruct the denoised image.

4. The joint denoising training method combining noise-free images and noisy images according to claim 1, characterized in that: In step S2, the data enhancement operation includes image rotation angle, contrast adjustment, brightness change and random cropping, and the enhanced image is used to further improve the generalization ability of the model.

5. The joint denoising training method combining noise-free images and noisy images according to claim 1, characterized in that: In step S3, the hybrid loss function is composed of the first loss function and the second loss function: First loss function: Input the noisy image into the denoised image generated by the image denoising model, and then calculate the first loss value based on the pixel value of the real sample image: in, is the denoised image output by the model, I $％ea( is a real clean image; ‖·‖ refers to the L1 norm operation, that is, calculating the absolute error of the image at each pixel position, and adding all these errors to obtain the image loss value; Second loss function: Input the clean image into the denoised image generated by the model, and then calculate the second loss based on the pixel value of the real sample image: in, is the clean image after denoising, I $％ea( For a real clean image; The expression of the mixed loss function is: Among them, λ1 and λ ρ is a hyperparameter that controls the weight of the two parts of the loss function.

6. The joint denoising training method combining noise-free images and noisy images according to claim 1, characterized in that: In step S4, the dynamic optimization strategy refers to adjusting the learning rate according to the denoising effect of the model during training; First, the warm-up strategy is adopted; in the warm-up phase, the learning rate is increased from η . Gradually increase until the preset initial learning rate η is reached 0a12345 , and keep this value until the end of the warm-up phase. The learning rate adjustment formula is: Among them, η6 is the learning rate of the current training round t step, η wa9:3；p represents the target learning rate at the end of the warm-up phase, η . is the learning rate at the beginning of the warm-up phase; T 0a12345 is the total number of training rounds in the warm-up phase; After the warm-up phase, the learning rate enters the cosine annealing phase, and the learning rate gradually decreases in the manner of a cosine function as the training progresses; the learning rate adjustment formula for the cosine annealing strategy is: Among them, η6 is the learning rate of the current training round t step, η 2i( is the minimum learning rate during training, η wa9:3；p is the target learning rate at the end of the warm-up phase; T HIHa％ is the total number of training rounds, T 0a12345 is the total number of training rounds in the warm-up phase.

7. The joint denoising training method combining noise-free images and noisy images according to claim 1, characterized in that: In step S5, the adaptive optimizer is an Adam optimizer, which uses a dynamic adjustment of the learning rate and a weight decay strategy to further improve the training efficiency and denoising effect of the model. The update formula of the Adam optimizer is as follows: Among them, θ6 is the model parameter of the current training round t step, θ6 is the parameter that needs to be optimized in the network (such as the convolution kernel weight in the shallow feature extraction module and the image reconstruction module, the weight matrix and bias term of the SwinTransformer self-attention layer in the UNet feature extraction module, etc.), η6 is the current learning rate, and are the first-order and second-order moment estimates of the gradient, and ε is a very small constant to prevent division by zero errors. First moment estimate and the second moment estimate The update rules are as follows: m6=β1m 631 +(1-β1)g6 v6=β ρ v 631 +(1-β ρ )g6 ρ Among them, g6 is the gradient of the loss function relative to the model parameters when the training round is t; β1 and β ρ Decay rate hyperparameters for the first and second moment estimates.

Images