A lightweight image denoising method based on information fusion

Abstract

The present invention discloses a lightweight image denoising method based on information fusion, which obtains better denoising results with low computational complexity. The basic module based on feature enhancement aggregates channel, spatial and intrinsic information, only transmits important features, and obtains better results with fewer layers. The information compensation module is a lightweight information comparison, correction and compensation module, which comprehensively considers the original image and intermediate features, and effectively supplements the missing information in the basic module. The up and down sampling modules realize the transformation of spatial resolution and number of channels with minimal computational cost, while reducing information loss. The present invention can process Gaussian noise images with known and unknown noise levels, and can obtain high-quality denoising results in mobile terminals and embedded devices with limited computing resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a lightweight image denoising method based on information fusion. Background Art

[0002] When capturing images using digital devices, due to limitations in the imaging environment and equipment, the images often contain a significant amount of noise, resulting in reduced image quality. Image denoising is a low-level vision task whose goal is to restore a high-quality, clean image from a noisy image. Noise removal not only improves image visual quality but also serves as a preliminary processing step for other computer vision tasks.

[0003] Image denoising methods can be divided into two categories: traditional methods and deep learning methods. Traditional methods are generally based on prior information and have good interpretability, but they have problems such as being unable to handle complex noise situations. Deep learning-based methods have better denoising effects. By adopting deeper and wider networks and complex branching structures, existing methods have a larger receptive field and can extract richer representations, thereby improving the denoising effect. However, they also have problems such as high computational complexity and difficulty in optimizing during network training. Therefore, there is an urgent need for a lightweight image denoising method that combines the advantages of low computational complexity and good denoising effects for application in mobile or embedded devices. Summary of the Invention

[0004] In order to overcome the problem of large computational complexity of network models in existing methods, the present invention proposes a lightweight image denoising method based on information fusion, which achieves the same or even better image denoising effect with a lightweight network.

[0005] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0006] The present invention first provides a lightweight image denoising method based on information fusion, which includes the following steps:

[0007] Step S1: Add additive Gaussian white noise to the noise-free color image to construct pairs of noise-free and noisy images as training sets;

[0008] Step S2: Construct a lightweight convolutional neural network as a denoising network. The denoising network consists of a multi-level symmetrical encoder and decoder. The feature map resolution in each encoder and decoder is different. Processing features at different resolutions can achieve better results. Each encoder and decoder contains multiple basic modules based on feature enhancement and an information compensation module. The basic module based on feature enhancement integrates the channel, spatial, and intrinsic information of the feature map and only transmits key information in the information flow. The information compensation module comprehensively considers the original information of the image and the output results of the intermediate features, adaptively compensates for the missing information in the basic module, and realizes the progressive supplementation of the original information and the smooth fusion of the features. The upsampling and downsampling modules realize the conversion of image resolution and number of channels with the lowest computational complexity, and use the relationship between pixels to reduce information loss during the upsampling and downsampling process.

[0009] Step S3: Using Charbonnier loss as the loss function, train the neural network to obtain the parameters of the denoising model;

[0010] Step S4: input the noisy image to be processed into the trained neural network to obtain a denoised image.

[0011] As a preferred solution of the present invention, the denoising network of step S2 is U-shaped as a whole; given a noisy image, the denoising network uses a layer of convolutional neural network to obtain low-level features, and the low-level features are converted into deep features through multi-level symmetrical encoding and decoding; the spatial resolution of each level of encoder is the same as that of the previous level. The number of channels is twice that of the previous level; skip connections are used to splice the encoder features and decoder features of the same level in the channel dimension, and a layer of convolutional neural network is used to halve the number of channels of the spliced ​​features; the spatial resolution of each level of decoder is 4 times that of the previous level, and the number of channels is The deep features of the first-level decoder are passed through a layer of convolutional neural network to obtain the denoised image.

[0012] As a preferred solution of the present invention, the basic module based on feature enhancement is specifically:

[0013] The input features are first normalized and passed into point-by-point convolution to increase the number of channels. Deep convolution is used in high dimensions. Then, important information in the features is aggregated through weighted channel attention and spatial attention, and the number of channels is compressed using point-by-point convolution. After another normalization operation, the number of channels is expanded using point-by-point convolution. After deep convolution, the features are divided into two parts in the channel dimension. One part of the features is used as intrinsic information and is not processed, and the other part of the features is used as redundant features. After nonlinear transformation, it acts on the intrinsic features, and finally point-by-point convolution is used to compress the number of channels.

[0014] As a preferred solution of the present invention, the information compensation module is specifically:

[0015] The input feature map is transformed to obtain a three-channel restored image. The restored image is compared with the original image to obtain an error map. After the error map is transformed into a feature map, it passes through several layers of convolutional neural networks to compensate for the missing information in the input feature map and obtain the output feature.

[0016] As a preferred solution of the present invention, the upsampling module is specifically:

[0017] For the upsampling process, a low-resolution feature map is input and the resolution is increased through a pixel shuffle operation; then a nonlinear transformation is performed, and the transformed feature map is concatenated with the feature map before and after the transformation to achieve channel expansion.

[0018] As a preferred solution of the present invention, the downsampling module is specifically:

[0019] During the downsampling process, the attention map of aggregated channel information is first generated through the channel attention operation, multiplied with the original feature map, and the important channels in the feature map are determined. Then, the convolutional neural network is used to reduce the number of channels, and the pixel unshuffle operation is used to reduce the resolution.

[0020] As a preferred embodiment of the present invention, step S3 is specifically as follows:

[0021] Use Charbonnier loss as the loss function: Among them, Y represents a high-quality labeled image; Y' is the output of the neural network, that is, the denoising result;

[0022] AdamW is used as the optimizer, and the cosine learning rate decay mechanism is used. That is, the entire training process requires N iterations. The learning rate is set to a certain value in the first a iterations, and then the learning rate decays from the fixed value to another value.

[0023] A progressive training strategy is adopted to achieve a trade-off between training speed and test accuracy. Small image blocks and large batch sizes are used in the early stages of training to allow the model to find convergence directions more quickly. In the later stages, the resolution of the image blocks is gradually increased and the batch size is reduced to allow the neural network to capture more image information.

[0024] The beneficial effects of the present invention are:

[0025] 1) The feature enhancement-based foundation module employed in this invention extracts only key information and restores important features, reducing the number of network layers and computational complexity. The information compensation module, a lightweight error estimation and compensation module, achieves progressive replenishment of original information and smoother feature fusion. The up- and down-sampling modules effectively reduce computational complexity and information loss by adjusting the order of feature transformations and leveraging inter-pixel relationships. This organic integration of these three modules solves the existing problem of achieving superior denoising results at the cost of extensive computation, thereby achieving high-quality image denoising with low computational complexity.

[0026] 2) The lightweight image denoising method based on information fusion proposed in this paper can achieve image denoising under Gaussian noise with known noise levels and unknown noise levels. Compared with other image denoising methods based on deep learning, it can obtain better image denoising results with lower computational complexity and has broader application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flowchart of the lightweight image denoising method based on information fusion.

[0028] Figure 2 This is the overall structure diagram of the neural network model.

[0029] Figure 3 This is the structural diagram of the basic module based on feature enhancement.

[0030] Figure 4 This is the structural diagram of the information compensation module.

[0031] Figure 5 This is the structural diagram of the sampling module. The left picture is the upsampling module, and the right picture is the downsampling module.

[0032] Figure 6 The image denoising effect of the present invention under different noise levels is shown in the upper figure. The upper figure is the image to be processed with Gaussian noise of different noise levels added, and the lower figure is the denoised image after being processed by the method of the present invention.

[0033] Figure 7 The performance comparison of different methods. DETAILED DESCRIPTION

[0034] The present invention will be further described and illustrated below in conjunction with specific embodiments. The embodiments are merely illustrative of the present disclosure and do not limit its scope. The technical features of the various embodiments of the present invention may be combined accordingly, provided that there is no conflict between them.

[0035] like Figure 1 The flowchart of a lightweight image denoising method based on information fusion shown in FIG. 1 includes the following steps:

[0036] Step S1: Add additive Gaussian white noise to the noise-free color image to construct pairs of noise-free and noisy images as training and test sets; at the same time, perform data enhancement on the training set by image flipping, rotation, random cropping, etc.

[0037] Step S2: Construct a lightweight convolutional neural network as a denoising network to extract information from the input noisy image and reconstruct a high-quality image. The network adopts a U-shaped structure with a three-level codec. A feature enhancement-based basic module extracts important features, an information compensation module supplements missing information, and a lightweight up- and down-sampling module achieves resolution and channel conversion.

[0038] Step S3: Use Charbonnier loss as the loss function to train the neural network and obtain the parameters of the denoising model.

[0039] Step S4: input the noisy image to be processed into the trained neural network model to obtain a denoised image.

[0040] Step S1 is described in detail as follows: We used 900 DIV2K images, 2650 Flickr2K images, 400 BSD500 images, and 4744 WED image samples. We added additive Gaussian noise of level σ to the original images to generate noisy images, which served as the training set. We also used the CBSD68, Kodak24, McMaster, and Urban100 datasets as the test sets. We added Gaussian noise of level σ to the original images to construct pairs of noisy and denoised images. For the known noise case, we set the noise level σ to 15, 25, and 50, and trained a denoising model for each noise level. For the unknown noise case, we set σ between 0 and 70, trained a single denoising model, and tested its denoising performance for noise levels σ = 10, 30, 50, and 70. We also performed random data augmentation on the training set, such as rotation, flipping, and cropping.

[0041] The detailed description of step S2 is as follows: The network as a whole adopts a U-shaped structure, and performs denoising at different image resolutions. The overall network structure is as follows: Figure 2 As shown. Given a noisy image X∈R H×W×3 , use a layer of 3×3 convolution to obtain shallow features ENC1∈R H×W×C , where H×W represents the spatial dimension and C represents the number of channels. Then, the shallow feature ENC1 is converted into the deep feature DEC1∈R through a three-level symmetrical encoder and decoder. H×W×C , and then after a layer of 3×3 convolution, the denoised image Y'∈R H×W×3Each level of encoder contains N basic modules FEM based on feature enhancement and 1 information compensation module ICM. The number of basic modules N in the encoders from the first to the third level is 3, 4, and 4 respectively. In the fourth level, 6 basic modules FEM and 1 information compensation module ICM are used to aggregate the information in the low-resolution features. The decoder converts the low-resolution features output by the fourth level network into As input, high-resolution features are gradually restored. The third and second level decoders contain 4 basic modules FEM based on feature enhancement and 1 information compensation module ICM. At the end of the model, the denoising network no longer needs to refer to the original image information, so the first level decoder only contains 3 basic modules FEM. The spatial resolution of each level encoder is the same as that of the previous level. The number of channels is twice that of the previous level. The spatial resolution of each level of decoder is 4 times that of the previous level, and the number of channels is To help the recovery process, skip connections are used to concatenate the encoder features and decoder features of the same level in the channel dimension, and the number of channels of the concatenated features is halved through 1×1 convolution.

[0042] The network of the present invention adopts a U-shaped structure as a whole, performs denoising at different image resolutions, and obtains reliable contextual information. The basic module FEM based on feature enhancement integrates the channel, space, and intrinsic information of the feature map, only extracts key information and restores important features, reducing the number of network layers and the amount of calculation. The information compensation module ICM is a lightweight error estimation and compensation module that comprehensively considers the information of the original image and intermediate features, and realizes the progressive supplement of the original information and smoother feature fusion. The up and down sampling modules reduce the amount of calculation by adjusting the order of feature transformation, and effectively reduce the information loss in the up and down sampling process by making full use of the relationship between pixels.

[0043] The above-mentioned feature enhancement-based basic module FEM only focuses on important features, avoids invalid operations on redundant features, and achieves better denoising effects with fewer network layers. This module adopts an inverse bottleneck structure, first using point-by-point convolution to increase the dimension of the feature map, then extracting features based on high-dimensional features based on depth convolution, and then using point-by-point convolution to reduce the dimension and aggregate information between channels. The computational complexity of depth-separable convolution is about 100% of that of ordinary convolution. K represents the size of the convolution kernel. To achieve effective compensation with low computational effort, spatial attention and channel attention are first used to aggregate information from space and channels, respectively. Adaptive parameters are used to adjust the emphasis on spatial and channel information. Features are then divided into intrinsic features and redundant features. Intrinsic features are not manipulated, while redundant features undergo a nonlinear transformation and are multiplied with the intrinsic features. Enhancement and calibration are performed based on the feature map's inherent information. The feature-enhanced depthwise separable convolution module (FEM) suppresses useless features and allows only features with richer information to be passed on. This allows for better denoising with fewer layers.

[0044] Specifically, the feature-enhanced basic module FEM of this embodiment is as follows: Figure 3 As shown in Figure 1. The input features first undergo layer normalization before being passed to pointwise convolution to increase the number of channels. Where c represents the number of channels, n represents the number of convolution kernels, k represents the kernel size, and g represents the number of groups. PWConv uses a 1x1 convolution kernel to expand the number of channels in the feature map from 8 to 24. In higher dimensions, depthwise convolution with a 3x3 kernel and 24 groups is used to extract features. Weighted channel attention (CA) and spatial attention (SA) are then used to aggregate important information within the features. A 1x1 convolution kernel is then used to compress the 24-channel feature map to an 8-channel feature map. Layer normalization is then applied again. Pointwise convolution expands the number of channels to 32. The depthwise convolution features are split into two parts in the channel dimension: 16 channels are treated as intrinsic information and remain unprocessed; the remaining 16 channels are redundant features that undergo a nonlinear transformation and are multiplied with the intrinsic features. Finally, pointwise convolution is used to reduce the number of channels. Depthwise separable convolution reduces the computational complexity of the network, while attention operations and intrinsic feature enhancement operations make up for the loss of accuracy, and better denoising effects can be achieved using fewer network layers.

[0045] The aforementioned information compensation module (ICM) first transforms the feature map to obtain a three-channel image, which is then compared with the original image to generate a relatively complete error map. This comparison of the original image and the intermediate result enables information collection. The error map is then corrected using several layers of depthwise separable convolutional layers with activation functions to achieve more accurate error predictions. These layers adaptively learn the information the network needs to generate or delete. Furthermore, in image restoration tasks, the nonlinear layers help solve underdetermined equations. Finally, the corrected error information is applied to the input features to supplement the information, achieving efficient information fusion. In this process, subtraction is used to compare image information and compensate for feature errors, making the network interpretable. The lightweight information compensation module (ICM) directly extracts information from the original image and uses this information to supervise the image restoration process. The existing results are iteratively improved and enhanced, gradually approaching the final high-quality image.

[0046] Specifically, the information compensation module ICM of this embodiment is as follows: Figure 4 As shown. The input feature map first undergoes pixel shuffle to transform the resolution and 1x1 convolution to transform the number of channels to obtain a 3-channel recovery map. The image is subtracted from the original image to obtain an error map, and the error map is used to supervise the reconstruction process. The error map is transformed into a feature map through 1x1 convolution and pixel unshuffle. After several layers of depth-separable convolution with ReLU activation, it is subtracted from the input feature map to obtain the output feature. The error map is optimized using convolution, so that the network can adaptively decide which information needs to be supplemented or deleted; ReLU, as a nonlinear function, is very helpful in solving underdetermined equations. The computational complexity of the information compensation module ICM is only that of the basic module FEM. It achieves more efficient information collection, transmission and fusion with less computational effort. Through multiple rounds of information compensation, the current denoising results are supplemented, improved and strengthened, and the output gradually approaches the final denoising result.

[0047] The above-mentioned up-sampling and down-sampling modules simplify the computation by adjusting the order of convolution and element transformation, and the computation is only 1 / 2 of the traditional method. In addition, the module also refers to the information of surrounding elements and channels to reduce the information loss during the sampling process. This is crucial in the U-shaped structure that requires frequent upsampling and downsampling. Specifically, the structure of the upsampling and downsampling module is as follows Figure 5 As shown. The upsampling module is as follows Figure 5 As shown on the left, input The low-resolution features are transformed into The feature map is transformed nonlinearly, and the transformed feature map is concatenated with the feature map before and after transformation to achieve channel expansion. The downsampling process is as follows Figure 5 As shown on the right, the H×W×C feature map is first processed through the channel attention operation CA to generate a 1×1×C feature map that aggregates channel information, which is multiplied by the original feature map to obtain the importance of the feature map; then, the number of channels is reduced to the original through 1×1 convolution. Finally, pixelunshuffle operation is performed to arrange the pixels of four adjacent positions in space into four adjacent elements in the channel, and we get The up-sampling and down-sampling modules reduce the amount of computation by adjusting the order of convolution and element transformation, and reduce information loss through generation and attention operations.

[0048] The detailed description of step S3 is as follows: Charbonnier loss is used as the loss function: Where Y represents a high-quality label image; Y' is the output of the neural network, i.e., the denoising result; ε = 10 -3 AdamW is used as the optimizer, and the cosine learning rate decay mechanism is used. The entire training process requires 600k iterations, and the learning rate is set to 3e for the first 184k iterations. -4 , after which the learning rate is 3e -4 Decay to 1e -6 A progressive training strategy is used to achieve a trade-off between training speed and test accuracy. In the early stages of training, small image blocks and large batch sizes are used to help the model converge more quickly. Later in the training phase, the resolution of the image blocks is gradually increased and the batch size is reduced, allowing the neural network to capture more image information.

[0049] The detailed description of step S4 is as follows: the noisy image to be processed is input into the trained neural network (denoising model) to obtain a denoised image.

[0050] The embodiments of the present invention use both qualitative and quantitative methods to evaluate the technical effects of the present invention. Qualitative evaluation primarily involves visually observing the quality of the reconstructed image to determine whether the generated image produces unrealistic information or is excessively smooth, resulting in loss of detail and texture information. To quantitatively evaluate its performance, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used for evaluation. The peak signal-to-noise ratio (PSNR) quantifies the quality of image reconstruction; a larger value indicates better results. Its calculation formula is: Where MAX represents the range of pixel values, which is 255 in the case of 8-bit images; W and H are the width and height of the image, respectively; Y(i,j) and Y'(i,j) are the label image and the output image of the neural network, respectively. Since PSNR ignores factors such as spatial frequency and image brightness when evaluating image quality, the evaluation results of PSNR are inconsistent with the effects of human eye perception. In order to evaluate image quality more reasonably and accurately, structural similarity SSIM is used as another evaluation criterion for image quality. It directly estimates the changes in signal structure between the original image and the processed image, and uses mean, standard deviation, and covariance to measure the brightness, contrast, and structural similarity of the image. SSIM reflects the similarity between two images. The larger the value, the better the performance. Its formula is: Among them, μ*, σ*, σ Y,Y' Respectively represent the mean value of matrix *, the standard deviation of matrix * and the covariance between matrices Y and Y'; c1 and c2 are two constants, set to (0.01*MAX) 2 and (0.03*MAX) 2 .

[0051] The qualitative evaluation results of the present invention are as follows: For unknown noise levels, the embodiment trains a denoising model. Under the input of 256*256*3, the computational load is 11.3G, and the denoising effect is as follows: Figure 6 As shown in the figure, the images to be processed with noise levels σ = 10, 30, 50, and 70 are added as shown in the figure. Figure 6 As shown above, the denoising results after processing in the embodiment are as follows Figure 6 As shown below, it can be seen from the figure that the embodiment can separate image content and noise when the Gaussian noise level is unknown, indicating that the method proposed in the present invention can obtain a good and clear denoised image.

[0052] The quantitative evaluation results of the present invention are as follows: In the embodiment, a model was trained with a known noise level σ = 15. Under the input of 256*256*3, the computational cost of the model was 11.3G, and its PSNR on the CBSD68 dataset was 34.24 and SSIM = 0.9337. The computational cost and PSNR were compared with other methods. Figure 7 As shown, the present invention achieves a higher PSNR value with less computational effort, that is, a better image denoising effect is achieved.

[0053] The above further describes the specific implementation methods of the present invention in detail with reference to specific examples. The above contents are all explanations of the present invention. All technical solutions under the concept of the present invention belong to the protection scope of the present invention.

[0054] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. Persons skilled in the art will readily appreciate that variations and modifications may be made without departing from the scope of the present invention, all of which fall within the scope of protection of the present invention.

Claims

1. A lightweight image denoising method based on information fusion, characterized in that: The steps include: Step S1: Add additive Gaussian white noise to the noise-free color image to construct pairs of noise-free and noisy images as training sets; Step S2: Construct a lightweight convolutional neural network as a denoising network. The denoising network consists of a multi-stage symmetrical encoder and decoder. The feature map resolution in each stage of the encoder and decoder is different. Processing features at different resolutions can achieve better results. Each stage of the encoder and decoder contains multiple basic modules based on feature enhancement and an information compensation module. The basic module based on feature enhancement integrates the channel, spatial, and intrinsic information of the feature map, and only transmits key information in the information flow; The information compensation module comprehensively considers the original information of the image and the output results of the intermediate features, adaptively compensating for the missing information in the basic module, achieving progressive replenishment of the original information and smooth fusion of features; the upsampling and downsampling modules achieve image resolution and channel number conversion with minimal computational effort, and utilize the relationship between pixels to reduce information loss during the upsampling and downsampling process; The basic modules based on feature enhancement are specifically: The input features are first normalized and passed into point-by-point convolution to increase the number of channels. Deep convolution is used in high dimensions. Then, important information in the features is aggregated through weighted channel attention and spatial attention. Finally, point-by-point convolution is used to compress the number of channels. After another normalization operation, point-by-point convolution is used to expand the number of channels. After deep convolution, the features are divided into two parts in the channel dimension. One part of the features is treated as intrinsic information and is not processed, while the other part is treated as redundant features. After nonlinear transformation, it acts on the intrinsic features. Finally, point-by-point convolution is used to compress the number of channels. The information compensation module is specifically: The input feature map is transformed to obtain a three-channel restored map. The restored map is compared with the original image to obtain an error map. After the error map is transformed into a feature map, it passes through several layers of convolutional neural networks to compensate for the missing information of the input feature map and obtain the output feature. The upsampling module is specifically: For the upsampling process, a low-resolution feature map is input and the resolution is increased through a pixel shuffle operation. Then a nonlinear transformation is performed, and the transformed feature map is concatenated with the feature map before and after the transformation to achieve channel expansion. The downsampling module is specifically: During the downsampling process, the attention map of aggregated channel information is first generated through the channel attention operation, multiplied with the original feature map, and the important channels in the feature map are determined. Then, the convolutional neural network is used to reduce the number of channels, and the pixel unshuffle operation is used to reduce the resolution. Step S3: Using Charbonnier loss as the loss function, train the neural network to obtain the parameters of the denoising model; Step S4: input the noisy image to be processed into the trained neural network to obtain a denoised image.

2. The lightweight image denoising method based on information fusion according to claim 1, characterized in that: The denoising network in step S2 is U-shaped as a whole; Given a noisy image, the denoising network uses a layer of convolutional neural network to obtain low-level features. The low-level features are converted into deep features through multi-level symmetrical encoders and decoders. The spatial resolution of each encoder level is the same as that of the previous level. The number of channels is twice that of the previous level; skip connections are used to splice the encoder features and decoder features of the same level in the channel dimension, and a layer of convolutional neural network is used to halve the number of channels of the spliced ​​features; the spatial resolution of each level of decoder is 4 times that of the previous level, and the number of channels is The deep features of the first-level decoder are passed through a layer of convolutional neural network to obtain the denoised image.

3. The lightweight image denoising method based on information fusion according to claim 1, characterized in that: The step S3 is specifically as follows: Use Charbonnier loss as the loss function: Among them, Y represents a high-quality labeled image; Y' is the output of the neural network, that is, the denoising result; AdamW is used as the optimizer, and the cosine learning rate decay mechanism is used. That is, the entire training process requires N iterations. The learning rate is set to a certain value in the first a iterations, and then the learning rate decays from the fixed value to another value. A progressive training strategy is adopted to achieve a trade-off between training speed and test accuracy. Small image blocks and large batch sizes are used in the early stages of training to allow the model to find convergence directions more quickly. In the later stages, the resolution of the image blocks is gradually increased and the batch size is reduced to allow the neural network to capture more image information.

Images