Dark light image noise reduction method based on differential convolution

Abstract

The invention provides a difference convolution-based dark light image noise reduction method, which comprises the following steps of: S1, constructing a dark light image data set for training a noise reduction model, and carrying out image preprocessing and data enhancement operation; s2, loading to a U-net model based on differential convolution; s3, training the network, and training the model by adopting an MAE loss function and an Adam optimizer; s4, performing re-parameterization on the trained model; and S5, loading the complete image into the re-parameterized model, and outputting a de-noised image, namely loading the pre-processed noise image which is not cut into the re-parameterized network for de-noising, and obtaining the de-noised image. By improving a network structure and using a reasonable data simulation synthesis method, the problem that details are seriously lost after image denoising in a dark light environment is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image noise reduction, and in particular relates to a dark light image noise reduction method based on differential convolution. Background Art

[0002] In the existing technology, noise will have a negative impact on the visual quality of the image. Noise reduction can reduce or eliminate the noise in the image, making it clearer, easier to observe and understand, which is very important for many application fields such as photography, medical imaging, and surveillance video. In many image analysis and recognition tasks, noise may interfere with the extraction and accuracy of key information. Through noise reduction processing, the signal quality of the image can be improved and the interference of noise can be reduced, thereby improving the performance of tasks such as image analysis, target detection, and image recognition.

[0003] In low-light environments, the number of photons received by the imaging sensor is small, the signal in the image is relatively weak, and the noise is more likely to be highlighted, making it more difficult to distinguish between signal and noise. In order to reduce the noise in the image, it is usually necessary to apply image denoising algorithms. However, since these algorithms cannot accurately model the real noise, they often lead to loss of details. In low-light environments, subtle details are crucial to restoring image quality, which not only affects the appearance of the final image, but may also further reduce the accuracy of downstream tasks such as face recognition and object detection.

[0004] The development of image denoising algorithms has always been one of the research hotspots in the field of digital image processing. Classic denoising algorithms include mean filtering, median filtering, Gaussian filtering, etc. These algorithms are simple and easy to implement, but have limited effects in complex noise and retaining image details. In recent years, deep learning has made significant progress in the field of image denoising. By constructing a deep neural network model, the mapping relationship between noisy images and noise-free images can be learned to achieve high-quality image denoising effects. Some commonly used deep learning models include autoencoders, convolutional neural networks, and generative adversarial networks.

[0005] The development trend of image denoising algorithms is to comprehensively utilize multiple methods, combining classical methods with emerging technologies such as deep learning, to improve denoising effects and preserve image details. In addition, customized algorithms for specific application scenarios and noise types have also received increasing attention and research. In dim light environments, the sources of noise are more complex. Some studies have achieved better dim light image denoising effects by physically modeling different noise sources and combining the powerful modeling capabilities of convolutional neural networks.

[0006] For image noise reduction in dark light environments, existing technologies mainly solve the problem in two aspects:

[0007] (1) Design a powerful neural network model to more accurately model the noise distribution. However, a network model with strong fitting capabilities usually has many parameters and large computational complexity, making it difficult to deploy on end-side devices with limited resources.

[0008] (2) Accurately model different noise sources physically and enhance the accuracy of network fitting noise distribution by synthesizing simulated data that is less different from real data. Since the noise sources in dark environments are complex, it is difficult to statistically model each type of noise, so there is still a problem of detail loss after image denoising.

[0009] In addition, the commonly used terms in the prior art include:

[0010] Black frame: An image captured by a camera in a completely dark environment, containing only noise that is unrelated to the signal.

[0011] RAW domain image: A picture with original information taken and recorded by the camera, without compression or any processing.

[0012] Bayer format: The raw data format obtained using the Bayer array sensor, its general format is: odd-numbered scanning line output RGRG...even-numbered scanning line output GBGB...

[0013] U-net: U-net was published in 2015 and is a variant of FCN. Its original intention was to solve problems in biomedical images. Due to its significant effect, it has been widely used in various directions of semantic segmentation.

[0014] Black level correction: Due to the influence of dark current, the actual raw data output by the sensor in a dark environment is not all 0. The process of subtracting the reference dark current signal from the obtained image signal is called black level correction.

[0015] Data enhancement: Data enhancement is an effective method to expand the size of data samples. Image data enhancement methods usually include spatial geometric transformation, pixel color transformation, and multi-sample synthesis. Summary of the invention

[0016] In order to solve the above problems, the purpose of this application is to solve the problem of serious detail loss after image denoising in dark environments by improving the network structure and using a reasonable data simulation synthesis method.

[0017] Specifically, the present invention provides a dark light image denoising method based on differential convolution, the method comprising the following steps:

[0018] S1. Construct a dark light image dataset for training the denoising model, and perform image preprocessing and data enhancement operations;

[0019] S2. Load it into the U-net model based on differential convolution, that is, based on the U-net model, improve the poor degree of image detail restoration after denoising in a dark environment. The input of the network is a noisy image, and the output is the corresponding noise map;

[0020] S3. Train the network and use the MAE loss function and Adam optimizer to train the model;

[0021] The noisy image X after preprocessing and data enhancement input Input into the improved U-net model and get the predicted noise map after forward propagation The MAE function is used to calculate the loss of the predicted image and the reference image; the Adam optimizer is used to update the network parameters, the cosine annealing learning rate reduction strategy is used during the training process, and the half cycle of the cosine function is set as the maximum number of iterations;

[0022] S4. Perform reparameterization on the trained model, that is, network structure reparameterization, and convert the differential convolution in the denoising network into a standard convolution through structural reparameterization to reduce the amount of model calculation and reduce the inference time;

[0023] S5. Load the complete image into the re-parameterized model and output the denoised image, that is, load the pre-processed, uncropped noisy image into the re-parameterized network for denoising to obtain the denoised image.

[0024] The step S1 further comprises:

[0025] S1.1, build a dark light image dataset: collect reference images with high signal-to-noise ratio and black frames containing only various noise sources under different analog gains, directly sample real noise from the black frames, and then synthesize them with the reference images to obtain images with real noise, and obtain the "real noise image-reference image" paired dataset;

[0026] S1.2, data preprocessing: perform black level correction and normalization on the Bayer format RAW image, convert the single-channel output image into a 4-channel one, and reduce the image spatial resolution to 1 / 4 of the original one; as shown in the following formula:

[0027]

[0028] In formula (1), bit is the number of bits of the RAW image. After black level correction and normalization, the single-channel output image is converted to 4-channel, and the image spatial resolution is reduced to 1 / 4 of the original. On the one hand, reducing the resolution can reduce the computational overhead of the network, and on the other hand, it can avoid destroying the Bayer format when performing data enhancement operations.

[0029] S1.3, data enhancement: Randomly crop the image after data preprocessing, randomly rotate and randomly flip the cropped image block, and use the enhanced image block as the input of the network.

[0030] The step S1.1 further comprises:

[0031] The real signal-independent noise is sampled directly from the black frame, and the signal-dependent noise that obeys the Poisson distribution is calculated based on the reference image. Then, the signal-dependent noise and the signal-independent noise are synthesized with the reference image to obtain an image with real noise, thereby obtaining a "real noise image-reference image" paired dataset.

[0032] The step 2 further comprises:

[0033] S2.1, build the encoder: The encoder consists of three alternating convolution modules and downsampling modules, which are used to extract local edge and texture features and high-level semantic features. Through feature map visualization technology, it is found that the detailed features of edges and textures are extracted by shallow convolution layers, so differential convolution is only used in the first two convolution modules in the encoding stage. Among them, the first convolution module is composed of N differential convolution stacks, the second convolution module is a differential residual module, and the third convolution module is a standard residual module. The downsampling module uses a standard convolution with a step size of 2.

[0034] The improved convolution form: In order to improve the problem of insufficient detail recovery in a dark environment, differential convolution is introduced in the improvement process. First, the pixels within the receptive field of the convolution kernel are subtracted from the central pixel, and then the output is obtained through ordinary convolution. The calculation process is shown in formula (2):

[0035]

[0036] Among them, p0 represents the position of the center element, p n represents any element in the neighborhood of p0.

[0037] The differential convolution can explicitly encode the gradient in the image, so that the network can focus on extracting the image gradient information during the training process; through the feature map visualization technology, it is found that the detailed features of the edge and texture are extracted by the shallow convolution layer, so the differential convolution is only used in the first two convolution modules in the encoding stage;

[0038] S2.2, build decoder: The decoding stage consists of two consecutive "upsampling + residual modules" and one "upsampling + channel attention module + residual module", which gradually restores high-resolution information and feature optimization through upsampling and convolution. In order to reduce the loss of spatial information caused by the downsampling process, after each upsampling layer, the same-level features in the encoder are introduced, and then merged with the upsampled features in the channel dimension; among them, the upsampling layer uses a transposed convolution with a step size of 2, and the channel attention module obtains global information through global pooling to improve the representation ability of the network. The process of applying channel attention to the input is shown in the following formula:

[0039]

[0040] In formula (3), FC represents the fully connected layer, GAP represents the global average pooling, and σ represents the Sigmoid activation function. Represents element-wise multiplication.

[0041] The step S3 of training the network further comprises:

[0042] The noisy image X after preprocessing and data enhancement input Input into the improved U-net and get the predicted noise map after forward propagation The reference image is Y, and the MAE function is used to calculate the loss of the predicted image and the reference image, as shown in the following formula:

[0043]

[0044] The step S4 further comprises:

[0045] The differential convolution with a convolution size of 3×3 can be equivalently converted into the difference between a 3×3 standard convolution and a 1×1 standard convolution. Furthermore, the weights can be fused after padding the 1×1 convolution to a 3×3 convolution, and the 3×3 convolution and the 1×1 convolution can be reparameterized into a single standard convolution, thereby reducing the amount of calculation during model inference. Taking the convolution calculation of a 3×3 filter on a 3×3 feature map as an example, the weight of the filter is {w i ,i=1,2,...,9}, the activation value of the feature map is {x i ,i=1,2,...,9}, the output of the convolution is y, and the reparameterization process is as follows:

[0046]

[0047] Therefore, the advantages of this application are:

[0048] (1) The denoising network model using differential convolution can explicitly encode the gradients in the image, allowing the network to focus on extracting image gradient information during training and to restore detail information to the greatest extent possible while denoising dark-light images.

[0049] (2) Benefiting from the characteristics of differential convolution, the denoising network model does not require too many convolutional layers to extract fine texture features, and the overall computational complexity is small. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0051] Figure 1 This is a schematic diagram of the application process.

[0052] Figure 2 This is a schematic diagram of first subtracting the center pixel from the pixels within the receptive field of the convolution kernel in step S2 of the present application, and then obtaining the output through ordinary convolution.

[0053] Figure 3 It is a schematic diagram of the improved overall network structure in this application.

[0054] Figure 4 Schematic diagram of the structure of the encoder of the present application consisting of three alternating convolution modules and downsampling modules. DETAILED DESCRIPTION

[0055] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0056] like Figure 1 As shown in the figure, this application proposes a dark light image denoising method based on differential convolution. The overall process is as follows Figure 1 As shown, the following steps are included:

[0057] Step S1. Construct a dark light image dataset for training the denoising model, perform necessary preprocessing on it, and expand the dataset size and increase the diversity of the data through data enhancement, which specifically includes the following steps:

[0058] Step S1.1 constructs a dark light image dataset. In a dark light environment, the noise in the image has multiple sources, and the differences between different sensors are large. It is time-consuming and laborious to establish a noise model for each sensor. In order to accurately model all noise sources, by collecting a reference image with a high signal-to-noise ratio and a black frame containing only various noise sources under different analog gains, the real signal-independent noise is directly sampled from the black frame, and the signal-dependent noise that follows the Poisson distribution is calculated based on the reference image. Then, the signal-dependent noise and the signal-independent noise are synthesized with the reference image to obtain an image with real noise. According to the above steps, a "real noise image-reference image" paired dataset can be obtained.

[0059] Step S1.2 Image preprocessing: Perform black level correction and normalization on the Bayer format RAW image, as shown in the following formula:

[0060]

[0061] In formula (1), bit is the number of bits of the RAW image. After black level correction and normalization, the single-channel output image is converted to a 4-channel image, and the image spatial resolution is reduced to 1 / 4 of the original. On the one hand, reducing the resolution can reduce the computational overhead of the network, and on the other hand, it can avoid destroying the Bayer format when performing data augmentation operations.

[0062] Step S1.3 Data enhancement: Randomly crop the image after data preprocessing, randomly rotate and randomly flip the cropped image block, and use the enhanced image block as the input of the network.

[0063] Step S2. Based on the U-net model, the problem of poor image detail restoration after denoising in dark environments is improved. The improved overall network structure is as follows: Figure 3 As shown, the following steps are included:

[0064] Step S2.1 improves the convolution form. In a dark environment, the existing denoising model often leads to the problem of blurred details. Considering that the existing models all use standard convolution, they usually directly aggregate the intensity information of the local area and it is difficult to represent fine-grained texture features. Therefore, differential convolution is introduced in the improvement process, such as Figure 2 As shown, the pixels within the receptive field of the convolution kernel are first subtracted from the central pixel, and then the output is obtained through ordinary convolution. The calculation process is shown in formula (2):

[0065]

[0066] Among them, p0 represents the position of the center element, p n represents any element in the neighborhood of p0.

[0067] Differential convolution can explicitly encode the gradient in the image, allowing the network to focus on extracting image gradient information during training. Through feature map visualization technology, it is found that detail features such as edges and textures are extracted by shallow convolution layers, so differential convolution is only used in the first two convolution modules in the encoding stage, such as Figure 3 shown.

[0068] Step S2.2 builds the encoder. The encoder consists of three alternating convolution modules and downsampling modules, which are mainly used to extract local edge, texture features and high-level semantic features. Among them, the first convolution module is composed of N differential convolution stacks, the second convolution module is a differential residual module, and the third convolution module is a standard residual module. The downsampling module uses a standard convolution with a step size of 2. The specific structure is as follows: Figure 4 shown.

[0069] Step S2.2 builds the decoder. The decoding stage consists of two consecutive "upsampling + residual modules" and one "upsampling + channel attention module + residual module". High-resolution information and feature optimization are gradually restored through upsampling and convolution. In order to reduce the loss of spatial information caused by the downsampling process, after each upsampling layer, the same-level features in the encoder are introduced, and then merged with the upsampled features in the channel dimension. Among them, the upsampling layer uses a transposed convolution with a step size of 2, and the channel attention module obtains global information through global pooling to improve the representation ability of the network. The structure is as follows: Figure 4 As shown, the process of applying channel attention to the input is as follows:

[0070]

[0071] In formula (2), FC represents the fully connected layer, GAP represents the global average pooling, and σ represents the Sigmoid activation function. Represents element-wise multiplication.

[0072] Step S3: Train the network. The noisy image X after preprocessing and data enhancement is trained. input Input into the improved U-net and get the predicted noise map after forward propagation The reference image is Y, and the MAE function is used to calculate the loss of the predicted image and the reference image, as shown in the following formula:

[0073]

[0074] The Adam optimizer is used to update the network parameters. The cosine annealing learning rate reduction strategy is used during the training process, and the half cycle of the cosine function is set as the maximum number of iterations.

[0075] Step S4. Reparameterization. The differential convolution with a convolution size of 3×3 can be equivalently converted into the difference between a 3×3 standard convolution and a 1×1 standard convolution. Furthermore, after padding the 1×1 convolution to a 3×3 convolution, the weights can be fused, and the 3×3 convolution and the 1×1 convolution can be reparameterized into a single standard convolution, thereby reducing the amount of calculation during model inference. For example, the convolution calculation of a 3×3 filter on a 3×3 feature map is performed. The weight of the filter is {w i ,i=1,2,...,9}, the activation value of the feature map is {x i ,i=1,2,...,9}, the output of the convolution is y, and the reparameterization process is as follows:

[0076]

[0077] Step S5: Load the preprocessed, uncropped noisy image into the re-parameterized network for denoising to obtain a denoised image.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A dark light image denoising method based on differential convolution, characterized in that: The method comprises the following steps: S1. Construct a dark light image dataset for training the denoising model, and perform image preprocessing and data enhancement operations; S2. Load it into the U-net model based on differential convolution, that is, based on the U-net model, improve the poor degree of image detail restoration after denoising in a dark environment. The input of the network is a noisy image, and the output is the corresponding noise map; S3. Train the network and use the MAE loss function and Adam optimizer to train the model; The noisy image X after preprocessing and data enhancement input Input into the improved U-net model and get the predicted noise map after forward propagation The MAE function is used to calculate the loss of the predicted image and the reference image; the Adam optimizer is used to update the network parameters, the cosine annealing learning rate reduction strategy is used during the training process, and the half cycle of the cosine function is set as the maximum number of iterations; S4. Perform reparameterization on the trained model, that is, network structure reparameterization, and convert the differential convolution in the denoising network into a standard convolution through structural reparameterization to reduce the amount of model calculation and reduce the inference time; S5. Load the complete image into the re-parameterized model and output the denoised image, that is, load the pre-processed, uncropped noisy image into the re-parameterized network for denoising to obtain the denoised image.

2. The dark light image denoising method based on differential convolution according to claim 1, characterized in that: The step S1 further comprises: S1.1, build a dark light image dataset: collect reference images with high signal-to-noise ratio and black frames containing only various noise sources under different analog gains, directly sample real noise from the black frames, and then synthesize them with the reference images to obtain images with real noise, and obtain the "real noise image-reference image" paired dataset; S1.2, data preprocessing: perform black level correction and normalization on the Bayer format RAW image, convert the single-channel output image into a 4-channel one, and reduce the image spatial resolution to 1 / 4 of the original one; as shown in the following formula: In formula (1), bit is the number of bits of the RAW image. After black level correction and normalization, the single-channel output image is converted to 4 channels, and the image spatial resolution is reduced to 1 / 4 of the original; S1.3, data enhancement: Randomly crop the image after data preprocessing, randomly rotate and randomly flip the cropped image block, and use the enhanced image block as the input of the network.

3. The dark light image denoising method based on differential convolution according to claim 2, characterized in that: The step S1.1 further comprises: The real signal-independent noise is directly sampled from the black frame, and the signal-dependent noise that follows the Poisson distribution is calculated based on the reference image. Then, the signal-dependent noise and the signal-independent noise are synthesized with the reference image to obtain an image with real noise, thereby obtaining a "real noise image-reference image" paired dataset.

4. The dark light image denoising method based on differential convolution according to claim 3, characterized in that: The step 2 further comprises: S2.1, build the encoder: The encoder consists of three alternating convolution modules and downsampling modules, which are used to extract local edge and texture features and high-level semantic features. Through feature map visualization technology, it is found that the detailed features of edges and textures are extracted by shallow convolution layers, so differential convolution is only used in the first two convolution modules in the encoding stage. Among them, the first convolution module is composed of N differential convolution stacks, the second convolution module is a differential residual module, and the third convolution module is a standard residual module. The downsampling module uses a standard convolution with a step size of 2. The improved convolution form: In order to improve the problem of insufficient detail recovery in a dark environment, differential convolution is introduced in the improvement process. First, the pixels within the receptive field of the convolution kernel are subtracted from the central pixel, and then the output is obtained through ordinary convolution. The calculation process is shown in formula (2): Among them, P0 represents the center element position, p n represents any element in the neighborhood of p0; S2.2, build decoder: The decoding stage consists of two consecutive "upsampling + residual modules" and one "upsampling + channel attention module + residual module". High-resolution information and feature optimization are gradually restored through upsampling and convolution. In order to reduce the loss of spatial information caused by the downsampling process, the same-level features in the encoder are introduced after each upsampling layer, and then merged with the upsampled features in the channel dimension. Among them, the upsampling layer uses a transposed convolution with a step size of 2, and the channel attention module obtains global information through global pooling to improve the representation ability of the network. The process of applying channel attention to the input is shown in the following formula: In formula (3), FC represents the fully connected layer, GAP represents the global average pooling, and σ represents the Sigmoid activation function. Represents element-wise multiplication.

5. The dark light image denoising method based on differential convolution according to claim 4, characterized in that: The step S3 of training the network further comprises: The noisy image X after preprocessing and data enhancement input Input into the improved U-net and get the predicted noise map after forward propagation The reference image is Y, and the MAE function is used to calculate the loss of the predicted image and the reference image, as shown in the following formula:

6. The dark light image denoising method based on differential convolution according to claim 5, characterized in that: The step S4 further comprises: The differential convolution with a convolution size of 3×3 can be equivalently converted into the difference between a 3×3 standard convolution and a 1×1 standard convolution. Furthermore, the weights can be fused after padding the 1×1 convolution to a 3×3 convolution, and the 3×3 convolution and the 1×1 convolution can be re-parameterized into a single standard convolution, thereby reducing the amount of calculation during model inference; a 3×3 filter is used to perform convolution calculation on a 3×3 feature map, and the weight of the filter is {w i ,i=1,2,...,9}, the activation value of the feature map is {x i ,i=1,2,...,9}, the output of the convolution is y, and the reparameterization process is as follows:

Images