Image denoising and deblurring method fusing native features

Abstract

The application discloses a kind of image denoising and deblurring methods of fusion native characteristics, belong to computer vision field, this method first obtains the original data in data set by camera shooting, and carries out data collection, obtains the clear image pair after synthesis of blur.Secondly, the native image estimation network is constructed, and the estimated native image is obtained.Finally, the image denoising and deblurring network of fusion native characteristics is constructed, realizes denoising and deblurring, and outputs the clear image after different scales of denoising and deblurring.The application is superior to previous method in the joint denoising and deblurring, it is helpful to improve image quality, reduce camera cost, reduce resource waste, and has very important significance to intelligent transportation, face recognition and other fields.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to an image denoising and deblurring method fusing native features. BACKGROUND

[0002] With the development of artificial intelligence technology, target detection and target recognition technology has been widely used in life, such as the use of license plate detection in traffic violation photography, face recognition technology used in facial payment and the like. However, due to the rapid movement of the shooting target in these scenarios, low-quality images such as blur and noise are generated, which greatly increases the requirements for camera hardware configuration and costs. Therefore, it is particularly important to provide corresponding image restoration technology to make the image clear for low-quality images shot by the camera, and it is also a problem that must be solved to reduce the cost of the camera.

[0003] In the image shooting process, the shooting time of long exposure is long, and when the target motion is complex, it will produce blur that is difficult to recover, so a relatively short exposure time is generally used in advanced image tasks such as vehicle detection and recognition, and the image shot by short exposure will generate a large amount of noise, which will also reduce the quality of the shot image. In order to balance the size of blur and noise in the image, the exposure time will not be set too long or too short, and such setting will produce an image with slight blur and partial noise. At present, the restoration technology for the noisy and blurred image is still immature. In the prior art, most methods only restore one kind of low-quality image problem of blur or noise. The image deblurring method can only remove the motion blur in the image, and cannot remove the noise generated in the shooting process; similarly, the image denoising method is only effective for noise, and the blur caused by motion still exists. Such method is invalid for the image with blur and noise at the same time, and it is difficult to restore a high-quality clear image. SUMMARY

[0004] The present application is aimed at low-quality images shot by the camera, considers the complexity of the camera imaging process, takes reducing the cost of camera shooting as the starting point, focuses on the problem of low-quality image denoising and deblurring, and proposes an image denoising and deblurring method fusing native features. The method can unify image denoising and deblurring in one framework model, realize high-quality restoration of low-quality images shot by the camera, can be used for upgrading and modification of the installed monitoring equipment, plays a role in reducing cost, reducing resource waste, and improving work efficiency.

[0005] The technical scheme adopted by the present application is as follows:

[0006] Step (1) original data acquisition: first, the original data in the data set is acquired, the data is shot by a high-speed camera, and each frame of data is exported as an image to obtain a clear image, which includes a plurality of scenes.

[0007] Step (2) data set synthesis: according to the clear images obtained in step (1), for each scene, 7-13 clear images successively taken are selected as a group. Each image uses inverse gamma function, inverse color conversion, demosaic, and inverse white balance in turn to convert the clear image from RGB space (red, green, and blue three primary color space) to the original image space. The group of original images is averaged to obtain a blurred original image. The blurred original image is added with signal-related Poisson noise component and signal-independent Gaussian noise component. Then, each noisy blurred original image is sequentially subjected to the camera image processing pipeline, including white balance, demosaic, color conversion, and gamma function mapping, to obtain a noisy blurred RGB image (hereinafter referred to as a blurred image). The middle image of the 7-13 clear images is selected as the clear image corresponding to the blurred image. Finally, the synthesized blurred clear image pair is obtained, and the blurred clear image pairs are generated for each scene, and are divided into a training set and a validation set for training and validation.

[0008] Step (3) data set preprocessing: for each blurred clear image pair used for training, the image pixel size is cropped, and random flipping and random rotation operations are performed to enhance the data set. Then, the blurred and clear image pairs are reshaped to the original image three scales, with the blurred image as the input during network training.

[0009] Step (4) original image estimation network: a three-layer U-shaped network is used for original image estimation. The specific steps are as follows: the (1x1) scale blurred image reshaped in step (3) is input into the three-layer U-shaped network, and convolution operation, residual connection, and ReLU activation function are used in each layer of the network in turn, and down-sampling and up-sampling operations are used between two layers to obtain original image features. The decoded original image features are converted into an image with a channel number of 1 using a convolution layer and output to obtain the estimated original image.

[0010] Step (5) image denoising and deblurring network fusing original features: a multi-scale network fusing original image features is used to realize the functions of denoising and deblurring. The specific steps are as follows: the three scales of blurred images reshaped in step (3) are input into the image denoising and deblurring network, and each scale of image is encoded and decoded, and the features obtained from the previous scale are fused with the features obtained from the current scale in the encoder and decoder. At the same time, the original image features of different scales decoded in step (4) and the RGB image features encoded or decoded in the image denoising and deblurring network are jointly input into the feature fusion module for fusion, and the fused features are added to the image denoising and deblurring network for further encoding and decoding, and finally the denoised and deblurred clear images of different scales are output.

[0011] Step (6) training network parameters: for the estimated image output by the native image estimation network in step (4) and its corresponding clear image, the native image obtained by using the inverse gamma function, inverse color conversion, mosaic, and inverse white balance in turn, the network loss is calculated using the L1 loss function, and the native image estimation network parameter θ1 is trained. For the clear image output by the image denoising and deblurring network fused with native features in step (5) and the three scale clear images obtained by remodeling in step (3), the network loss is calculated using the L1 loss function, and the image denoising and deblurring network fused with native features parameter θ2 is trained.

[0012] Step (7) test new samples in the validation set under the optimal parameters: for a blurred image, use the estimated network parameters θ1 and θ2 in step (6), respectively, into the network in steps (4) and (5), to get the denoised and deblurred clear image.

[0013] The beneficial effects of the present application: the present application uses a fusion native feature method for restoring noisy and blurred images for low-quality images taken by a camera. In the data set stage, the image is converted from RGB space to native image space using the inverse process of the camera processing pipeline, the noise is averaged and added to get the noisy and blurred image in the native space, and the noisy and blurred image for training is obtained through the camera image processing pipeline. In the image denoising and deblurring stage, a three-layer U-shaped network is used to estimate the native image, the decoded native image features are input into the multi-scale network for fusion, and finally the restored clear image is obtained. Through experiments on the synthetic data set, it is proved that the method is superior to the previous method in joint denoising and deblurring, which helps to improve the image quality, reduce the camera cost and reduce the resource waste. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a framework flowchart of the method;

[0015] Figure 2 is a network model diagram of the method;

[0016] Figure 3 is a data set flowchart;

[0017] Figure 4 is an experimental test effect diagram. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the present application clearer, the following will combine the attached Figure 1 , 2 , 3, 4, to further explain the fusion native feature image denoising and deblurring method provided by the present application for noisy and blurred images.

[0019] Step (1) Original data acquisition: First, the original data in the GOPRO dataset is acquired, which is shot by a high-speed camera at a rate of 240 frames per second, and for each frame data, an image is derived, which includes 33 scenes, a total of 34874 clear images.

[0020] Step (2) Data collection: For the clear images acquired in step (1), for each scene, 7-13 clear images successively shot are selected as a group. Each image is converted from RGB space to the native image space using inverse gamma function, inverse color conversion, demosaicing, and inverse white balance in turn. The group of native images is averaged to obtain a blurred native image:

[0021]

[0022] where R blur represents the blurred native image, M represents the number of sampling frames, and R[i] represents the i-th frame of the native image. Gaussian noise and Poisson noise are added to the blurred native image:

[0023] R bn = α(R blur +N s )+N c (2) where R bn represents the noisy blurred native image, R blur represents the blurred native image, N s and N c represent the signal-dependent noise and signal-independent noise introduced by the sensor, respectively, and α is the total gain determined according to the analog gain and digital gain, which is set to 8.8915e-5. (R blur +N s ) is modeled as a signal-dependent Poisson distribution, and N c is modeled as a signal-independent Gaussian distribution:

[0024]

[0025]

[0026] where and represent the Poisson distribution and the Gaussian distribution, respectively. β represents the variance of the Gaussian distribution, which is set to 2.9430e-5. The image processing pipeline of the camera is used in sequence for each native image, including white balance, demosaicing, color conversion, and gamma function mapping, to obtain a noisy blurred image:

[0027]

[0028] where B SynThe noisy blurred image represents the final synthesized image, and the ISP is the simulation of the camera image processing pipeline. The middle image of the 7-13 clear images is selected as the clear image corresponding to the blurred image. Finally, 3214 pairs of synthesized blurred and clear images are obtained, of which 2103 image pairs are used for training and 1111 image pairs are used for testing.

[0029] Step (3) data set preprocessing: for each blurred and clear image pair used for training, the pixel size is cropped from 1280x720 to 256x256 image, and random flipping and random rotation operations are performed to enhance the data set. Then, the blurred and clear image pair is reshaped to the original image Three scales, where the blurred image is used as the input during network training.

[0030] Step (4) native image estimation network: a three-layer U-shaped network is used for native image estimation. The specific steps are as follows:

[0031] 4.1 The (1x1) blurred image B1 reshaped in step (3) is input into the three-layer U-shaped network. After a 3x3 convolution with 32 channels in the first layer network, the number of channels of the image changes from 3 to 32. After passing through 4 residual blocks with 32 channels, it is input into the second layer network.

[0032] 4.2 The second layer network first performs a 3x3 convolution with a step size of 2 and a channel number of 64 on the input feature map to downsample it. After passing through 4 residual blocks with 64 channels, the feature map is transmitted to the third layer network.

[0033] 4.3 The third layer network first performs a 3x3 convolution with a step size of 2 and a channel number of 128 to further downsample the feature map. Then, after passing through 8 residual blocks with 128 channels, the feature at this time is defined as F3.

[0034] 4.4 In the second layer, the feature map is upsampled using a deconvolution operation with a step size of 2 and a channel number of 64. After passing through 4 residual blocks with 64 channels, the feature is defined as F2. Then F2 is passed to the first layer network.

[0035] 4.5 In the first layer, the feature map is upsampled using a deconvolution operation with a step size of 2 and a channel number of 32 to restore the image information. After passing through 4 residual blocks with 32 channels, the feature is defined as F1. Finally, a 3x3 convolution is performed to restore the image channel number to 1, and the estimated native image is obtained.

[0036] Step (5) image denoising and deblurring network that fuses native features: a multi-scale network that fuses native features is used to realize the functions of denoising and deblurring. The specific steps are as follows:

[0037] 5.1 Input the reshaped blurred images B1, B2, and B3 from step (3) into the network. B2 and B3 are obtained by downsampling B1 and B2, respectively. B1 is first input into the encoder EB1 in the first layer of the network. In the encoder EB1, it first goes through a 3×3 convolution with 32 channels, and the number of channels of the image changes from 3 to 32. The feature fusion module is used to fuse the RGB image features with the original image features F1 obtained in step (4.5). After passing through 8 residual modules with 32 channels, it is input into the encoder EB2 in the second layer of the network. (In the feature fusion module, the RGB image features are multiplied with the original image features. The max pooling and average pooling of the multiplied features are concatenated, and the weights are obtained by normalization after 7×7 convolution. The weights are then multiplied with the RGB image features and output.)

[0038] 5.2 In the second layer network EB2, the input feature map is first downsampled by a 3×3 convolution with a stride of 2 and 64 channels. The RGB features are then fused with the original image features F2 obtained in step (4.4) through the feature fusion module. In the feature attention module, the feature map is multiplied by the features extracted by B2 through the shallow convolution operation SCM2, and then fused by a 3×3 convolution with 64 channels. After passing through 8 residual blocks with 64 channels, the feature map is fed into the encoder EB3 in the third layer network. (In SCM2, B2 first inputs two 3×3 convolutions with 16 channels, then inputs a 1×1 convolution with 29 channels. After concatenation with B2 in the channel dimension, it is output by a 1×1 convolution with 32 channels).

[0039] 5.3 In the third layer network EB3, the feature map is first further downsampled by a 3×3 convolution with the same stride of 2 but 128 channels. The RGB features are then fused with the original image features F3 obtained in step (4.3) through the feature fusion module. In the feature attention module, the feature map is multiplied by the features extracted by SCM3 from B3 and fused by a 3×3 convolution with 128 channels. After passing through 8 residual blocks with 128 channels, the feature map is fed into the decoder DB3 in the third layer network. (SCM3 and SCM2 use the same convolution operation. The resulting features are concatenated with B3 in the channel dimension and then output by a 1×1 convolution with 32 channels).

[0040] 5.4 In the third layer of the network DB3, the RGB features are first fused with the original image features F3 through a feature fusion module. Then, after passing through 8 residual modules with 128 channels, a 4×4 deconvolution operation with a stride of 2 and 64 channels is used to upsample the features to the second layer of the network.

[0041] 5.5 In the second layer network, first concatenate the features output by the asymmetric feature fusion module AFF2, then input the decoder DB2. In DB2, go through a 1x1 convolution with 64 channels, a feature fusion module to fuse the RGB features with the native image features F2, 8 residual blocks with 64 channels, and a deconvolution layer with a stride of 2 and 32 channels, and then upsample the features to the first layer network. (In AFF2, concatenate the features of EB1, EB2, EB3 after encoding in the channel dimension, and go through two 1x1 convolutions with 64 channels.)

[0042] 5.6 In the first layer network, first concatenate the features output by the asymmetric feature fusion module AFF1, then input the decoder DB1. In DB1, go through a 1x1 convolution with 32 channels, a feature fusion module to fuse the RGB features with the native image features F1, 8 residual blocks with 32 channels, and finally go through a 3x3 convolution with 3 channels, and then add the original image to obtain the restored image Similarly, in the second and third layers, use a 3x3 convolution with 3 channels to obtain the restored image and (In AFF1, concatenate the features of EB1, EB2, EB3 after encoding in the channel dimension, and go through two 1x1 convolutions with 32 channels.)

[0043] Step (6) Train network parameters:

[0044] According to step (4), train the θ1 parameters of the native image estimation network:

[0045] Use the L1 loss function to calculate the loss between the native image estimated by the network and the native image of the clear image obtained by reshaping in step (3):

[0046]

[0047]

[0048] where, represents the output of the native image estimation network, R s represents the native image of the clear image, represents the Fourier transform operation, and ‖·‖1 represents the L1 loss function.

[0049] According to step (5), train the θ2 parameters of the image denoising and deblurring network that fuses native features:

[0050] Use the L1 loss function to calculate the loss between the restored image output by the network and the clear image obtained by reshaping in step (3):

[0051]

[0052]

[0053] wherein, denotes the output of the kth layer of the image denoising and deblurring network fusing the native features, S k denotes the clear image of the corresponding scale, and ||·||1 denotes the L1 loss function.

[0054] L c1 (θ1), L F1 (θ1), L c2 (θ2), L F2 (θ2) are spliced, and a full connection network with 4 inputs and 1 output is input to obtain the loss L total of the entire network:

[0055] L total = W x Concat(L c1 (θ1), L F1 (θ1), L c2 (θ2), L F2 (θ2)) (10)

[0056] wherein W is a parameter obtained by optimization using the full connection network, and Concat(·) denotes a splicing operation.

[0057] Step (7) testing new samples under optimal parameters: for a blurred image to be restored, the network parameters θ1 and θ2 estimated in step (6) are respectively put into the networks of steps (4) and (5) to obtain the denoised and deblurred clear image. The experimental results are shown in Figure 4 Fig. 2, from which it can be seen that accurate denoised and deblurred images can be obtained by the present application.

Claims

1. A method for image denoising and deblurring by fusing native features, characterized in that, Includes the following steps: Step 1: Obtain the raw data from the dataset, which was captured by a high-speed camera. Export each frame of data as an image to obtain a clear image, which includes several scenes. Step 2: Based on the clear image, for each scene, obtain blurry and clear image pairs, and divide them into training and validation sets; Step 3: For each blurry-to-sharp image pair in the training set, enhance it, and then reconstruct the blurry-to-sharp image pair back to the original image. Three scales; Step 4: Construct a native image estimation network using a three-layer U-shaped network to obtain the estimated native image from the reconstructed (1×1) scale blurred image; Step 5: Construct an image denoising and deblurring network that integrates native features. From the reconstructed blurred images at three scales, obtain clear images at different scales after denoising and deblurring. Step 6: Train the network parameters to obtain the original image estimation network parameters θ1 and the image denoising and deblurring network parameters θ2 that fuse the original features. The specific process is as follows: 6.1 For the estimated image output by the native image estimation network in step 4 and its corresponding clear image, the native image obtained by sequentially applying the inverse gamma function, inverse color conversion, mosaic, and inverse white balance is used. The network loss is calculated using the L1 loss function, and the parameters θ1 of the native image estimation network are trained as follows: The L1 loss function is used to calculate the loss between the original image estimated by the network and the original image of the sharpened image obtained in step 3: in, R represents the output of the native image estimation network. s The original image representing a clear image. represents the Fourier transform operation, and ||·||1 represents the L1 loss function; 6.2 For the clear image output by the image denoising and deblurring network that integrates native features in step 5 and the three clear images at different scales obtained from the reshaping in step 3, the network loss is calculated using the L1 loss function, and the parameters θ2 of the image denoising and deblurring network that integrates native features are trained as follows: The L1 loss function is used to calculate the loss between the restored image output by the network and the clear image obtained from the reconstruction in step 3: in, S represents the output of the k-th layer of the image denoising and deblurring network that incorporates native features. k A clear image representing the corresponding scale; L c1 (θ1), L F1 (θ1), L c2 (θ2), L F2 (θ2) Concatenation: Input a fully connected network with 4 inputs and 1 output to obtain the loss L of the entire network. total : L total =W×Concat(L c1 (θ1),L F1 (θ1),L c2 (θ2),L F2 (θ2)) Where W is the parameter obtained by optimization using a fully connected network, and Concat(·) represents the concatenation operation; Step 7: Obtain the blurred images in the validation set, and use network parameters θ1 and θ2 to feed them into the native image estimation network and the image denoising and deblurring network, respectively, to obtain the clear images after denoising and deblurring.

2. The image denoising and deblurring method according to claim 1, characterized in that, Step 2 is as follows: 2.1 For each scene, select 7 to 13 clear images taken consecutively as a group; 2.

2. For each clear image, apply the inverse gamma function, inverse color conversion, mosaic, and inverse white balance in sequence to convert the clear image from RGB space to the original image space. Average these original images to obtain the blurred original image. 2.

3. For the blurred original image, add a Poisson noise component that is related to the signal and a Gaussian noise component that is independent of the signal; 2.

4. For each noisy and blurred original image, sequentially apply the camera's image processing pipeline, including white balance, demosaic, color conversion, and gamma function mapping, to obtain a noisy blurred RGB image, i.e., a blurred image. Select the middle image of this set of clear images as the clear image corresponding to this blurred image to obtain the synthesized blurred and clear image pair. 2.

5. Iterate through each scene to generate blurred and clear image pairs, and divide them into training set and validation set.

3. The image denoising and deblurring method according to claim 2, characterized in that, The specific process for enhancing the blurred and sharp image in step 3 is as follows: cropping the image to its pixel size and performing random flipping and rotation operations.

4. The image denoising and deblurring method according to claim 3, characterized in that, Step 4 is as follows: 4.1 Input the reshaped (1×1) scale blurred image into a three-layer U-shaped network. In each layer of the network, convolution operation, residual connection and ReLU activation function are used in sequence. Downsampling and upsampling operations are used between two layers to obtain the original image features. 4.

2. Use convolutional layers to transform the original image features into an image with 1 channel and output it to obtain the estimated original image.

5. The image denoising and deblurring method according to claim 4, characterized in that, Step 5 is as follows: The three blurred images after reshaping are input into the image denoising and deblurring network. The image at each scale is encoded and decoded respectively. The features obtained from the previous scale are fused with the features obtained from the current scale in the encoder and decoder. Simultaneously, the original image features of different scales decoded in the original image estimation network in step 4 and the RGB image features encoded or decoded in the image denoising and deblurring network are input into the feature fusion module for fusion. The fused features are then added to the image denoising and deblurring network for further encoding and decoding. Finally, the decoded features are passed through a convolutional layer and added to the reconstructed blurred image of the corresponding scale to output clear images after denoising and deblurring at different scales.

Images