Image bit depth expansion method based on convolutional neural network with noise fusion

Abstract

The present invention discloses a method for expanding the image bit depth based on a convolutional neural network with fusion noise. The method comprises the following steps: (1) preparing a data set, adding an appropriate amount of Gaussian noise, and then quantizing it. The noise variance depends on the quantization level, and generating corresponding training and test sets; (2) building a multi-stage feature fusion convolutional neural network model, including two convolution layers, four dense connection modules, and a feature attention module; (3) training the multi-stage feature fusion convolutional neural network model and updating the model parameters; (4) inputting the image to be processed into the trained neural network model and outputting a high-bit image. The method of the present invention can effectively restore low-bit images and obtain high-quality high-bit images. In particular, for extremely low-bit images, it can effectively solve problems such as artifacts and distortion and restore them to standard bit depth images.

Description

Technical Field

[0001] The present invention relates to the fields of image processing and computer vision, and in particular to a noise-fused image bit depth expansion method based on a convolutional neural network. Background Art

[0002] Image bit depth refers to the number of bits in each channel of an image pixel, which determines the total number of colors that can be described by that pixel. Bit depth expansion refers to the process of reconstructing a high bit depth (HBD) image from a corresponding low bit depth (LBD) image. The goal is to expand the image bit depth, restore some lost details or recover color distortion in the low-bit image, and preserve the texture information and image details of the real image as much as possible. The difficulty of the task lies in removing the severe artifacts and color distortion caused by quantization. Existing image bit depth algorithms are divided into two categories: traditional algorithms and deep learning-based algorithms.

[0003] For extremely low-bit images (e.g., 2-bit images) with less than 4 bits, severe visual artifacts and color distortion occur due to the large quantization process involved in their generation. Furthermore, in real-world situations, sensors are inevitably affected by noise, including thermal noise and shot noise. Current image bit depth expansion algorithms still cannot achieve satisfactory restoration results for extremely low-bit images. However, there is still a significant demand for extremely low-bit image restoration in practice, making this task of significant research significance and application value. Summary of the Invention

[0004] The object of the present invention is to provide an image bit depth method based on convolutional neural network with fusion noise, which helps to restore extremely low-bit images to high-quality standard images.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The noise-fused image bit depth expansion method based on convolutional neural network includes the following steps:

[0007] Step S1, prepare the training dataset: add different levels of Gaussian noise to the public dataset according to the quantization level, and then quantize it to the corresponding number of bits to generate the training set and test set respectively;

[0008] Step S2: Building a multi-stage feature fusion convolutional neural network model, which includes two convolutional layers, four densely connected modules, and a feature attention module. The outputs of the two convolutional layers and the four densely connected modules are connected to the input of the feature attention module.

[0009] Step S3, training the multi-stage feature fusion convolutional neural network model with the goal of minimizing the loss function to form an image bit depth expansion network model;

[0010] Step S4: input the noisy low-bit test image into the trained image bit depth expansion network model, and output the restored image.

[0011] Furthermore, the specific steps of step S1 are:

[0012] Step S11: For bit depth expansion tasks with different spans, different levels of quantization are performed on the public dataset to obtain quantized images. To preserve more structural information and simulate a noisy environment, appropriate noise is added before quantization. Assuming that the bit depth of the image before quantization is m and the bit depth of the image after quantization is n, Gaussian noise with a mean of 0 and a variance of σ is added. The size of σ can be calculated as follows:

[0013]

[0014] Step S12: Cut at the same position of the original image and the quantized image to obtain multiple sets of paired images of input and label, and perform the same flipping and rotation operations on the image pairs to enhance the training set data.

[0015] Furthermore, in the convolutional neural network model of multi-stage feature fusion described in step S2, each densely connected module is composed of five residual sub-modules densely connected, and a convolution layer is added between the residual sub-modules to fuse the features of each channel; the features output by each densely connected module are combined together in the channel dimension and then connected to the feature attention module.

[0016] Furthermore, the residual submodule includes 6 convolutional layers and 2 skip connections, wherein the second and third convolutional layers fuse ordinary convolution and dilated convolution; skip connections are added between the output of the first convolutional layer and the output of the fourth convolutional layer, and between the input of the residual submodule and the output of the sixth convolutional layer; a weight module is added to each skip connection, and the input of the weight module passes through 2 convolutional layers and a sigmoid activation function to obtain a learnable weight distribution, and is multiplied by the input of the weight module to obtain the output.

[0017] Furthermore, the combination of the output of the two convolutional layers in step S2 and the output of the four densely connected modules in the channel dimension is used as the input of the feature attention module. The feature attention module divides the input signal into two paths, each of which is connected to two convolutional layers. One path adds the LeakyReLU activation function to obtain the features of the fusion of each stage, and the other path adds the Sigmoid activation function to obtain the attention distribution map. The results of the two paths are then multiplied, and finally pass through a convolution layer to obtain the output of the feature attention module.

[0018] Furthermore, the loss function in step S3 is a combination of the MSE loss function and the perceptual loss function. The formula is as follows, where the first term is the MSE loss function and the second term is the perceptual loss function:

[0019]

[0020] Among them, f represents the image bit depth expansion network model, Represents the quantized low-bit image, I i Represents the original high-bit image, ZP represents the zero-filling operation, F represents the pre-trained VGG network, N represents the total number of pixels, and α is a parameter.

[0021] Furthermore, the low-bit test image with noise in step S4 is specifically 2-4 bits, and the restored image is 8 bits.

[0022] Compared to existing technologies, this invention takes noise into account and leverages the presence of noise to restore low-bit image depth, yielding high-quality, high-bit image quality. In particular, this invention effectively addresses artifacts and distortion in very low-bit images, restoring them to standard bit depth. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Flowchart of the method of the present invention.

[0024] Figure 2 A diagram of the neural network structure used in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Reference Figure 1 In this embodiment, a noise-fused image bit depth method based on a convolutional neural network is described, and the specific steps are as follows:

[0027] Step S1: Prepare a training dataset. Add different levels of Gaussian noise to the public dataset according to the quantization level, and then quantize it to the corresponding number of bits to form a training set and a test set respectively.

[0028] For bit depth expansion tasks with different spans, we perform different levels of quantization on public datasets to obtain input. To preserve more structural information and simulate real-world noisy environments, we add appropriate noise before quantization. Assuming the image bit depth before quantization is m and the image bit depth after quantization is n, we add Gaussian noise with a mean of 0 and a variance of σ. The magnitude of σ can be calculated using the following formula:

[0029]

[0030] The noise added in this embodiment is specifically as follows:

[0031] 2-bit-8-bit: Add Gaussian noise with a mean of 0 and a variance of 8.5 to the dataset, and normalize it after dividing it by 64.

[0032] 3-bit-8-bit: Add Gaussian noise with a mean of 0 and a variance of 4.25 to the dataset, and normalize it after dividing it by 32.

[0033] 4-bit-8-bit: Add Gaussian noise with a mean of 0 and a variance of 2.125 to the dataset, and normalize the noise after dividing it by 16.

[0034] Cut at the same position of the original image and the quantized image to obtain multiple sets of input and label paired images, and perform the same flip and rotation operations on the image pairs to enhance the training set data.

[0035] Step S2: Build a multi-stage feature fusion convolutional neural network model, which includes two convolutional layers, four dense connection modules and a feature attention module, such as Figure 2 shown.

[0036] Each densely connected module consists of five densely connected residual submodules, with a convolutional layer added between each residual submodule to fuse the features of each channel. The features output by each densely connected module are combined together in the channel dimension and then connected to the feature attention module.

[0037] The residual submodule consists of 6 convolution layers with a kernel size of 3×3 and a stride of 1 and 2 skip connections. The specific structure is as follows: The input is represented by F r1 , first passes through a convolution layer with a convolution kernel size of 3×3 and a stride of 1, and the feature representation is F r2 ; Then it is divided into two paths, feature F r2 After a normal convolution and a dilated convolution with a dilation factor of 2, the two features are connected and then connected again with a normal convolution and a dilated convolution with a dilation factor of 2; then the two features are connected in the channel dimension and passed through a convolution layer with a convolution kernel size of 3×3 and a stride of 1 to obtain the feature F. r3 , feature F r2 After a weight module and F r3 Add; then pass through two ordinary convolution layers with a convolution kernel size of 3×3 and a step size of 1 to obtain the feature F r4 , and finally input feature F r1 After the weight module and F r4 Add together to get the output of the residual submodule, which is denoted as F r5The LeakyReLU activation function is added after each convolution layer. The weight module is composed as follows: Assume that the input feature is represented by F w1 First, after two convolution layers with a convolution kernel size of 3×3 and a step size of 1 and a sigmoid activation function, the weight distribution map F is obtained. w2 , F w2 With input F w1 Multiply as the final feature output F w3 .

[0038] The input of the feature attention module is the combination of the output of the first and second convolutional layers of the network and the output of the four dense residual connection modules in the channel dimension. If the output features of the two convolutional layers and the output features of the four dense residual connection modules are represented as F a1 、F a2 、F a3 、F a4 、F a5 , the feature attention module first converts F a1 、F a2 、F a3 、F a4 、F a5 Connect together in the channel dimension, then divide into two paths and connect two convolution layers with a convolution kernel size of 3×3 and a step size of 1. One path uses the LeakyReLU activation function to obtain the feature F a6 , the other way uses Sigmoid activation function to get the attention distribution map F a7 , feature F a6 With F a7 After multiplication, it passes through a convolution layer with a convolution kernel size of 3×3 and a step size of 1 and a LeakyReLU activation function. The output of the feature attention module is represented as F a8 .

[0039] Step S3: Based on the constructed network model, the multi-stage feature fusion network model is trained with the goal of minimizing the loss function to form an image bit depth expansion network model.

[0040] The loss function used is a combination of the MSE loss function and the perceptual loss function. The formula is as follows: the first term is the MSE loss function, and the second term is the perceptual loss function.

[0041]

[0042] Among them, f represents the image bit depth expansion network model, Represents the quantized low-bit image, I i Represents the original high-bit image, ZP represents the zero-filling operation, F represents the pre-trained VGG network, N represents the total number of pixels, and α is a parameter.

[0043] Step S4: Input the low-bit test set with noise into the trained image bit depth expansion model and output the restored image.

[0044] The images used in this embodiment are low-bit images quantized after adding noise to the public data set other than the training data set, specifically 2-4 bits, which are input into the corresponding image bit depth expansion network (2-8bit, 3-8bit, 4-8bit) to restore the corresponding 8-bit standard bit depth image.

Claims

1. A noise-infused image bit depth expansion method based on a convolutional neural network, characterized in that: The following steps are involved: Step S1, prepare the training dataset: add different levels of Gaussian noise to the public dataset according to the quantization level, and then quantize it to the corresponding number of bits to generate the training set and test set respectively; Step S2: Building a multi-stage feature fusion convolutional neural network model, which includes two convolutional layers, four densely connected modules, and a feature attention module. The outputs of the two convolutional layers and the four densely connected modules are connected to the input of the feature attention module. Step S3, training the multi-stage feature fusion convolutional neural network model with the goal of minimizing the loss function to form an image bit depth expansion network model; Step S4: input the noisy low-bit test image into the trained image bit depth expansion network model, and output the restored image.

2. The image bit depth expansion method based on convolutional neural network with fusion noise according to claim 1, characterized in that: The specific steps of step S1 are: Step S11: For bit depth expansion tasks with different spans, different levels of quantization are performed on the public dataset to obtain quantized images. To preserve more structural information and simulate a noisy environment, appropriate noise is added before quantization. Assuming that the bit depth of the image before quantization is m and the bit depth of the image after quantization is n, Gaussian noise with a mean of 0 and a variance of σ is added. The size of σ can be calculated as follows: Step S12: Cut at the same position of the original image and the quantized image to obtain multiple sets of paired images of input and label, and perform the same flipping and rotation operations on the image pairs to generate a training set.

3. The image bit depth expansion method based on convolutional neural network and fusion noise according to claim 1 is characterized in that: In the multi-stage feature fusion convolutional neural network model described in step S2, each densely connected module is composed of five residual sub-modules densely connected, and a convolution layer is added between the residual sub-modules to fuse the features of each channel; the features output by each densely connected module are combined together in the channel dimension and then connected to the feature attention module.

4. The image bit depth expansion method based on convolutional neural network and fusion noise according to claim 3 is characterized in that: The residual submodule contains 6 convolutional layers and 2 skip connections, where the second and third convolutional layers fuse ordinary convolution and dilated convolution; skip connections are added between the output of the first convolutional layer and the output of the fourth convolutional layer, and between the z input of the residual submodule and the output of the sixth convolutional layer; each skip connection adds a weight module, the input of the weight module passes through 2 convolutional layers and a sigmoid activation function to obtain a learnable weight distribution, and is multiplied by the input of the weight module to obtain the output.

5. The image bit depth expansion method based on convolutional neural network and fusion noise according to claim 1 is characterized in that: The combination of the outputs of the two convolutional layers in step S2 and the outputs of the four densely connected modules in the channel dimension is used as the input of the feature attention module. The feature attention module divides the input signal into two paths, each of which is connected to two convolutional layers. One path adds the LeakyReLU activation function to obtain the fused features of each stage, and the other path adds the Sigmoid activation function to obtain the attention distribution map. The results of the two paths are then multiplied, and finally pass through a convolution layer to obtain the output of the feature attention module.

6. The image bit depth expansion method based on convolutional neural network and fusion noise according to claim 1 is characterized in that: The loss function in step S3 is a combination of the MSE loss function and the perceptual loss function. The formula is as follows, where the first term is the MSE loss function and the second term is the perceptual loss function: Among them, f represents the image bit depth expansion network model, Represents the quantized low-bit image, I i Represents the original high-bit image, ZP represents the zero-filling operation, F represents the pre-trained VGG network, N represents the total number of pixels, and α is a parameter.

7. The image bit depth expansion method based on convolutional neural network and fusion noise according to claim 1 is characterized in that: The low-bit test image with noise in step S4 is specifically 2-4 bits, and the restored image is 8 bits.

Images