Image super-resolution method based on noise level

Abstract

The present invention relates to image processing technology, and discloses an image super-resolution method based on noise level, which includes a preprocessing module, an image enhancement module and a reconstruction module, and the method includes: constructing an image super-resolution database, sampling image blocks, and processing the image blocks to obtain low-resolution image blocks; processing the low-resolution image blocks through a multi-scale image preprocessing module; enhancing the preprocessed image blocks through an image enhancement module; reconstructing the image blocks through a residual network structure through a reconstruction module; splicing the preprocessed image, the enhanced image and the reconstructed image to form a super-resolution model, and simultaneously training the model's preprocessing module, image enhancement module and reconstruction module. The present invention can achieve better super-resolution effects through the preprocessing module, the enhancement module and the reconstruction module, so that users can have a clear visual experience when watching on a high-definition device.

Description

Technical Field

[0001] The invention relates to image processing technology, and in particular to an image super-resolution method based on noise level. Background Art

[0002] Image super-resolution is to restore high-resolution images from low-resolution images, for example, to double the resolution of a picture from 960*540 to 1920*1080, so that users can watch it on large-size display devices. Image super-resolution is one of the basic problems in image processing and has a wide range of practical needs and application scenarios. Especially in the fields of medical images, satellite images, and videos, when low-resolution images of the same scene are easily available, using super-resolution reconstruction technology to obtain high-resolution images can better help subsequent image processing operations.

[0003] In recent years, with the continuous development of deep learning technology, convolutional neural networks (CNNs) have received widespread attention in the field of computer vision and achieved remarkable results. In the field of image super-resolution, CNNS have also made great contributions.

[0004] At present, most super-resolution methods based on deep learning have shown excellent performance and can obtain high objective image quality evaluation indicators, such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). However, from a subjective point of view, the images super-resolved by these models are often too smooth. In practical applications, especially on some high-definition devices, such images cannot give people a clear visual experience. Summary of the invention

[0005] Aiming at the problem that the super-resolved image in the prior art is too smooth and the image display on some high-definition equipment is not clear, the present invention provides an image super-resolution method based on noise level.

[0006] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0007] The image super-resolution method based on noise level includes a preprocessing module, an image enhancement module and a reconstruction module, and the method includes:

[0008] Image data preparation, building an image super-resolution database, sampling image blocks, and processing image blocks to obtain low-resolution image blocks;

[0009] Image preprocessing: low-resolution image blocks are processed through a multi-scale image preprocessing module;

[0010] Image enhancement: The preprocessed image blocks are enhanced through the image enhancement module;

[0011] Image reconstruction, through the reconstruction module, the image block is reconstructed;

[0012] To train the model, the preprocessing module, image enhancement module and reconstruction module are connected to form a super-resolution model, and these three modules are trained simultaneously.

[0013] Preferably, data preparation includes,

[0014] Obtain high-resolution images. Obtain high-resolution images through image databases;

[0015] Acquisition of image blocks: for the acquired high-resolution image, sampling of the image blocks is performed by setting the image block size to acquire multiple image blocks;

[0016] The low-resolution image is obtained, and the image block is blurred, denoised and down-sampled to obtain a low-resolution image block.

[0017] Preferably, the image preprocessing module includes 1 convolution layer, 9 DoubleConv2d modules, 4 downsampling layers, 4 upsampling layers and 4 concatenation layers; the kernel size of the first convolution layer is 3*3, the number is 16, the filling mode is SAME, and the step size is 1; the kernel size of the convolution layer of the DoubleConv2d module is 3*3, the number is the same as the input feature Figure 1 The downsampling layer uses a convolutional layer with a stride of 2, the kernel size is 3*3, the number is twice that of the input feature map, and the filling mode is SAME; the upsampling uses bilinear interpolation for sampling; the concatenation layer concatenates the two input feature maps in the channel dimension.

[0018] Preferably, the enhancement module is a noise level enhancement module, including 3 convolutional layers, 2 upsampling layers, 1 LeakyReLU activation layer and 1 transformation operation; wherein the kernel size of the convolutional layers is 3*3, the filling mode is SAME, the step size is 1, and the number of kernels is 3, 3, and 16 respectively;

[0019] Upsampling uses bilinear interpolation method;

[0020] The negative slope parameter of the LeakyReLU activation layer is 0.1;

[0021] The input of the transformation operation is a vector of noise level parameters corresponding to the low-resolution image, which is linearly transformed with the matrix parameters of a randomly initialized trainable normal distribution.

[0022] Preferably, the reconstruction module is an image reconstruction module based on a residual module, including 1 convolution layer, 6 ResBlock modules, 2 convolution layers, and 1 ADD layer. The input is the feature map after the image enhancement module, and the output is the reconstructed super-resolution result.

[0023] ResBlock consists of 1 convolutional layer, 1 LeakyReLU activation layer, and 1 convolutional layer;

[0024] The kernel size of the convolutional layers is 3*3. Except for the last convolutional layer, which has 3 kernels, the rest have 64 kernels and the convolution step size is 1.

[0025] The negative slope parameter of the LeakyReLU activation layer is 0.1.

[0026] Preferably, the training of the model includes the calculation of a loss function, which is:

[0027]

[0028] in, Represents the pixel value of a clear, high-resolution image at position (x, y), Represents a low-resolution image I LR The pixel value of the image at position (x, y) after super-resolution by the super-resolution model G, W and H are the width and height of the image respectively;

[0029] Training parameters

[0030] The training parameters are optimized by the Adam algorithm, where the training parameters are set and the initial learning rate is set to 10 -4 , the number of training iterations is set to 200 epochs, and the learning rate is reduced by 1 / 2 every 50 epochs.

[0031] The present invention has significant technical effects due to the adoption of the above technical solution:

[0032] The input of the model of the present invention is a low-resolution image. The image features are extracted through the preprocessing module, and a preliminary super-resolution image is obtained in the enhancement module. Prior information based on the corresponding noise level is introduced, and finally it is adjusted to high-frequency information through the reconstruction module to obtain a clearer super-resolution image.

[0033] The present invention introduces prior information based on noise level by constructing an enhancement module, and can obtain a clearer super-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is the framework of the super-resolution model of the present invention

[0035] Among them, PreProcess represents the preprocessing module, Enhance represents the enhancement module, and Reconstruction represents the reconstruction module. The input of this framework is a low-resolution image, and the output is a high-resolution image after super-resolution.

[0036] Figure 2 It is a structural diagram of the preprocessing module of the present invention

[0037] Among them, Conv2d represents the convolution layer, DoubleConv2d represents Figure 3 The double-layer convolution structure shown in the figure, Downsampling represents the downsampling layer, Upsampling represents the upsampling layer, and Concate represents the concatenation layer; I LR represents a low-resolution image, X out Represents the output of the preprocessing module.

[0038] Figure 3 This is the structural diagram of the DoubleConv2d module of the present invention

[0039] Among them, Conv2d represents the convolution layer, and LeakyReLU represents the LeakyReLU activation layer;

[0040] Figure 4 This is a structural diagram of the enhancement module of the present invention.

[0041] Among them, I LR represents a low-resolution image, X mid_out represents the preliminary super-resolution result outputted by the enhancement module, P noise Represents the noise level parameter corresponding to the low-resolution image, and Transform represents the transformation operation performed on the noise level parameter;

[0042] Figure 5 is a structural diagram of the reconstruction module of the present invention

[0043] Among them, ResBlock represents Figure 6 The residual structure shown, X mid_out represents the preliminary super-resolution result output in the enhancement module, Represents the pixel addition layer, G(I LR ) represents the high-resolution image after super-resolution;

[0044] Figure 6 This is a structural diagram of the residual block of the present invention.

[0045] Among them, X in Represents the input of residualblock, X out Represents the output of residualblock. DETAILED DESCRIPTION

[0046] The following is combined with Figure 1-6 The present invention is further described in detail with examples.

[0047] Example 1

[0048] The image super-resolution method based on noise level includes a preprocessing module, an image enhancement module and a reconstruction module, and the method includes:

[0049] Image data preparation, building an image super-resolution database, sampling image blocks, and processing image blocks to obtain low-resolution image blocks;

[0050] Image preprocessing: low-resolution image blocks are processed through a multi-scale image preprocessing module;

[0051] Image enhancement: The preprocessed image blocks are enhanced through the image enhancement module;

[0052] Image reconstruction, through the reconstruction module, the image block is reconstructed;

[0053] To train the model, the preprocessing module, image enhancement module and reconstruction module are connected to form a super-resolution model, and these three modules are trained simultaneously.

[0054] Data preparation includes:

[0055] Obtain high-resolution images. Obtain high-resolution images through image databases;

[0056] Acquisition of image blocks: for the acquired high-resolution image, sampling of the image blocks is performed by setting the image block size to acquire multiple image blocks;

[0057] The low-resolution image is obtained, and the image block is blurred, denoised and down-sampled to obtain a low-resolution image block.

[0058] The image preprocessing module includes 1 convolution layer, 9 DoubleConv2d modules, 4 downsampling layers, 4 upsampling layers and 4 concatenation layers. The kernel size of the first convolution layer is 3*3, the number is 16, the padding mode is SAME, and the step size is 1. The kernel size of the convolution layer of the DoubleConv2d module is 3*3, and the number is the same as the input feature Figure 1 The downsampling layer uses a convolutional layer with a stride of 2, the kernel size is 3*3, the number is twice that of the input feature map, and the filling mode is SAME; the upsampling uses bilinear interpolation for sampling; the concatenation layer concatenates the two input feature maps in the channel dimension.

[0059] The enhancement module is a noise level enhancement module, which includes 3 convolutional layers, 2 upsampling layers, 1 LeakyReLU activation layer and 1 transformation operation; the kernel size of the convolutional layers is 3*3, the filling mode is SAME, the step size is 1, and the number of kernels is 3, 3, and 16 respectively;

[0060] Upsampling uses bilinear interpolation method;

[0061] The negative slope parameter of the LeakyReLU activation layer is 0.1;

[0062] The input of the transformation operation is a vector of noise level parameters corresponding to the low-resolution image, which is linearly transformed with the matrix parameters of a randomly initialized trainable normal distribution.

[0063] The reconstruction module is an image reconstruction module based on the residual module, including 1 convolution layer, 6 ResBlock modules, 2 convolution layers, and 1 ADD layer. The input is the feature map after the image enhancement module, and the output is the reconstructed super-resolution result.

[0064] ResBlock consists of 1 convolutional layer, 1 LeakyReLU activation layer, and 1 convolutional layer;

[0065] The kernel size of the convolutional layers is 3*3. Except for the last convolutional layer, which has 3 kernels, the rest have 64 kernels and the convolution step size is 1.

[0066] The negative slope parameter of the LeakyReLU activation layer is 0.1.

[0067] The training of the model includes

[0068] Calculation of loss function, the loss function is:

[0069]

[0070] in, Represents the pixel value of a clear, high-resolution image at position (x, y), Represents a low-resolution image I LR The pixel value of the image at position (x, y) after super-resolution by the super-resolution model G, W and H are the width and height of the image respectively;

[0071] Training parameters

[0072] The training parameters are optimized by the Adam algorithm, where the training parameters are set and the initial learning rate is set to 10 -4 , the number of training iterations is set to 200 epochs, and the learning rate is reduced by 1 / 2 every 50 epochs.

[0073] Example 2

[0074] Based on Example 1, the image super-resolution method based on noise level in this embodiment includes: preparing data

[0075] Step 1.1: The dataset consists of five public image databases, namely Vimeo, RealSR, REDS, DIV2K and Flickr2K. Vimeo and REDS are video datasets, and each scene sequence consists of multiple frames of continuous images. Each database provides high-resolution images, and some databases provide corresponding low-resolution images. The present invention only uses high-resolution images, and low-resolution images are generated from high-resolution images. The content, size, and number of images contained in each database are different.

[0076] Step 1.2: For the Vimeo and REDS datasets, in order to avoid image content duplication, only one frame of each scene sequence is taken. Image blocks are sampled for each high-resolution image, and the image block size is 128*128*3. In order to increase the amount of data, multiple image blocks are randomly sampled for large-size images. For example, 4 128*128*3 image blocks are randomly sampled for a high-definition image with a size of 1920*1080*3, and these image blocks are used as labels for super-resolution training.

[0077] Step 1.3: Blur, add noise, and downsample the high-definition 128*128*3 image block to obtain the corresponding 64*64*3 low-resolution image block. The blurring process is to blur the image with different blur kernels, including isotropic Gaussian blur kernel, anisotropic Gaussian blur kernel, and motion blur kernel. The noise adding process adds Gaussian noise of random noise level to the image. The noise level parameter and the corresponding low-resolution image are both inputs of the super-resolution model. Downsampling is to perform 0.5 times bicubic interpolation on the image.

[0078] Step 2: Preprocessing module (PreProcess)

[0079] The present invention constructs a multi-scale image preprocessing module, the network structure is as follows Figure 2 As shown. Specifically, it includes 1 convolution layer, 9 DoubleConv2d modules, 4 downsampling layers, 4 upsampling layers and 4 concatenation layers. Among them, the kernel size of the first convolution layer is 3*3, the number is 16, the filling mode is SAME, and the step size is 1. The DoubleConv2d module structure is as follows Figure 3 , the kernel size of the convolution layer is 3*3, and the number is similar to the input features Figure 1The padding mode is SAME and the stride is 1. The convolution layer with a stride of 2 is used for downsampling, and the kernel size is 3*3, which is twice the number of the input feature map, and the padding mode is SAME. The upsampling method uses bilinear interpolation. The concatenation layer concatenates the two input feature maps in the channel dimension.

[0080] Step 3: Enhance

[0081] The present invention constructs an enhancement module based on noise level, and the network structure is as follows: Figure 4 As shown. Specifically, it includes 3 convolutional layers, 2 upsampling layers, 1 LeakyReLU activation layer and 1 transformation operation. Among them, the kernel size of the convolutional layer is 3*3, the filling mode is SAME, the step size is 1, and the number of kernels is 3, 3, and 16 respectively. The upsampling method uses bilinear interpolation. The negative slope parameter of the LeakyReLU activation layer is 0.1. The input of the transformation operation is a vector composed of the noise level parameters corresponding to the low-resolution image, which is linearly transformed with the matrix parameters of the randomly initialized trainable normal distribution.

[0082] Step 4: Reconstruction

[0083] The present invention constructs an image reconstruction module based on the residual module, which specifically has 1 convolution layer, 6 ResBlock modules, 2 convolution layers, and 1 ADD layer. Figure 5 The input is the feature map after the image enhancement module, and the output is the reconstructed super-resolution result. Among them, ResBlock consists of 1 convolutional layer, 1 LeakyReLU activation layer, and 1 convolutional layer, such as Figure 6 The kernel size of the convolutional layers is 3*3, except for the last convolutional layer, which has 3 kernels. The rest have 64 kernels and the convolution step size is 1. The negative slope parameter of the LeakyReLU activation layer is 0.1.

[0084] Step 5: Model training

[0085] Connect the modules of step 2, step 3 and step 4 to form a super-resolution model, and train these three modules at the same time.

[0086] Step 5.1: Loss Function

[0087]

[0088] in, Represents the pixel value of a clear, high-resolution image at position (x, y), Represents a low-resolution image I LRThe pixel value of the image at position (x, y) after super-resolution by the super-resolution model G, W and H are the width and height of the image respectively.

[0089] Step 5.2: Training parameters

[0090] Set the training parameters and set the initial learning rate to 10 -4 The number of training iterations is set to 200 epochs. The learning rate is reduced by 1 / 2 every 50 epochs. The optimization algorithm uses the Adam algorithm.

Claims

1. The image super-resolution method based on noise level is characterized by: It includes a preprocessing module, an image enhancement module and a reconstruction module, and the method includes: Image data preparation, building an image super-resolution database, sampling image blocks, and processing image blocks to obtain low-resolution image blocks; Image preprocessing: low-resolution image blocks are processed through a multi-scale image preprocessing module; Image enhancement: The preprocessed image blocks are enhanced through the image enhancement module; Image reconstruction, through the reconstruction module, the image block is reconstructed; Model training: connect the preprocessing module, image enhancement module and reconstruction module to form a super-resolution model, and train these three modules simultaneously; The image preprocessing module includes 1 convolution layer, 9 DoubleConv2d modules, 4 downsampling layers, 4 upsampling layers and 4 splicing layers; among them, the kernel size of the first convolution layer is 3*3, the number is 16, the filling mode is SAME, and the step size is 1; the kernel size of the convolution layer of the DoubleConv2d module is 3*3, the number is the same as the input feature map, the filling mode is SAME, and the step size is 1; the convolution layer with a step size of 2 used for downsampling, the kernel size is 3*3, the number is twice that of the input feature map, and the filling mode is SAME; upsampling uses bilinear interpolation method for sampling; the splicing layer splices two input feature maps in the channel dimension; The enhancement module is a noise level enhancement module, which includes 3 convolutional layers, 2 upsampling layers, 1 LeakyReLU activation layer and 1 transformation operation; the kernel size of the convolutional layers is 3*3, the filling mode is SAME, the step size is 1, and the number of kernels is 3, 3, and 16 respectively; Upsampling uses bilinear interpolation method; The negative slope parameter of the LeakyReLU activation layer is 0.1; The input of the transformation operation is a vector of noise level parameters corresponding to the low-resolution image, which is linearly transformed with the matrix parameters of a randomly initialized trainable normal distribution.

2. The image super-resolution method based on noise level according to claim 1, characterized in that: Data preparation includes: Obtain high-resolution images. Obtain high-resolution images through image databases; Acquisition of image blocks: for the acquired high-resolution image, sampling of the image blocks is performed by setting the image block size to acquire multiple image blocks; The low-resolution image is obtained, and the image block is blurred, denoised and down-sampled to obtain a low-resolution image block.

3. The image super-resolution method based on noise level according to claim 1, characterized in that: The reconstruction module is an image reconstruction module based on the residual module, including 1 convolution layer, 6 ResBlock modules, 2 convolution layers, and 1 ADD layer. The input is the feature map after the image enhancement module, and the output is the reconstructed super-resolution result. ResBlock consists of 1 convolutional layer, 1 LeakyReLU activation layer, and 1 convolutional layer; The kernel size of the convolutional layer is 3*3. Except for the last convolutional layer, the kernel number is 3, and the rest are 64, and the convolution step size is 1. The negative slope parameter of the LeakyReLU activation layer is 0.

1.

4. The image super-resolution method based on noise level according to claim 1, characterized in that: The training of the model includes Calculation of loss function, the loss function is: in, Represents the pixel value of a clear, high-resolution image at position (x, y), Represents a low-resolution image I LR The pixel value of the image at position (x, y) after super-resolution by the super-resolution model G, W and H are the width and height of the image respectively; Training parameters The training parameters are optimized by the Adam algorithm, where the training parameters are set and the initial learning rate is set to 10 -4 , the number of training iterations is set to 200 epochs, and the learning rate is reduced by 1 / 2 every 50 epochs.

Images