A deep learning-based method for ultra-high-resolution image denoising

Abstract

The present invention discloses a method for ultra-high-resolution image denoising based on deep learning, which belongs to the technical field of image processing. The method comprises the following steps: step 1, constructing an ultra-high-resolution image cutting and coding model; step 2, inputting a noisy image and a clean image into the cutting and coding model to obtain image blocks; step 3, normalizing the image blocks to different degrees, constructing an image block sequence, and obtaining training and test data; step 4, constructing an image denoising network model; step 5, inputting the image blocks into the image denoising network model to obtain clean image blocks; step 6, decoding the clean image blocks output by the network and splicing them into the original image to obtain a denoised ultra-high-resolution image. The method proposed in the present invention has achieved good results on ultra-high-resolution images, and the network model can be trained end-to-end, which is highly practical in actual ultra-high-resolution image processing scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an ultra-high-resolution image denoising method based on deep learning. Background Art

[0002] With advances in imaging technology, the resolution of images captured by cameras has greatly increased, resulting in the emergence of many ultra-high-resolution images (e.g., resolution exceeding 100 million pixels). People's demands for the quality of such images are also increasing. However, compared to ordinary images, these ultra-high-resolution images generally have higher noise levels and cannot meet people's visual needs.

[0003] Image denoising is a fundamental technology in image processing. Due to its superior denoising performance, current cutting-edge denoising methods are generally based on deep learning. However, the greatest difficulty in using deep learning to process ultra-high-resolution images is the issue of computational resources. Existing deep learning-based denoising methods typically directly input images into the network for training and inference to obtain denoised images. However, these methods involve numerous convolution operations, which consume a lot of computational resources and prevent the input image from being too large. Therefore, such ultra-large-scale images cannot be directly applied to these denoising methods. One solution is to downsample such images to a smaller size before performing network processing. However, due to the loss of a large amount of information during the downsampling process, this method has very poor processing effects. Summary of the Invention

[0004] In response to the technical problem that most existing denoising methods are only applicable to ordinary low-resolution images and cannot be directly used for ultra-high-resolution images, the present invention proposes an ultra-high-resolution image denoising method based on deep learning.

[0005] In order to solve the above problems, the present invention adopts the following technical solutions:

[0006] A deep learning-based ultra-high-resolution image denoising method comprises the following steps:

[0007] Step 1: obtain an ultra-high resolution image as a clean image and obtain its noisy image by adding noise;

[0008] Step 2: performing quadtree cutting and position encoding on the clean image and the noisy image respectively to obtain clean image blocks and noisy image blocks of different sizes;

[0009] Step 3: normalize the clean image blocks and the noisy image blocks to different degrees to make all image blocks of the same size, construct a clean image block sequence and a noisy image block sequence, and obtain training and test data;

[0010] Step 4: Build an image denoising network model, which includes three stages: preliminary denoising, fine denoising, and result fine-tuning. Each stage includes downsampling to enhance denoising performance.

[0011] Step 5: input the noisy image block sequence into the image denoising network model to obtain clean image blocks;

[0012] In step 6, the clean image blocks obtained in step 5 are decoded and spliced ​​to obtain a denoised ultra-high resolution image.

[0013] Furthermore, when performing quadtree cutting in step 2, the noise image is first regarded as the root node of a tree, and it is determined whether the absolute value of the difference between the maximum value and the minimum value of all pixel values ​​therein is less than a threshold value. If it is less than, it is considered that the current node already belongs to an area with simple texture and the quadrisection is stopped. If it is greater than, it is considered that the texture of the current node is still relatively complex and the quadrisection is continued. The above process is repeated until all nodes meet the preset conditions or the stop condition.

[0014] Furthermore, the preset condition includes that the size of the current node reaches a minimum value of 128*128.

[0015] Furthermore, the preset condition also includes a maximum size of the current node of 2048*2048. If the size of the current node is greater than the maximum size when the current node meets the stopping condition, the node is forcibly divided into four parts.

[0016] Furthermore, in step 2, the position encoding takes the position of the first pixel in the upper left corner of the node on the original ultra-high resolution image, and the size of the node is calculated at the same time. The position encoding on the noise map is performed synchronously with the quadtree cutting.

[0017] Furthermore, in step 2, the noise image is first quad-tree cut and position-coded to obtain all image blocks, and then the image blocks at corresponding positions on the clean image are found according to the position coding, and cut and made to correspond one-to-one with the image blocks of the noise image.

[0018] Furthermore, in step 3, the specific method of normalization is: downsampling the image blocks larger than the minimum value to the minimum size, so that all image blocks have the same size, and all image blocks are arranged into an image block sequence.

[0019] Furthermore, in step 4, the three stages are independent of each other, the input and output sizes of each stage are the same, and the three stages are trained together.

[0020] Furthermore, in step 4, the number of downsampling decreases as the stage increases, and then features are obtained through multiple convolution operations and activation functions, and the low-size features are upsampled from low to high and fused into high-size features. There is no information loss in the upsampling process, and after a subsequent series of convolution operations, the output is residually connected.

[0021] Furthermore, in step 6, the clean image blocks output by the image denoising network model are extracted in sequence, decoded according to their encoded position information, their positions are restored to the positions on the original ultra-high-resolution image, and their sizes are restored to the sizes when they were cut on the original image, and all image blocks are spliced ​​to obtain the final denoised ultra-high-resolution image.

[0022] The present invention cuts and preprocesses the image, normalizes it, and then uses a convolutional neural network (CNN) to perform noise reduction, ultimately obtaining a denoised image. This allows deep learning methods to be applied to such images, achieving excellent denoising results. The network model can be trained end-to-end, making it highly practical in actual ultra-high-resolution image processing scenarios. Furthermore, the method of the present invention can be used not only for denoising but can even be extended to many computer vision tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a basic flow chart of the method of the present invention;

[0024] Figure 2 Detailed diagram of the image denoising network model constructed by the present invention;

[0025] Figure 3 The effect diagram of the present invention includes the original noise image, the clean image after cutting and the output denoised image;

[0026] Figure 4 The performance of the present invention is compared with other existing works, including peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). DETAILED DESCRIPTION

[0027] In the following description, a number of specific details are provided to clarify the technical means, creative objectives, and effects achieved by the present invention. This invention does not provide a more detailed description of some common knowledge issues in the field. To avoid confusion with other examples, this invention only describes its own technology.

[0028] The purpose of this invention is to solve the problem that existing denoising methods are not applicable to ultra-high resolution images, and propose a denoising method for ultra-high resolution images so that deep learning methods can be applied to such images. The specific method flow is as follows Figure 1 As shown, including:

[0029] Step 1: Build an ultra-high-resolution image cutting and coding model, including quadtree-based multi-level image cutting and position coding;

[0030] Step 2: Input the noisy image and the clean image into the cutting and encoding model respectively to obtain image blocks;

[0031] Step 3: Normalize the image blocks to different degrees to make all image blocks of the same size, construct image block sequences, and obtain training and test data;

[0032] Step 4: Build an image denoising network model, which includes three stages: preliminary denoising, fine denoising, and result fine-tuning. Each stage includes downsampling to enhance denoising performance.

[0033] Step 5: Input the image block sequence into the image denoising network model to obtain clean image blocks;

[0034] Step 6: Decode and concatenate the clean image blocks output by the network to obtain a denoised ultra-high-resolution image.

[0035] This invention is based on deep learning, so before processing, an image dataset must be constructed for training purposes. The present invention uses the publicly available PANDA dataset, a billion-pixel dataset, as clean images. This dataset contains ultra-high-resolution images of a large number of scenes, with each image having a resolution between 100 million and 1 billion. This embodiment adds different levels of normally distributed additive Gaussian noise to the clean images, with noise standard deviations of 10, 30, and 50.

[0036] This embodiment first cuts the image to avoid losing information. A simple method is to directly perform carpet cutting and then input it into the network for processing. However, this method is slow. In addition, this type of image has some special features. For example, most images are taken of large outdoor scenes with a wide viewing angle, such as cities, starry skies, and the earth. The images contain a large number of texture-free areas, and denoising these areas is extremely convenient. However, some areas with complex textures require complex analysis. If these areas of different complexity are carpet cut and the same processing is applied, it will lead to further waste of time, which will be particularly obvious in such a large amount of data. Therefore, this embodiment proposes a quadtree-based multi-level image cutting method, which aims to quickly cut while taking advantage of the complexity differences of different areas of the image itself to apply different degrees of processing to avoid wasting computing resources.

[0037] In step 1, the present invention first treats the noisy image as the root node of a tree and continuously divides it into four parts. Before each division, the node complexity is calculated. Specifically, the node complexity is determined by determining the difference between the maximum and minimum values ​​within the node. A threshold k is set. The range of this value can be adjusted according to the noise level of the image. Experiments have shown that for 8-bit images with noise standard deviations of 10, 30, and 50, setting k to approximately 25, 50, and 75 results in optimal final denoising results. During the division process, the node complexity is continuously determined to be less than the threshold. If it is less than the threshold, the node is considered to belong to a region with simple texture and the division stops. If it is greater than the threshold, the node texture is considered to still be relatively complex and the division continues. This process is repeated until all nodes reach the minimum size specified by the present invention or the stopping condition is reached. A minimum size that is too large will result in suboptimal segmentation results and reduced final denoising performance. A size that is too small will increase the number of image blocks and reduce the perceived block boundaries. Experiments have shown that a minimum size of 128*128 results in optimal final denoising results. In addition, to prevent the resulting image blocks from being too large in images with less texture information, a maximum size of 2048*2048 is added. This means that if the node size exceeds the maximum size when the stopping condition is met, the node is forced to be quartered. During this process, each node is position-encoded, recording its size and the position of its top-left pixel on the original ultra-high-resolution noise image. This facilitates subsequent decoding of the output back to the original image. This process can be expressed mathematically as follows:

[0038]

[0039] Where x is a node, x1, x2, x3, and x4 are its four child nodes, QD(·) represents a quarter-cut, and f(·) is the complexity of judging the node, which is |max(x)-min(x)|.

[0040] In order to obtain one-to-one corresponding image blocks for the noise image and the clean image, this embodiment first performs the above-mentioned cutting step on the noise image. After obtaining all image blocks, image blocks of the same position and size are directly cut out on the clean image based on the position and size information of all nodes, so that all two types of image blocks can achieve one-to-one correspondence.

[0041] For step 3, after obtaining simple and complex image blocks, the simpler the image blocks, the more violent and efficient the noise reduction processing can be applied. Specifically, the complex image blocks are temporarily not processed, and the simple image blocks are downsampled to varying degrees. That is, for image blocks larger than 128*128, they are uniformly reduced to 128*128. The larger the size, the simpler the texture, the greater the downsampling force, and the more direct the noise reduction method. Afterwards, these image blocks are arranged into a sequence, and finally, an image sequence of 128*128 size is obtained. This process is performed on both the noise image and the clean image. This process can be mathematically expressed as follows:

[0042] x′=Down(x)

[0043] Where x is the node of a simple image block, x' is the node after downsampling, and Down(·) is the downsampling process.

[0044] For step 4, in order to obtain an efficient denoising model, this embodiment constructs a progressive denoising convolutional neural network model based on multiple stages and multiple scales, which specifically includes three stages. If too few stages are set, the performance will be reduced, and too many stages will not bring much improvement, but will increase the computational complexity. In the first stage, the input noise map size (b, c, h, w) is (1, 3, 128, 128), where b represents batch_size batch size, c represents channel number, h represents height image block height, and w represents width image block width. It is downsampled multiple times, 3 times in the first stage, 2 times in the second stage, and 1 time in the third stage. Each downsampling will reduce the height and width to the original value. Figure 2 The c dimension is quadrupled, resulting in three downsampled inputs: (1,12,64,64), (1,48,32,32), and (1,192,16,16). These inputs are then convolved to obtain feature maps. The convolution kernel size is 3*3, and the number of channels in the feature map increases as the height and width decrease. After multiple convolution operations, the low-sized feature maps are upsampled using the inverse of the original downsampling process. The c dimension is reduced to 1 / 4 of the original, while the height and width are doubled. They are then fused with the earlier feature maps of the previous size. Specifically, the two feature maps are concatenated in the c dimension before subsequent convolution operations. This process is repeated until all low-sized feature maps are fused to their original size. After a series of convolutions, the output channel c is 3 in the final layer to ensure that the output is the same size as the original input. To ensure the effectiveness of this stage, a residual connection is added from input to output, which is mathematically expressed as:

[0045] x1x0+F(x0)

[0046] Where x0 is the input, x1 is the output of one stage, and F(·) is all operations in one stage.

[0047] The subsequent second and third stages perform similar operations. The more operations are performed at the back end, the more fine-tuning the output is, thus reducing the number of corresponding downsampling. After the convolution operation, ReLU is added as the activation function. The structure of the entire network is as follows Figure 2 shown.

[0048] For step 5, the present invention uses the noisy and clean image sequences as training data, uses the Adam optimizer, uses the MSE loss function, sets the batch_size to 8, performs 300 epochs, and performs 153,600 iterations. The learning rate is 0.0001 in the first 100 epochs, then drops to 0.00001 for the next 100 epochs, and drops to 0.000001 for the final 100 epochs. The GPU used is a single RTX 3090. After training the denoising model, the present invention tests it on the test data, inputting only the noisy image block sequence to obtain the denoised image block sequence.

[0049] In step 6, the present invention stitches together the obtained denoised image blocks one by one. Specifically, the blocks are decoded based on their encoded position information, restored to their positions on the original ultra-high-resolution image, and resized to their original, uncut dimensions using image interpolation and upsampling. Finally, all the blocks are stitched together to obtain the denoised ultra-high-resolution image.

[0050] The method used in the present invention has the following effects on denoising: Figure 3 To illustrate the advantages of the present invention, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) performance of different methods are compared on the Panda dataset. This dataset is a public dataset of billions of pixels. DnCNN is the most classic deep learning denoising method. FFDNet is an earlier proposed denoising method that is superior to DnCNN. CBDNet is an improvement on FFDNet. SGN is a denoising method with good performance. The final results are shown in the figure. Figure 4 .

Claims

1. A deep learning-based ultra-high-resolution image denoising method, characterized in that: The steps include: Step 1: obtain an ultra-high resolution image as a clean image and obtain its noisy image by adding noise; Step 2: performing quadtree cutting and position encoding on the clean image and the noisy image respectively to obtain clean image blocks and noisy image blocks of different sizes; Step 3: normalize the clean image blocks and the noisy image blocks to different degrees to make all image blocks of the same size, construct a clean image block sequence and a noisy image block sequence, and obtain training and test data; Step 4: Construct an image denoising network model, which includes three stages: preliminary denoising, fine denoising, and result fine-tuning. Each stage includes downsampling to enhance denoising performance. The three stages are independent of each other, with the same input and output sizes. The three stages are trained together. The number of downsampling decreases as the stage increases. After that, features are obtained through multiple convolution operations and activation functions. Low-size features are upsampled and fused into high-size features in order from low to high. No information is lost in the upsampling process. After a series of subsequent convolution operations, the output is residually connected. Step 5: input the noisy image block sequence into the image denoising network model to obtain clean image blocks; In step 6, the clean image blocks obtained in step 5 are decoded and spliced ​​to obtain a denoised ultra-high resolution image.

2. The method for ultra-high resolution image denoising based on deep learning according to claim 1, characterized in that: When performing quadtree cutting in step 2, the noise image is first regarded as the root node of a tree, and the absolute value of the difference between the maximum and minimum values ​​of all pixel values ​​is determined to be less than a threshold. If it is less than a threshold, the current node is considered to belong to a region with simple texture and the quadrisection is stopped. If it is greater than a threshold, the texture of the current node is considered to be still relatively complex and the quadrisection is continued. The above process is repeated until all nodes meet the preset conditions or the stop condition.

3. The method for ultra-high resolution image denoising based on deep learning according to claim 2, characterized in that: The preset condition includes that the size of the current node reaches a minimum value of 128*128.

4. The method for ultra-high resolution image denoising based on deep learning according to claim 3, characterized in that: The preset condition also includes a maximum size of the current node of 2048*2048. If the size of the current node is greater than the maximum size when the current node meets the stop condition, the node is forcibly divided into four parts.

5. The method for ultra-high resolution image denoising based on deep learning according to claim 1, characterized in that: In step 2, the position encoding takes the position of the first pixel in the upper left corner of the node on the original ultra-high resolution image, and the size of the node is calculated at the same time. The position encoding on the noise map is performed simultaneously with the quadtree cutting.

6. The method for ultra-high resolution image denoising based on deep learning according to claim 1, characterized in that: In step 2, the noise image is first subjected to quadtree cutting and position coding to obtain all image blocks, and then the image blocks at corresponding positions on the clean image are found according to the position coding, and the blocks are cut and made to correspond one-to-one with the image blocks of the noise image.

7. The method for ultra-high resolution image denoising based on deep learning according to claim 1, characterized in that: In step 3, the specific method of normalization is: down-sampling the image blocks larger than the minimum value to the minimum size, so that all image blocks have the same size, and all image blocks are arranged into an image block sequence.

8. The method for ultra-high resolution image denoising based on deep learning according to claim 1, characterized in that: In step 6, the clean image blocks output by the image denoising network model are extracted in sequence, decoded according to their encoded position information, their positions are restored to the positions on the original ultra-high resolution image, and their sizes are restored to the sizes when they were cut on the original image. All image blocks are spliced ​​together to obtain the final denoised ultra-high resolution image.

Images