Ultrahigh-resolution image denoising method based on pixel recombination sampling partitioning

Abstract

The invention discloses an ultrahigh-resolution image denoising method based on pixel recombination sampling and partitioning. The method comprises the following specific steps: carrying out pixel recombination down-sampling partitioning on paired noisy and noiseless ultrahigh-resolution images to generate a data set; constructing a deep learning network which uses the image blocks and the image blocks with adjacent pixels and is composed of two denoising networks and a fusion network; jointly training the whole network based on three loss functions which respectively constrain the two denoising networks and the fusion network; when reasoning is carried out, pixel recombination down-sampling blocking is carried out on a noisy ultrahigh-resolution image, image blocks and image blocks adjacent to pixels of the image blocks are sent into a network to be denoised, and pixel recombination up-sampling splicing is carried out, so that a total image denoising result of the ultrahigh-resolution image is obtained. According to the method, the local information and the non-local information of the ultrahigh-resolution image are fully utilized, and the defects that the non-local information is difficult to utilize during block denoising and the detail textures of the image are damaged by traditional downsampling are overcome.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and deep learning, and in particular relates to an ultra-high resolution image denoising method based on pixel reorganization sampling and blocking. Background Art

[0002] The advent of the digital imaging era and the continuous evolution of CMOS image sensors have led to an exponential growth in the amount of data to be processed in the field of computer vision. However, existing neural networks are designed to perform well on relatively small data sets, usually at the expense of increased computing power. However, when faced with the challenge of processing large-scale data such as ultra-high resolution (UHR) images, many networks become unsuitable due to the sharp increase in computing requirements. Therefore, a major challenge in the field of computer vision is how to effectively handle larger image sizes within the constraints of limited computing resources, while addressing the difficulties that networks generally encounter when processing UHR image data.

[0003] In the field of ultra-high resolution image denoising, deep learning-based image denoising algorithms have made significant progress in recent years. However, none of them is suitable for solving the problem of ultra-high resolution image denoising due to the huge computational requirements involved. Therefore, a practical approach has emerged: split the ultra-high resolution image into small blocks and then input them into the traditional denoising network. Although this approach alleviates the computational constraints, it hinders the utilization of long-range information and makes the performance of the denoising network more susceptible to pixel neighbor noise.

[0004] Meanwhile, conventional non-local denoising algorithms have demonstrated their potential to improve performance. In particular, for ultra-high resolution images that contain a large amount of redundant information in a single image, utilizing spatially long-range non-local correlation information can significantly enhance the denoising effect. However, since the segments are independent of each other, the block-based approach makes it impossible to directly utilize non-local correlation information, making it complicated to fully utilize non-local correlation for ultra-high resolution image denoising. Therefore, how to balance the computational requirements and restoration quality in the ultra-high resolution image restoration problem, and how to compensate for the performance degradation caused by the image block-based approach, are currently unresolved problems. Summary of the invention

[0005] In view of the above analysis of the prior art and the existing problems, the purpose of the present invention is to propose an ultra-high resolution image denoising method based on pixel re-sampling and blocking.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] The ultra-high resolution image denoising method based on pixel reorganization sampling block includes the following steps:

[0008] Step 1: add noise to several ultra-high resolution images, and then perform pixel reorganization and downsampling on paired images with and without noise to obtain several paired pixel-adjacent image blocks of ultra-high resolution images to generate a data set;

[0009] Step 2, constructing a dual-input and single-output deep learning network consisting of two denoising networks and a fusion network, and sending the noisy image block obtained in step 1 and the image blocks with adjacent pixels around it to the two denoising networks respectively;

[0010] Step 3, taking the deep learning network constructed in step 2 as a whole, based on the data set generated in step 1, respectively calculating the loss of the outputs of the two denoising networks and the output of the fusion network, and using the weighted sum of the three loss functions to train the deep learning network as a whole;

[0011] Step 4, in the inference link of ultra-high resolution image denoising, a noisy ultra-high resolution image is pixel-reorganized and downsampled as described in step 1 to obtain a number of pixel-adjacent image blocks, and each image block and its surrounding pixel-adjacent image blocks are sent to the deep learning network trained in step 3, so that each image block can be denoised with the assistance of its own pixel-adjacent image blocks, and finally all denoised image blocks are pixel-reorganized and upsampled and spliced ​​to obtain the full-image denoising result of the ultra-high resolution image.

[0012] In view of the computational complexity of the denoising network, the existing block processing method has to be used for ultra-high resolution images. However, the present invention is different from the traditional block processing. By reorganizing and sampling the pixels of the ultra-high resolution image, the pixels that were originally geometrically far away are aggregated in the same image block to fully utilize the non-local information of the ultra-high resolution image and overcome the impact of pixel-related noise on the performance of the denoising network. At the same time, in order to ensure the utilization of local information and the retention of image detail texture, the present invention uses pixel adjacent image blocks to assist in denoising, and uses the differential loss between image blocks and adjacent blocks to assist in training the network. Compared with the existing technology, the present invention makes full use of the local information and non-local information of ultra-high resolution images, making up for the shortcomings that it is difficult to utilize non-local information in block denoising and traditional downsampling will destroy the image detail texture. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the process of the present invention;

[0014] Figure 2 Schematic diagram of a pixel reorganization downsampling block and pixel reorganization upsampling splicing method according to an embodiment of the present invention;

[0015] Figure 3The overall structure diagram of the deep learning network composed of two denoising networks and one fusion network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The method of the present invention is described in detail below, with detailed implementation methods and specific operation processes provided, but the protection scope of the present invention is not limited to the following implementations.

[0017] In order to solve the problem that the data volume is too large when processing the whole image and the long-distance information cannot be used when processing the image directly by blocks, this embodiment provides a method for denoising an ultra-high resolution image based on pixel reorganization sampling blocks. Figure 1 , Figure 2 and Figure 3 As shown, the following steps are included:

[0018] Step 1: First, add additive Gaussian white noise to the ultra-high resolution image dataset, with a mean of 0 and variances of 20, 30, and 40, respectively. Then, perform pixel reorganization and downsampling of the same magnification on the image pairs before and after adding noise: split an ultra-high resolution image I with a height of H and a width of W into k×k image blocks I m,n , 0≤m,n≤k and represents the integer domain, kk is the pixel reorganization downsampling block multiplier, and each pixel in the image block is:

[0019]

[0020] in, is the image block I m,n The pixel at position (x,y) in I k×x+m，k×y+n is the pixel at position (k×x+m, k×y+n) in image I, The ultra-high resolution image I is pixel-recombined into k×k image blocks as its own downsampling, and each pixel of the image I will only be assigned to one of the image blocks. The distance between adjacent pixels in each image block in the original image I is k, which is the same as the pixel-recombined downsampling block magnification k. This step is to process several ultra-high resolution image pairs with and without noise to obtain several pixel-recombined downsampling blocks as the data set for the subsequent steps. In the image block pair, the noise-free image block is used as the true value of the noisy image block. The denoising network is trained by calculating the loss function between the network output and the true value and back-propagating in a common way in deep learning.

[0021] Step 2: Build a deep learning network with two inputs and one output, which consists of two denoising networks and one fusion network. Figure 3As shown, the noisy image block obtained in step 1 and the image blocks with adjacent pixels around it are respectively sent to two denoising networks. Among them, the image blocks with adjacent pixels are the ones with respect to the original image block I m,n I m,n±1 ,I m±1,n ,I m±1,n±1 There are 8 image blocks in total, each of which is identical to the original image block I m,n The pixels at the same position in the original super-high resolution image I are adjacent pixels, so we call it I m,n±1 ,I m±1,n ,I m±1,n±1 are pixel adjacent image blocks of image I, and these pixel adjacent image blocks are image block I m,n Add an image block with a 1-pixel offset on the horizontal and vertical coordinates of the super-high resolution image I, so that its content is similar. In particular, when m,n∈{0,k}, that is, the original image block I m,n When there are no pixel adjacent image blocks in one or two directions, I m,n The pixel adjacent image blocks are used as the local similarity information of the original image blocks in steps 3 and 4, and the original image blocks I are assisted by another denoising network and fusion network. m,n Denoising.

[0022] Among them, the two denoising networks Denoise1 and Denoise2 process two inputs respectively. The first input is the noisy image block I generated in step 1. m,n , the second input is the noisy image block I m,n The 8-pixel adjacent image block I m,n±1 ,I m±1,n ,I m±1,n±1 Then, the denoising results of the two networks are spliced ​​along the channel dimension and sent to the fusion network Fusion for fusion. The formula is as follows:

[0023] I' m,n =Denoise1(I m,n )

[0024] I pixel_similar

[0025] =concat(I m,n+1 ,I m,n-1 ,I m+1,n , I m-1,n , I m+1,n61 , I m+1,n-1 , I m-1,n+1 , I m-1,n-1 )

[0026] I' pixel_similar =Denoise2(Ipixel_similar )

[0027] Among them, I' m,n For the denoising network Denoise1, the image block I m,n The denoising result. concat() is the concatenation of image blocks along the channel dimension. pixel_similar It is the local similarity information obtained by splicing 8 adjacent pixel image blocks along the channel dimension, and the number of channels is 24. pixel_similar For the denoising network Denoise2 pixel_similar The denoising result, the number of channels is also 24, this output result is the joint denoising result of each pixel adjacent image block I' m±1,n , I' m,n±1 , I' m±1,n±1 Concatenate across the channel dimension. Fusion for I' m,n and I' pixel_similar The processed results are concatenated and fused along the channel dimension. As the output of the entire network, it is the image block I m,n Final denoising result with the assistance of 8-pixel neighboring image patches.

[0028] Step 3: Take the two denoising networks and one fusion network constructed in step 2 as a whole, and calculate the loss of the outputs of the two denoising networks and the output of the network as a whole based on the training data set generated in step 1. Use the weighted sum of three loss functions to train the network. The formulas of these three loss functions are:

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] Loss total =α×Loss1+β×Loss2+γ×Loss3

[0035] in, and I m,n and I pixel_similar Corresponding to the image block without noise, and There are 8 and I'm,n The concatenation is performed along the channel dimension to facilitate the subsequent calculation of the score of the image blocks adjacent to each pixel. ||||2 is the two-norm loss, and α, β, and γ are the constant weights of the three loss functions. Loss1 calculates the output I' of the first denoising network m,n with I m,n The corresponding truth value The loss is used to supervise the training of the first denoising network. Loss2 calculates the output I' of the first denoising network m,n The output of the second denoising network I' pixel_similar The difference value of each pixel in the adjacent image block and the noise-free image block and The loss between the difference values ​​of each noise-free pixel adjacent to the image block is used to supervise the second denoising network, so that the pixel adjacent image blocks can provide sufficient edge and texture details to provide local information of the image for the denoising of the original image block in the subsequent fusion network. Loss3 calculates the final denoising result output by the fusion network With noise-free image patches The loss is used to supervise the fusion network. The total loss Loss obtained by weighting the three loss functions is total train.

[0036] Step 4, in the inference link of ultra-high resolution image denoising, a noisy ultra-high resolution image is pixel-reorganized and downsampled as described in step 1 to obtain a number of pixel-adjacent image blocks, and each image block and its surrounding pixel-adjacent image blocks are sent to the two denoising networks and fusion network trained in step 3, so that each image block can be denoised with the assistance of its own pixel-adjacent image blocks. Finally, all denoised image blocks are pixel-reorganized, upsampled and spliced ​​back to an ultra-high resolution image, which is the full-image denoising result.

[0037] The pixel reorganization upsampling splicing of all denoised image blocks is the inverse process of the pixel reorganization downsampling block in step 1: k×k denoised image blocks are Stitched into a denoised ultra-high resolution image with a height of H and a width of W 0≤m,n≤k and , k is the pixel reorganization upsampling splicing ratio, which is the same as the pixel reorganization downsampling block ratio in step 1, where the image Each pixel of is:

[0038]

[0039] in, For images The pixel at position (k×x+m, k×y+n) in For image blocks The pixel at position (x,y) in

[0040] Based on the above description, the present invention can aggregate pixels that are originally far apart in geometric distance into the same image block by pixel reorganization downsampling block and pixel reorganization upsampling splicing of ultra-high resolution images, so as to make full use of the non-local information of ultra-high resolution images and overcome the influence of pixel-related noise on the performance of denoising networks. At the same time, in order to ensure the utilization of local information and the retention of image detail texture, the present invention utilizes pixel-adjacent image blocks to assist in denoising, and utilizes the differential loss between image blocks and adjacent blocks to assist in training the network. Compared with the prior art, the present invention makes full use of the local information and non-local information of ultra-high resolution images, making up for the shortcomings that it is difficult to utilize non-local information in block denoising and that traditional downsampling will destroy image detail texture. Thus, a solution is proposed to balance the computational requirements and restoration quality in the problem of ultra-high resolution image restoration, and to compensate for the performance degradation caused by the image block method.

Claims

1. An ultra-high resolution image denoising method based on pixel reorganization sampling and blocking, characterized in that: The following steps are involved: Step 1: add noise to several ultra-high resolution images, and then perform pixel reorganization and downsampling on paired images with and without noise to obtain several paired pixel-adjacent image blocks of ultra-high resolution images to generate a data set; Step 2, constructing a dual-input and single-output deep learning network consisting of two denoising networks and a fusion network, and sending the noisy image block obtained in step 1 and the image blocks with adjacent pixels around it to the two denoising networks respectively; Step 3, taking the deep learning network constructed in step 2 as a whole, based on the data set generated in step 1, respectively calculating the loss of the outputs of the two denoising networks and the output of the fusion network, and using the weighted sum of the three loss functions to train the deep learning network as a whole; Step 4, in the inference link of ultra-high resolution image denoising, a noisy ultra-high resolution image is pixel-reorganized and downsampled as described in step 1 to obtain a number of pixel-adjacent image blocks, and each image block and its surrounding pixel-adjacent image blocks are sent to the deep learning network trained in step 3, so that each image block can be denoised with the assistance of its own pixel-adjacent image blocks, and finally all denoised image blocks are pixel-reorganized and upsampled and spliced ​​to obtain the full-image denoising result of the ultra-high resolution image.

2. The ultra-high resolution image denoising method based on pixel reorganization sampling and blocking according to claim 1 is characterized in that: In step 1, the specific steps of pixel reorganization and downsampling are: divide an ultra-high resolution image I with a height of H and a width of W into k×k image blocks I m,n , 0≤m,n≤k and represents the integer domain, k is the pixel reorganization downsampling block multiplier, and each pixel in the image block is: in, is the image block I m,n The pixel at position (x,y) in I k×m+x，k×n+y is the pixel at position (k×x+m, k×y+n) in image I, The ultra-high resolution image I is pixel-wise reorganized into k×k image blocks as its own downsampling, and each pixel of the image I will only be assigned to one of the image blocks, and the distance between adjacent pixels in each image block in the original image I is k.

3. The ultra-high resolution image denoising method based on pixel reorganization sampling and blocking according to claim 1 is characterized in that: In step 2, the image block with adjacent pixels on all sides is image block I. m,n Add an image block with an offset of 1 pixel to the horizontal and vertical coordinates of the super-high resolution image I; When the image block I m,n When there are no pixel adjacent image blocks in one or two directions, I m,n The missing pixel neighboring image blocks are replaced by themselves for subsequent processing.

4. The ultra-high resolution image denoising method based on pixel reorganization sampling and blocking according to claim 3 is characterized in that: In step 2, the image block I m,n Image block I adjacent to 8 pixels m,n±1 ,I m±1,n ,I m±1,n±1 The two denoising networks Denoise1 and Denoise2 are respectively sent to each other, and the output results of the two denoising networks are fused through the fusion network Fusion. The formula is: I’ m,n =Denoise1(I m,n ) I pixel_similar =concat(I m,n+1 ,I m,n-1 ,I m+1,n ,I m-1,n ,I m+1,n+1 ,I m+1,n-1 ,I m-1,n+1 ,I m-1,n-1 ) I’ pixel_similar =Denoise2(I pixel_similar ) Among them, I' m,n For the denoising network Denoise1, the image block I m,n The denoising result; concat() is the concatenation of image blocks along the channel dimension, I pixel_similar is the local similarity information obtained by splicing 8 adjacent pixel image blocks along the channel dimension, I' pixel_similar For the denoising network Denoise2 pixel_similar The denoising result is Fusion for I' m,n and I' pixel_similar Concatenate and fuse the processed results along the channel dimension.

5. The ultra-high resolution image denoising method based on pixel reorganization sampling and blocking according to claim 4 is characterized in that: In step 3, the losses are calculated for the outputs of the two denoising networks and the output of the fusion network respectively, and the weighted sum of the three loss functions is used to train the deep learning network as a whole. The specific implementation formula is: Loss total =α×Loss1+β×Loss2+γ×Loss3 in, and I m,n and I pixel_similar Corresponding to the image block without noise, and There are 8 and I' m,n Splicing is performed along the channel dimension to facilitate the subsequent calculation of the score of the image blocks adjacent to each pixel; || ||2 is the two-norm loss, α, β, and γ are the constant weights of the three loss functions; the loss function Loss1 calculates the output I' of the first denoising network m,n with I m,n The corresponding truth value The loss is used to supervise the training of the first denoising network; the loss function Loss2 calculates the output I' of the first denoising network m,n The output of the second denoising network I' pixel_similar The difference value of each pixel in the adjacent image block and the noise-free image block and The loss between the difference values ​​of adjacent image blocks of each noise-free pixel in is used to supervise the second denoising network; the loss function Loss3 calculates the final denoising result output by the fusion network With noise-free image patches The loss is used to supervise the fusion network.

6. The method for ultra-high resolution image denoising based on pixel reorganization and sampling block according to claim 1, characterized in that: In step 4, the specific implementation steps of pixel reorganization upsampling splicing of all denoised image blocks are: k×k denoised image blocks Stitched into a denoised ultra-high resolution image with a height of H and a width of W 0≤m,n≤k and represents the integer domain, k is the pixel reorganization upsampling splicing ratio, where the image Each pixel of is: in, For images The pixel at position (k×x+m, k×y+n) in For image blocks The pixel at position (x,y) in

Images