Super high resolution image denoising method based on pixel recombination sampling block

Abstract

The application discloses a kind of based on pixel reorganization sampling block's super high resolution image denoising method.The specific steps are as follows: the pixel reorganization undersampling block of paired noisy and noiseless super high resolution image is carried out to generate data set;Depth learning network constituted by two denoising networks and a fusion network is constructed using image block and its pixel adjacent image block;Based on three loss functions respectively constraining two denoising networks and a fusion network, the whole network is trained jointly;Inference, the pixel reorganization undersampling block of noisy super high resolution image is carried out, image block and its pixel adjacent image block are sent into network denoising, pixel reorganization upsampling splicing, to obtain the full image denoising result of super high resolution image.This method makes full use of the local information and non-local information of super high resolution image, makes up the shortcoming that non-local information is difficult to use when block denoising, and traditional undersampling will destroy image details texture.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing and deep learning, and particularly relates to a super high resolution image denoising method based on pixel reorganization sampling block. BACKGROUND

[0002] The advent of the digital imaging era and the continuous evolution of CMOS image sensors have led to an exponential increase in the amount of data to be processed in the field of computer vision. However, existing neural networks are designed to perform well on relatively small data sets, often at the cost of increased computational power. However, when faced with the challenge of processing large-scale data such as super high resolution (UHR) images, many networks become unsuitable due to the dramatic increase in computational demand. Therefore, a major challenge in the field of computer vision is how to effectively process larger image sizes within the constraints of limited computational resources, while addressing the difficulties commonly encountered by networks in processing super high resolution image data.

[0003] In the field of super high resolution image denoising, deep learning-based image denoising algorithms have made significant progress in recent years. However, due to the enormous computational demand involved, none of them are suitable for solving the problem of super high resolution image denoising. Therefore, a practical method has emerged: dividing the super high resolution image into small blocks and then inputting them into traditional denoising networks. Although this method alleviates the computational constraints, it hinders the use of long-distance information, making the performance of the denoising network more susceptible to pixel adjacent noise.

[0004] At the same time, traditional non-local denoising algorithms have proven their potential to improve performance. In particular, for super high resolution images containing a large amount of redundant information in a single image, utilizing non-local related information over long distances in space can significantly enhance the denoising effect. However, since the blocks are independent of each other, the block-based method makes it impossible to directly utilize non-local related information, making it complex to fully utilize non-local correlation for super high resolution image denoising. Therefore, how to balance the computational demand and recovery quality in the problem of super high resolution image restoration, and how to compensate for the performance decline caused by image block method, are difficult problems to be solved. SUMMARY

[0005] In view of the above analysis of the prior art and existing problems, the purpose of the present application is to propose a super high resolution image denoising method based on pixel reorganization sampling block.

[0006] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0007] The super high resolution image denoising method based on pixel reorganization sampling block includes the following steps:

[0008] Step 1, add noise to several super high resolution images, then perform pixel reorganization down-sampling and block division on the pairs of noisy and non-noisy images to obtain several pairs of pixel-adjacent image blocks of super high resolution images to generate a data set;

[0009] Step 2, construct a deep learning network composed of two denoising networks and a fusion network, and input the noisy image blocks and their pixel-adjacent image blocks obtained in step 1 into the two denoising networks respectively;

[0010] Step 3, take the deep learning network constructed in step 2 as a whole, calculate the loss of the outputs of the two denoising networks and the output of the fusion network based on the data set generated in step 1, and train the deep learning network as a whole by using the weighted sum of the three loss functions;

[0011] Step 4, in the inference link of super high resolution image denoising, perform pixel reorganization down-sampling and block division on a noisy super high resolution image to obtain several pixel-adjacent image blocks, input each image block and its pixel-adjacent image blocks into the deep learning network trained in step 3, so that each image block is denoised with the assistance of its pixel-adjacent image blocks, and finally perform pixel reorganization up-sampling and splicing on all denoised image blocks to obtain the full image denoising result of the super high resolution image.

[0012] In view of the consideration of the calculation complexity of the denoising network, the existing method for super high resolution images has to use the block processing method, but the present application is different from the traditional block processing. By performing pixel reorganization sampling on the super high resolution image, the pixels originally far apart in geometric distance are aggregated in the same image block, so as to fully utilize the non-local information of the super high resolution image and overcome the influence of pixel-related noise on the performance of the denoising network. At the same time, in order to ensure the utilization of local information and the preservation of image details and textures, the present application utilizes the pixel-adjacent image blocks to assist in denoising, and utilizes the difference loss between the image block and the adjacent block to assist in training the network. Compared with the prior art, the present application fully utilizes the local information and non-local information of the super high resolution image, and makes up for the shortcomings that it is difficult to utilize non-local information when denoising by block and the traditional down-sampling will damage the image details and textures. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 It is a flowchart of the method of the present application;

[0014] Figure 2 It is a pixel reorganization down-sampling and block division and pixel reorganization up-sampling and splicing method diagram of the embodiment of the present application;

[0015] Figure 3A deep learning network overall structure diagram of two denoising networks and a fusion network of embodiments of the present application. DETAILED DESCRIPTION

[0016] The method of the present application is described in detail below, and detailed embodiments and specific operation processes are given, but the scope of protection of the present application is not limited to the following embodiments.

[0017] For the problem of too large data volume in full image processing and inability to use long-distance information in direct block processing in super high resolution image denoising, the present embodiment provides a super high resolution image denoising method based on pixel reorganization sampling block, as shown in Figure 1 、 Figure 2 and Figure 3 , comprising the following steps:

[0018] Step 1, first add additive Gaussian white noise to the super high resolution image data set, with a mean of 0 and a variance of 20, 30 and 40 respectively. Then, the image pairs before and after adding noise are simultaneously pixel reorganization down-sampling block with the same magnification: a super high resolution image I of height H and width W is divided into k×k image blocks I m,n , 0≤m,n≤k and represent the integer domain, kkk is the magnification of pixel reorganization down-sampling block, and each pixel in the image block is:

[0019]

[0020] wherein, is the pixel at position (x, y) in the image block I m,n , I k×x+m，k×y+n is the pixel at position (k×x+m, k×y+n) in the image I, The super high resolution image I is pixel reorganized into k×k image blocks as its own down-sampling, and each pixel of the image I is only divided into one of the image blocks, and the adjacent pixels in each image block are at a distance of k in the original image I, that is, the same as the magnification k of the pixel reorganization down-sampling block. This step obtains a plurality of image block pairs through pixel reorganization down-sampling by processing a plurality of noisy and non-noisy super high resolution image pairs, which are used as the data set for the subsequent steps. In the image block pair, the non-noisy image block is used as the true value of the noisy image block, and the denoising network is trained by the common method of calculating the loss function between the network output and the true value in deep learning and back propagation.

[0021] Step 2, a deep learning network of double-in and single-out is constructed by two denoising networks and one fusion network, and the structure is as shown in Figure 3As shown, the noisy image block obtained in step 1 and its surrounding pixel-adjacent image blocks are respectively fed into two denoising networks. Among them, the pixel-adjacent image blocks are the pixels of the original image block I. m,n I m,n±1 , I m±1,n , I m±1,n±1 There are 8 image blocks in total, each of which is the same as the original image block I m,n The pixels at the same position in the original super-high resolution image I are adjacent pixels, so we call it I m,n±1 , I m±1,n , I m±1,n±1 are pixel-adjacent image blocks of image I, and these pixel-adjacent image blocks are image blocks I m,n Add an image block with a 1-pixel offset on the horizontal and vertical coordinates of the super-high-resolution image I, so that its content is similar. In particular, when m,n∈{0,k}, that is, the original image block I m,n When there is no pixel adjacent image block in one or two directions, I m,n The pixel adjacent image blocks are used as the local similarity information of the original image blocks in steps 3 and 4, and the original image blocks are assisted by another denoising network and fusion network. m,n denoising.

[0022] Among them, the two denoising networks Denoise1 and Denoise2 process two inputs respectively. The first input is the noisy image block I generated in step 1. m,n , the second input is the noisy image block I m,n 8 pixel adjacent image blocks I m,n±1 , I m±1,n , I m±1,n±1 , then the denoising results of the two networks are spliced ​​along the channel dimension and sent to the fusion network Fusion for fusion. The formula is as follows:

[0023] I' m,n =Denoise1(I m,n )

[0024] I pixel_similar

[0025] =concat(I m,n+1 ,I m,n-1 ,I m+1,n , I m-1,n , I m+1,n61 , I m+1,n-1 , I m-1,n+1 , I m-1,n-1 )

[0026] I' pixel_similar =Denoise2(Ipixel_similar )

[0027] Among them, I' m,n For the denoising network Denoise1, the image block I m,n The denoising result. concat() is the concatenation of image blocks along the channel dimension. pixel_similar It is the local similarity information obtained by splicing 8 adjacent pixel blocks along the channel dimension, and the number of channels is 24. pixel_similar For the denoising network Denoise2 pixel_similar The denoising result, the number of channels is also 24, this output result is the joint denoising result of each pixel adjacent image block I' m±1,n 、I' m,n±1 、I' m±1,n±1 Concatenate across the channel dimension. Fusion for I' m,n and I' pixel_similar The processed results are concatenated and fused along the channel dimension. As the output of the entire network, it is the image block I m,n Final denoising result with the assistance of 8-pixel neighboring image patches.

[0028] In step 3, the two denoising networks and the fusion network constructed in step 2 are taken as a whole. Based on the training data set generated in step 1, the loss is calculated for the output of the two denoising networks and the output of the network as a whole. The network is trained using the weighted sum of the three loss functions. The formulas of these three loss functions are:

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] Loss total =α×Loss1+β×Loss2+γ×Loss3

[0035] in, and I m,n and I pixel_similar Corresponding to the image block without noise, and There are 8 and I' m,n are concatenated along the channel dimension to facilitate the subsequent calculation of the loss value with each pixel-adjacent image patch of the original image. |||2 is the L2 norm loss, and a, b, g are the constant weights of the three loss functions. Loss1 calculates the loss of the output I' m,n and the output I' m,n of the first denoising network. Loss2 calculates the loss of the output I' m,n and the output I' pixel_similar of the second denoising network. Loss3 calculates the loss of the final denoising result I' and the noise-free image patch of the fusion network. Loss4 calculates the loss of the final denoising result I' and the noise-free image patch of the fusion network. Loss is the total loss of the network obtained by weighting the three loss functions. total .

[0036] Step 4: In the inference stage of super-high resolution image denoising, a noisy super-high resolution image is pixel-reorganized and down-sampled and divided into image patches as described in step 1, and each image patch and its four surrounding pixel-adjacent image patches are input into the two denoising networks and the fusion network trained in step 3, so that each image patch is denoised with the assistance of its pixel-adjacent image patches. Finally, all the denoised image patches are pixel-reorganized and up-sampled and spliced to obtain a super-high resolution denoised image, which is the final denoised result of the whole image.

[0037] wherein the pixel-reorganization and up-sampling and splicing of all the denoised image patches is the inverse process of the pixel-reorganization and down-sampling and division in step 1: the k x k denoised image patch is spliced into a denoised super-high resolution image with a height of H and a width of W 0≤m,n≤k and , k is the pixel-reorganization and up-sampling and splicing rate, which is the same as the pixel-reorganization and down-sampling and division rate in step 1, wherein each pixel of the image is:

[0038]

[0039] wherein, is the pixel at the position (k x x + m, k x y + n) in the image ,​ for the image block a pixel with a position (x, y) in the image block,

[0040] Based on the above description, the present application can be used by reorganizing the super high resolution image pixel down-sampling block and pixel reorganization up-sampling splicing, aggregating pixels originally far apart in geometric distance in the same image block, to make full use of the non-local information of the super high resolution image, and overcome the influence of pixel related noise on the performance of the denoising network. At the same time, in order to ensure the use of local information and the preservation of image detail texture, the present application uses the adjacent image block of the pixel to assist in denoising, and uses the difference loss of the image block and the adjacent block to assist in training the network. Compared with the prior art, the present application makes full use of the local information and non-local information of the super high resolution image, makes up for the shortcomings that it is difficult to use non-local information when denoising by block, and the traditional down-sampling will destroy the image detail texture. Thus, a solution is proposed to balance the calculation demand and the recovery quality in the super high resolution image recovery problem, and to make up for the performance decline caused by the image block method.

Claims

1. An ultra-high resolution image denoising method based on pixel re-sampling and blocking, characterized in that: The following steps are involved: Step 1: Noise several ultra-high-resolution images, and then simultaneously perform pixel reconstructing and downsampling on the paired noisy and noise-free images to obtain several pairs of pixel-adjacent image blocks of the ultra-high-resolution images to generate a dataset; Step 2: Construct a deep learning network with two inputs and one output, consisting of two denoising networks and a fusion network. The noisy image block obtained in step 1 and its surrounding pixel-adjacent image blocks are fed into the two denoising networks respectively. Step 3: Taking the deep learning network constructed in step 2 as a whole, based on the data set generated in step 1, respectively calculate the loss of the outputs of the two denoising networks and the output of the fusion network, and use the weighted sum of these three loss functions to train the deep learning network as a whole; Step 4: In the inference phase of ultra-high-resolution image denoising, a noisy ultra-high-resolution image is pixel-reassembled and downsampled as described in step 1 to obtain several pixel-adjacent image blocks. Each image block and its surrounding pixel-adjacent image blocks are fed into the deep learning network trained in step 3, so that each image block is denoised with the assistance of its pixel-adjacent image blocks. Finally, all denoised image blocks are pixel-reassembled and upsampled to obtain the full-image denoising result of the ultra-high-resolution image.

2. The ultra-high resolution image denoising method based on pixel re-sampling and blocking according to claim 1 is characterized in that: In step 1, the specific steps of pixel reorganization and downsampling are as follows: a super-high-resolution image I with a height of H and a width of W is divided into k×k image blocks I m,n , 0≤m,n≤k and represents the integer domain, k is the pixel reorganization downsampling block magnification, and each pixel in the image block is: in, is the image block I m,n The pixel at position (x,y) in I k×m+x，k×n+y is the pixel at position (k×x+m, k×y+n) in image I, The ultra-high-resolution image I is pixel-wise reorganized into k×k image blocks as its own downsampling, and each pixel of the image I will only be assigned to one of the image blocks, and the distance between adjacent pixels in each image block in the original image I is k.

3. The ultra-high resolution image denoising method based on pixel re-sampling and blocking according to claim 1 is characterized in that: In step 2, the image block with adjacent pixels on all sides is image block I. m,n Add an image block with an offset of 1 pixel to the horizontal and vertical coordinates of the super-high resolution image I; When the image block I m,n When there is no pixel adjacent image block in one or two directions, I m,n The missing pixel-neighboring image blocks are replaced by themselves for subsequent processing.

4. The ultra-high resolution image denoising method based on pixel re-sampling and blocking according to claim 3 is characterized in that: In step 2, the image block I m,n Image block I adjacent to 8 pixels m,n±1 , I m±1,n , I m±1,n±1 The two denoising networks Denoise1 and Denoise2 are respectively fed into the network, and the output results of the two denoising networks are fused through the fusion network Fusion. The formula is: I’ m,n =Denoise1(I m,n ) I pixel_similar =concat(I m,n+1 ,I m,n-1 ,I m+1,n ,I m-1,n ,I m+1,n+1 ,I m+1,n-1 ,I m-1,n+1 ,I m-1,n-1 ) I’ pixel_similar =Denoise2(I pixel_similar ) Among them, I' m,n For the denoising network Denoise1, the image block I m,n The denoising result; concat() is the splicing of image blocks along the channel dimension, I pixel_similar is the local similarity information obtained by splicing 8 adjacent pixel blocks along the channel dimension, I' pixel_similar For the denoising network Denoise2 pixel_similar The denoising result is Fusion for I' m,n and I' pixel_similar Concatenate and fuse the processed results along the channel dimension.

5. The ultra-high resolution image denoising method based on pixel re-sampling and blocking according to claim 4 is characterized in that: In step 3, the losses are calculated for the outputs of the two denoising networks and the output of the fusion network respectively, and the weighted sum of these three loss functions is used to train the deep learning network as a whole. The specific implementation formula is: Loss total =α×Loss1+β×Loss2+γ×Loss3 in, and I m,n and I pixel_similar Corresponding to the image block without noise, and There are 8 and I' m,n Splicing is performed along the channel dimension to facilitate the subsequent calculation of the score of the image blocks adjacent to each pixel; || ||2 is the two-norm loss, α, β, and γ are the constant weights of the three loss functions respectively; the loss function Loss1 calculates the output I' of the first denoising network m,n with I m,n The corresponding truth value The loss is used to supervise the training of the first denoising network; the loss function Loss2 calculates the output I' of the first denoising network m,n and the output of the second denoising network I' pixel_similar The difference value of each pixel in the adjacent image block and the noise-free image block and The loss between the difference values ​​of adjacent image blocks of each noise-free pixel in is used to supervise the second denoising network; the loss function Loss3 calculates the final denoising result output by the fusion network With noise-free image patches The loss is used to supervise the fusion network.

6. The ultra-high resolution image denoising method based on pixel re-sampling and blocking according to claim 1, characterized in that: In step 4, the specific implementation steps of pixel reorganization and upsampling of all denoised image blocks are as follows: k×k denoised image blocks Stitched into a denoised ultra-high resolution image with a height of H and a width of W 0≤m,n≤k and represents the integer domain, k is the pixel reorganization upsampling splicing magnification, where the image Each pixel of is: in, For images The pixel at position (k×x+m, k×y+n) in For image blocks The pixel at position (x,y) in

Images