An upsampling re-optimization method for mean-pooling down-sampled images

Abstract

The present application relates to a kind of upsampling reoptimization method for mean pooling downsampling image.The method includes the following steps: step 1, the HR (high resolution) image data after upsampling is reordered, so that HR image N*N area (N is the up-sampling multiple) and the pixel point of LR (low resolution) image correspond.Step 2, the pixel point of HR image N*N area is handled separately according to RGB three channels, and the corresponding optimization coefficient is obtained.Step 3, the pixel value in each channel is multiplied by optimization coefficient, and after rounding and depolarization operation, the optimized pixel value is obtained.Step 4, the optimized HR pixel point is reordered again according to normal order, and image is output.The method of the present application can be used as reoptimization processing of any upsampling method for low resolution image obtained by mean pooling downsampling, to further improve the indicators such as PSNR, SSIM, LPIPS of image.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of up-sampling re-optimization method for mean pooling down-sampling image, belong to image super-resolution reconstruction technical field. BACKGROUND

[0002] In the field of GPU graphics processing, pixel performance is often the bottleneck of system performance, and reducing the load of pixel engine is of great significance to improve the graphics performance of the chip. Image up-sampling, i.e. converting a low-resolution (LR) image to a high-resolution (HR) image, has become a widely used performance optimization method after rendering a low-resolution image to the desired size by an algorithm. It is widely used from NVIDIA's DLSS to AMD's FSR. On the other hand, the high-resolution (HR) image after up-sampling can provide more details and better views, and is also widely used in digital high-definition television, medical imaging, terminal equipment and other fields.

[0003] Currently, traditional up-sampling methods include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos interpolation, etc. However, existing super-resolution up-sampling techniques usually do not consider further optimization from the perspective of image down-sampling. Mean pooling down-sampled images retain the mean information of the original high-resolution image, and have a great potential for optimization. SUMMARY

[0004] In view of the above potential optimization space, the present application provides an up-sampling re-optimization method for mean pooling down-sampled images, which fully considers the original image features retained by mean pooling down-sampling. After obtaining a high-resolution image by any up-sampling method, the image is further optimized to improve the quality of the up-sampled image.

[0005] The technical solution adopted by the present application is as follows:

[0006] An up-sampling re-optimization method for mean pooling down-sampled images includes the following steps:

[0007] Step 1: up-sampling the down-sampled image, then dividing the high-resolution image HR data after up-sampling into several N*N regions, where N is the up-sampling factor of the image; sequentially buffering the image HR data and reordering it, so that the image HR pixel data of each N*N region corresponds to the low-resolution image LR pixel data before up-sampling;

[0008] Step 2: separate the reordered N*N image HR pixel data according to the RGB three channels, and process each channel. Calculate the sum SUM of N*N pixel data for each channel, multiply the corresponding image LR pixel data LR(i,j) by N*N to obtain the image information Y of the original region, and the formula for calculating Y is:

[0009] Y = LR(i,j) * N * N (1)

[0010] Then, the image information Y is divided by SUM to obtain the optimization coefficient K. The formula for calculating K is:

[0011]

[0012] Step 3, extract the HR pixel data of the N*N region of the image. 优化前 (i HR ,j HR Multiply by the optimization factor K, and after rounding and depolarization operations, obtain the optimized pixel value HR. 优化后 (i HR ,j HR );

[0013] Step 4: Rearrange the optimized pixel data in the order starting from HR of the high-resolution image, and output the final image.

[0014] The present invention has the following beneficial effects:

[0015] This method is applicable to the re-optimization of all low-resolution images obtained by mean pooling after upsampling. The high-resolution images obtained after upsampling can further improve their PSNR, SSIM, LPIPS, and other performance metrics using this re-optimization method. This invention has promising application prospects. Attached Figure Description

[0016] Figure 1 The flowchart illustrates an upsampling and re-optimization method provided by this invention.

[0017] Figure 2 This is a diagram showing the correspondence between HR and LR in an embodiment of the present invention.

[0018] Figure 3 A flowchart of another upsampling and re-optimization method provided by the present invention.

[0019] Figure 4 For LR(x) src ,y src The location map of the surrounding 16 pixels.

[0020] Figure 5 This is the graph of the bicubic interpolation weighting function. Detailed Implementation

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] like Figure 1As shown, the present application provides an up-sampling re-optimization method for mean-pooling down-sampled images, which specifically includes the following steps:

[0023] Step 1, the image up-sampling rate is N, and the up-sampled HR (high resolution) image data is divided into several N*N regions. The HR data is sequentially cached and reordered so that the pixel data of each N*N region corresponds to the LR (low resolution) pixel data before up-sampling, and cannot be misaligned, and should be consistent with the corresponding relationship of the original image after pool processing to obtain a low resolution image. There is no restriction on the up-sampling method used. The LR image data before up-sampling is obtained by mean-pooling down-sampling of the original image.

[0024] Step 2, the N*N HR pixel data after reordering is processed according to the RGB three channels, and the sum SUM of N*N pixel data is calculated for each channel, and the corresponding LR pixel data LR(i,j) is multiplied by N*N to obtain the image information Y of the original region, and Y is divided by SUM to obtain the optimization coefficient K.

[0025] The formula for calculating Y is:

[0026] Y = LR(i,j)*N*N (1)

[0027] The formula for calculating K is:

[0028]

[0029] Here, the quotient of Y and SUM can be calculated first to obtain the re-optimization coefficient K, and then the product of K and N*N data can be calculated. Alternatively, the optimization coefficient can not be calculated separately, and the product of each HR image data in N*N and Y can be calculated directly in the optimization step, and then the quotient of the sum SUM is calculated to obtain the optimized result, as shown in Figure 3 .

[0030] Step 3, the HR pixel value HR 优化前 (i HR ,j HR ) of the N*N region is multiplied by the optimization coefficient K, and after rounding and depolarization operations, the optimized pixel value HR 优化后 (i HR ,j HR ) is obtained. The optimization formula is:

[0031] HR 优化后 (i HR ,j HR ) = HR 优化前 (i HR ,j HR )*K (3)

[0032] Step 4, rearrange the optimized pixel data according to the order of the start of the HR image, and output the final image.

[0033] In step 1, the image reordering can be implemented during the caching of the picture, and at least N rows of images need to be cached in the case of speed matching. In steps 1 and 2, the HR image caching speed before optimization is N times the HR image optimization speed to meet the speed matching. If the caching speed is greater than N times the optimization speed, data will be lost, resulting in an error. If the caching speed is less than N times the optimization speed, it can work normally, but the optimal speed of optimization cannot be achieved.

[0034] Embodiment 1

[0035] With the continuous improvement of game image resolution, the pixel engine load is gradually increasing, and often through the first down-sampling of the image, various rendering processing is completed at low resolution, and then a certain up-sampling method is used to improve the visual perception of the game. This embodiment is aimed at the up-sampling and optimization of game pictures, and the LR (low resolution) image resolution is selected as 960*540, the HR (high resolution) image resolution after up-sampling is 3840*2160, and the up-sampling ratio is 4.

[0036] In this embodiment, the bicubic interpolation algorithm is used in the up-sampling process. Compared with other up-sampling methods, bicubic interpolation is simple and has better performance. The original LR image is 960*540 in size, and the HR image after being enlarged by 4 times is 3840*2160 in size. Since the LR image pixels are known, each point HR(x dst ,y dst ) of the high-resolution image (HR) in the bicubic interpolation has its corresponding point LR(x src ,y src ) on the low-resolution image (LR), and the coordinate transformation formula from (x dst ,y dst ) to (x src ,y src ) is formula (3).

[0037]

[0038] According to the calculation, the position coordinates of LR(x src ,y src ) will be a decimal number. Therefore, it is assumed that LR(x src ,y srcThe coordinates of the image in the low-resolution image are LR(x+u,y+v), where x and y represent the integer parts of the coordinates, and u and v represent the fractional parts. To ensure that the 16 nearest surrounding points can be found when the interpolation point is an edge point, the LR image to be interpolated matrix is ​​expanded by two rows and two columns at the beginning and end, that is, expanded by a total of four rows and four columns. The expanded pixels are the pixels at the edge of the LR image.

[0039] According to the bicubic interpolation calculation model, HR(x) dst ,y dst The pixels of ) can be obtained by LR(x) src ,y src LR(x) is obtained by weighting the nearest 4x4 pixels in a low-resolution image. src ,y src The positions of the nearest 4x4 pixels in the low-resolution image are shown in the attached image. Figure 4 As shown. Furthermore, their corresponding weights can be calculated using the weight function in formula (4), and the corresponding function graph is shown in the attached figure. Figure 5 As shown, where a is usually taken as -0.5, and |x1| is the distance from the point to LR(x). src ,y src The distances along the horizontal or vertical axis, respectively.

[0040]

[0041] Because the pixels in the same row are to LR(x) src ,y src The vertical axis distances of pixels in the same column are equal, and the distances from the pixels in the same column to LR(x) are equal. src ,y src Since the x-axis distances of pixels are equal, pixels with the same x-coordinate will be multiplied by the same y-coordinate, and pixels with the same y-coordinate will be multiplied by the same y-coordinate. Therefore, the same y-coordinates can be extracted to obtain matrix A as shown in formula (5), where u represents LR(x) src ,y src =LR(x+u,y+v) for the fractional part of the x-coordinate; extract the same vertical weights to obtain matrix C as shown in formula (6), where v represents LR(x+u,y+v). src ,y src ) = LR(x+u,y+v) The decimal part of the ordinate, with the superscript T indicating transpose; while matrix B in formula (7) is a matrix formed by arranging 4*4 pixels according to their x and y coordinates, then formula (8) can be used to obtain HR(x) by weighted summation of the 4*4 pixels. dst ,y dst (pixels).

[0042] A=[W(u+1) W(u) W(1-u) W(2-u)] (5)

[0043] C = [W(v+1) W(v) W(1-v) W(2-v)] T (6)

[0044]

[0045] HR(x dst , y dst ) = A * B * C (8)

[0046] The HR image is obtained by bicubic interpolation algorithm, and is divided into several 4*4 regions and reordered. Each time, only 4 rows of HR image data can be processed. It should be noted that the processing speed is 4 times the cache speed to achieve speed matching. If the cache speed is greater than 4 times the optimization speed, data will be lost, resulting in an error. If the cache speed is less than 4 times the optimization speed, it can work normally, but the best speed cannot be achieved. The optimization process is shown in the accompanying Figure 2 , LR1(i1, j1) is a pixel point before interpolation, corresponding to a 4*4 region of the up-sampled HR image, and bic(i, j) is a pixel point in a 4*4 region before optimization. Here k1 is 16, SUM is the sum of the pixels of the 16 pixel points around bic(i, j), and the optimization process can also be represented by formula (1) and formula (2).

[0047] In this embodiment, 47 images are tested respectively, and evaluated from three indexes of PSNR (peak signal-to-noise ratio, the higher the better), SSIM (structural similarity, the closer to 1 the better), and LPIPS (image perceptual similarity index, the lower the better). The average PSNR of the bicubic interpolation up-sampled image is 31.59, the average SSIM is 0.864, and the average LPIPS is 0.273. After using the re-optimization method of the application, the average PSNR is 31.94, the average SSIM is 0.875, and the average LPIPS is 0.221. All three indexes are significantly improved. Therefore, through the re-optimization process, the relationship between the interpolated pixel points and the corresponding points of the original image can be established, and the image quality and visual perception can be further improved.

[0048] Example 2

[0049] In this embodiment, the up-sampling process uses the Lanczos interpolation algorithm, which is more complex than the bicubic interpolation algorithm but has better up-sampling effect. The image set used is the same as in Embodiment 1, and the average PSNR of the image after Lanczos interpolation up-sampling is 31.7, the average SSIM is 0.867, and the average LPIPS is 0.278. After using the re-optimization method of the application, the average PSNR is 32.07, the average SSIM is 0.876, and the average LPIPS is 0.235. All three indicators are significantly improved, and the PSNR exceeds 32, so the re-optimization method has obvious effect on the Lanczos interpolation algorithm with higher performance than the bicubic interpolation algorithm.

[0050] Embodiment 3

[0051] In this embodiment, the up-sampling process uses the bilinear interpolation algorithm, which is based on the core idea of linear interpolation in two directions. Compared with the bicubic interpolation algorithm and the Lanczos interpolation algorithm, the performance is poorer, but the implementation is the simplest and the running speed is the fastest. The image set used is the same as in Embodiment 1, and the average PSNR of the image after bilinear interpolation up-sampling is 30.88, the average SSIM is 0.849, and the average LPIPS is 0.293. After using the re-optimization method of the application, the average PSNR is 31.73, the average SSIM is 0.872, and the average LPIPS is 0.212. All three indicators are significantly improved, so the re-optimization method also has obvious effect on the bilinear interpolation algorithm with poorer performance than the bicubic interpolation algorithm.

Claims

1. An up-sampling re-optimization method for mean-pooling down-sampled images, characterized in that, The method comprises the following steps: Step 1, up-sampling the down-sampled image obtained by down-sampling the original image through mean pooling, and then dividing the high-resolution image HR data after up-sampling into a plurality of N*N regions, wherein N is the up-sampling multiple of the image; sequentially buffering the image HR data and reordering, so that the image HR pixel data of each N*N region corresponds to the low-resolution image LR pixel data before up-sampling; the reordered N*N image HR pixel data corresponds to the low-resolution image LR pixel data before up-sampling one by one without dislocation, and the correspondence is consistent with that of the original image and the low-resolution image obtained through pooling; Step 2, processing the reordered N*N image HR pixel data according to the three channels of RGB, First, the optimization coefficient K is obtained by dividing the image information Y by the sum SUM, and then the product of the optimization coefficient K and the N*N data is obtained, specifically: for each channel, the sum SUM of the N*N pixel data is obtained, the corresponding image LR pixel data LR(i,j) is multiplied by N*N to obtain the image information Y of the original region, and the formula for Y is: Y=LR(i,j)*N*N (1) Then, the image information Y is divided by the sum SUM to obtain the optimization coefficient K, and the formula for K is: Or, without calculating the optimization coefficient K, first calculate the product of each image HR pixel data in N*N and the image information Y, and then calculate the quotient of the sum SUM, and then obtain the optimized pixel data. Step 3, multiply the image HR pixel data HR 优化前 (i HR ,j HR ) of the N*N region by the optimization coefficient K, and after rounding and depolarization operation, obtain the optimized pixel data HR 优化后 (i HR ,j HR ); Step 4, rearranging the optimized pixel data according to the order of the high-resolution image HR, and outputting to obtain the final image.

2. The method of claim 1, wherein, In step 2, the sum operation of the N*N pixel data and the operation of multiplying the corresponding image LR pixel data LR(i,j) by N*N to obtain the image information Y are performed in parallel.

3. The method of claim 1, wherein, In step 1, the reordering of the image is realized in the process of buffering the picture, and at least N rows of images need to be buffered in the case of speed matching.

4. The method of claim 3, wherein, In steps 1 and 2, the buffering speed of the image HR before optimization is N times the optimization speed of the image HR, which can meet the speed matching.

Images