Up-sampling re-optimization method for mean pooling down-sampling image

Abstract

The invention relates to an up-sampling re-optimization method for a mean pooling down-sampling image. The method comprises the following steps: step 1, reordering up-sampled HR (High Resolution) image data, so that pixel points of an N * N region (N is up-sampling multiplying power) of an HR image correspond to pixel points of an LR (Low Resolution) image; 2, pixel points of an N * N area of the HR image are processed separately according to three channels of RGB, and corresponding optimization coefficients are obtained; and step 3, multiplying the pixel value in each channel by an optimization coefficient, and obtaining an optimized pixel value after operations of rounding off, depolarization and the like. And step 4, reordering the optimized HR pixel points according to a normal sequence, and outputting an image. According to the method, the low-resolution image obtained by means of mean pooling down-sampling can be used as re-optimization processing of any up-sampling method, and indexes such as the PSNR, the SSIM and the LPIPS of the image are further improved.

Description

Technical Field

[0001] The invention relates to an upsampling and re-optimization method for a mean pooling downsampled image, and belongs to the technical field of image super-resolution reconstruction. Background Art

[0002] In the field of GPU graphics processing, pixel performance is often the bottleneck of system performance. Reducing the load of the pixel engine is of great significance to improving the performance of chip graphics. Image upsampling, which converts low-resolution (LR) images into high-resolution (HR) images and enlarges them to the desired size by algorithms after rendering low-resolution graphics, has become a widely used performance optimization method. It has been widely used from NVIDIA's DLSS to AMD's FSR. On the other hand, the upsampled high-resolution (HR) images can provide more details and better views, and are also widely used in digital high-definition television, medical imaging, terminal equipment and other fields.

[0003] At present, traditional upsampling methods include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos interpolation, etc. However, existing super-resolution upsampling technologies usually do not consider further optimization from the perspective of image downsampling, while mean pooling downsampling images retains the mean information of the original high-resolution image and has great potential for optimization. Summary of the invention

[0004] In view of the above potential optimization space, the present invention provides an upsampling and re-optimization method for mean pooling downsampled images, which fully considers the original image features retained by mean pooling downsampling, and further optimizes the high-resolution image obtained after any upsampling, which can further improve the quality of the upsampled image.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A method for upsampling and reoptimizing a mean pooled downsampled image comprises the following steps:

[0007] Step 1, upsample the downsampled image, and then divide the upsampled high-resolution image HR data into several N*N regions, where N is the image upsampling ratio; cache the image HR data in sequence and reorder them so that the image HR pixel data of each N*N region corresponds to the low-resolution image LR pixel data before upsampling;

[0008] Step 2: Process the reordered N*N image HR pixel data separately according to the three channels of RGB, calculate the sum SUM of the N*N pixel data for each channel, and multiply the corresponding image LR pixel data LR(i,j) by N*N to obtain the image information Y of the original area. The formula for calculating Y is:

[0009] Y=LR(i,j)*N*N (1)

[0010] Then divide the image information Y by the sum SUM to get the optimization coefficient K. The formula for K is:

[0011]

[0012] Step 3: convert the image HR pixel data HR of the N*N area 优化前 (i HR ,j HR ) is multiplied by the optimization coefficient K, and after rounding and depolarization operations, the optimized pixel value HR is obtained 优化后 (i HR ,j HR );

[0013] Step 4: rearrange the optimized pixel data in the order starting from the high-resolution image HR, and output the final image.

[0014] The present invention has the following beneficial effects:

[0015] This method is applicable to the re-optimization of all low-resolution images obtained by mean pooling after upsampling. After the high-resolution image obtained by upsampling is subjected to the re-optimization method of the present invention, the PSNR, SSIM, LPIPS and other index results of the image can be further improved. The method of the present invention has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention provides a flowchart of an upsampling and re-optimization method.

[0017] Figure 2 1 is a diagram showing the correspondence between HR and LR in an embodiment of the present invention.

[0018] Figure 3 A flowchart of another upsampling and re-optimization method provided by the present invention.

[0019] Figure 4 is LR(x src ,y src ) Position map of 16 pixels around it.

[0020] Figure 5 is the bicubic interpolation weight function image. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0022] like Figure 1As shown, the present invention provides an upsampling and re-optimization method for a mean pooling downsampled image, which specifically includes the following steps:

[0023] Step 1, the image upsampling ratio is N, and the upsampled HR (high resolution) image data is divided into several N*N regions. The HR data is cached and reordered in sequence, so that the pixel data of each N*N region corresponds to the LR (low resolution) pixel data before upsampling, and cannot be misaligned. It must be consistent with the corresponding relationship of the low-resolution image obtained by pooling the original image. There is no restriction on the upsampling method used. The LR image data before upsampling is obtained by downsampling the original image through mean pooling.

[0024] Step 2: Process the reordered N*N HR pixel data separately according to the three RGB channels. First, calculate the sum SUM of the N*N pixel data for each channel. At the same time, multiply the corresponding LR pixel data LR(i,j) by N*N to obtain the image information Y of the original area, and divide Y by SUM to obtain the optimization coefficient K.

[0025] The formula for finding Y is:

[0026] Y=LR(i,j)*N*N (1)

[0027] The formula for finding K is:

[0028]

[0029] Here, you can choose to first calculate the quotient of Y and SUM to obtain the optimization coefficient K, and then calculate the product of K and N*N data. You can also not calculate the optimization coefficient separately, and directly calculate the product of each HR image data in N*N and Y in the optimization step, and then calculate the quotient of and SUM to obtain the optimized result. Figure 3 shown.

[0030] Step 3: HR pixel value HR of N*N area 优化前 (i HR ,j HR ) is multiplied by the optimization coefficient K, and after rounding and depolarization operations, the optimized pixel value HR is obtained 优化后 (i HR ,j HR ). The optimization formula is:

[0031] HR 优化后 (i HR ,j HR )=HR 优化前 (i HR ,j HR )*K (3)

[0032] Step 4: rearrange the optimized pixel data in the order starting from the HR image and output the final image.

[0033] Among them, in step 1, the image reordering can be realized in the process of caching the image, and at least N rows of images need to be cached in the case of speed matching. In steps 1 and 2, the HR image cache speed before optimization is N times the HR image optimization speed to meet the speed matching. If the cache speed is N times greater than the optimization speed, data will be lost and errors will occur. If the cache speed is less than N times the optimization speed, it can work normally, but the optimal speed of optimization cannot be achieved.

[0034] Example 1

[0035] As the resolution of game images continues to increase, the pixel engine load is also gradually increasing. Often, the image is first downsampled, various rendering processes are completed at a low resolution, and then a certain upsampling method is used to improve the visual experience of the game. This embodiment optimizes the upsampling of game images, and the selected LR (low resolution) image resolution is 960*540, the HR (high resolution) image resolution after upsampling is 3840*2160, and the upsampling ratio is 4.

[0036] In this embodiment, the upsampling process uses a bicubic interpolation algorithm. Compared with other upsampling methods, bicubic interpolation is simple to implement and has better performance. The original LR image is 960*540 in size, and the HR image after being magnified 4 times is 3840*2160 in size. Since the pixels of the LR image are known, each point HR (x dst ,y dst ) has a point LR(x) corresponding to the low-resolution image (LR) src ,y src ), and (x dst ,y dst ) to (x src ,y src ) is represented by formula (3).

[0037]

[0038] According to the calculation, LR(x src ,y src ) will have decimals. So assuming LR(x src ,y src) in the low-resolution image is LR(x+u,y+v), where x and y represent the integer part of the coordinates, and u and v represent the decimal part of the coordinates. To ensure that when the interpolation point is an edge point, the nearest 16 points can be found, the LR image matrix to be interpolated is expanded by two rows and two columns before and after, that is, a total of four rows and four columns are expanded, and the expanded pixels are the pixels at the edge of the LR image.

[0039] According to the calculation model of bicubic interpolation, HR(x dst ,y dst ) can be represented by LR(x src ,y src ) is obtained by weighting the nearest 4*4 pixels in the low-resolution image, LR(x src ,y src ) The nearest 4*4 pixel positions in the low-resolution image are shown in the attached figure. Figure 4 As shown. And their corresponding weights can be calculated by the weight function of formula (4), and the corresponding function graph is shown in the attached Figure 5 As shown, a is usually -0.5, |x1| is the distance from the point to LR(x src ,y src ) are the distances in the horizontal or vertical directions respectively.

[0040]

[0041] Because the pixels in the same row to LR(x src ,y src ) are equal in vertical axis distance, and the distance from the pixel in the same column to LR(x src ,y src ) are equal in horizontal axis distance, so the pixels with the same horizontal coordinate will be multiplied by the same horizontal weight, and the pixels with the same vertical coordinate will be multiplied by the same vertical weight. Therefore, the same horizontal weight can be extracted to obtain the matrix A shown in formula (5), where u represents LR(x src ,y src )=LR(x+u,y+v) decimal part of the horizontal coordinate; extract the same vertical weight to obtain the matrix C shown in formula (6), where v represents LR(x src ,y src )=LR(x+u,y+v) decimal part of the ordinate, superscript T indicates transposition; while the matrix B in formula (7) is a matrix formed by arranging the 4*4 pixels according to their horizontal and vertical coordinates, then formula (8) can be used to perform the weighted summation of the 4*4 pixels to obtain HR(x dst ,y dst ) pixels.

[0042] A=[W(u+1) W(u) W(1-u) W(2-u)] (5)

[0043] C=[W(v+1) W(v) W(1-v) W(2-v)] T (6)

[0044]

[0045] HR(x dst ,y dst )=A*B*C (8)

[0046] The HR image is obtained by bicubic interpolation algorithm, divided into several 4*4 areas and reordered. Only 4 lines of HR image data can be processed each time. It should be noted that the processing speed is 4 times the cache speed to achieve speed matching. If the cache speed is 4 times greater than the optimization speed, data will be lost and errors will occur. If the cache speed is less than 4 times the optimization speed, it can work normally, but the optimal speed cannot be achieved. The optimization process is shown in the attached figure. Figure 2 As shown in the figure, HR1(i,j) is the pixel after optimization, LR1(i1,j1) is the pixel before interpolation, corresponding to a 4*4 region of the HR image obtained by upsampling, and bic(i,j) is a pixel in the 4*4 region before optimization. Here k1 is 16, SUM is the sum of the 16 pixels around bic(i,j), and the optimization process can also be expressed by formula (1) and formula (2).

[0047] This embodiment tested 47 images respectively, and evaluated them from three indicators: PSNR (peak signal-to-noise ratio, the higher the better), SSIM (structural similarity, the closer to 1 the better), and LPIPS (image perception similarity index, the lower the better). The average PSNR of the image after bicubic interpolation upsampling is 31.59, the average SSIM is 0.864, and the average LPIPS is 0.273. After using the re-optimization method of the present invention, the average PSNR is 31.94, the average SSIM is 0.875, and the average LPIPS is 0.221. All three indicators are significantly improved. Therefore, through the re-optimization process, the interpolated pixel points can be connected with the corresponding points of the original image to further improve the image quality and visual perception.

[0048] Example 2

[0049] In this embodiment, the upsampling process uses the Lanczos interpolation algorithm, which is more complex than the bicubic interpolation algorithm, but has a better upsampling effect. The same picture set as in Example 1 is used, and the average PSNR of the image after Lanczos interpolation upsampling is 31.7, the average SSIM is 0.867, and the average LPIPS is 0.278. After using the re-optimization method of the present invention, the average PSNR is 32.07, the average SSIM is 0.876, and the average LPIPS is 0.235. All three indicators are significantly improved, and the PSNR exceeds 32. Therefore, the re-optimization method has a significant effect on the Lanczos interpolation algorithm, which has higher performance than the bicubic interpolation algorithm.

[0050] Example 3

[0051] In this embodiment, the upsampling process uses a bilinear interpolation algorithm. The core idea of ​​the bilinear interpolation algorithm is to perform linear interpolation in two directions respectively. Compared with the bicubic interpolation algorithm and the Lanczos interpolation algorithm, the performance is poor, but the implementation is simplest and the running speed is the fastest. The same picture set as in Example 1 is used. The average PSNR of the image after bilinear interpolation upsampling is 30.88, the average SSIM is 0.849, and the average LPIPS is 0.293. After using the re-optimization method of the present invention, the average PSNR is 31.73, the average SSIM is 0.872, and the average LPIPS is 0.212. All three indicators are significantly improved. Therefore, the re-optimization method also has a significant effect on the bilinear interpolation algorithm whose performance is worse than the bicubic interpolation algorithm.

Claims

1. A method for upsampling and reoptimizing a mean pooled downsampled image, characterized in that: The steps include: Step 1, upsample the downsampled image, and then divide the upsampled high-resolution image HR data into several N*N regions, where N is the image upsampling ratio; cache the image HR data in sequence and reorder them so that the image HR pixel data of each N*N region corresponds to the low-resolution image LR pixel data before upsampling; Step 2: Process the reordered N*N image HR pixel data separately according to the three channels of RGB, calculate the sum SUM of the N*N pixel data for each channel, and multiply the corresponding image LR pixel data LR(i,j) by N*N to obtain the image information Y of the original area. The formula for calculating Y is: Y=LR(i,j)*N*N (1) Then divide the image information Y by the sum SUM to get the optimization coefficient K. The formula for K is: Step 3: convert the image HR pixel data HR of the N*N area 优化前 (i HR ,j HR ) is multiplied by the optimization coefficient K, and after rounding and depolarization operations, the optimized pixel value HR is obtained 优化后 (i HR ,j HR ); Step 4: rearrange the optimized pixel data in the order starting from the high-resolution image HR, and output the final image.

2. The upsampling and re-optimization method for a mean pooling downsampled image according to claim 1, characterized in that: In step 1, the downsampled image is obtained by downsampling the original image through mean pooling.

3. The upsampling and re-optimization method for a mean pooling downsampled image according to claim 1, characterized in that: The HR pixel data of the N*N images after reordering in step 1 correspond one-to-one with the LR pixel data of the image before upsampling and cannot be misaligned, and the corresponding relationship is consistent with the low-resolution image obtained by pooling the original image.

4. The upsampling and re-optimization method for mean pooling downsampled images according to claim 1, characterized in that: In step 2, the summing operation of N*N pixel data and the operation of multiplying the corresponding image LR pixel data LR(i,j) by N*N to obtain the image information Y are performed in parallel.

5. The upsampling and re-optimization method for mean pooling downsampled images according to claim 1, characterized in that In step 2, the quotient of the image information Y and SUM is first calculated to obtain the optimization coefficient K, and then the product of the optimization coefficient K and N*N data is calculated.

6. The upsampling and re-optimization method for mean pooling downsampled images according to claim 1, characterized in that In step 2, the product of each image HR pixel data and the image information Y in N*N is first calculated, and then the quotient of the sum SUM is calculated to obtain the optimized result.

7. The upsampling and re-optimization method for mean pooling downsampled images according to claim 1, characterized in that: In step 1, the images are reordered during the caching process. In the case of speed matching, at least N rows of images need to be cached.

8. The upsampling and re-optimization method for mean pooling downsampled images according to claim 7, characterized in that: In step 1 and step 2, the cache speed of the image HR before optimization is N times the optimization speed of the image HR to meet the speed matching.

Images