Up-sampling implementation method based on mean value down-sampling image

Abstract

The invention discloses an up-sampling implementation method, and belongs to the technical field of digital image up-sampling. For a given mean value down-sampling image, an existing interpolation algorithm, a transform domain algorithm, a deep learning algorithm and the like are difficult to realize high-performance and low-complexity up-sampling. According to the up-sampling implementation method based on the mean value down-sampling image, up-sampled pixels are fitted through four steps of step fitting, mean value adjustment, Gaussian smoothing and mean value adjustment, jumping of boundary pixels is simulated through step fitting, the high-frequency component of the image is increased, and the up-sampling effect of the up-sampling image is improved. The image after up-sampling is smoother through Gaussian smoothing, and meanwhile mean value adjustment is interspersed to ensure that the image after up-sampling can still obtain a source image after mean value down-sampling. The up-sampling algorithm provided by the invention has the advantages of low complexity, high processing speed, excellent performance and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image upsampling, and in particular relates to a high-performance, high-throughput, low-complexity upsampling implementation method based on mean downsampling images. Background Art

[0002] In the process of image processing, in order to reduce storage bandwidth, reduce transceiver bandwidth or reduce pixel engine load, the image can be first downsampled to obtain a low-resolution image, and then the low-resolution image is processed accordingly. After processing, the image is restored to its original resolution through the upsampling module. For example, in the field of GPU image processing, pixel performance is often the bottleneck of system performance. Therefore, reducing the load of the pixel engine is of great significance to improving the image processing capabilities of the chip. Rendering low-resolution images and then enlarging the images to the desired size through the upsampling module has become a widely used performance optimization method.

[0003] At present, the main image upsampling algorithms include: interpolation-based upsampling algorithm, reconstruction-based upsampling algorithm, and learning-based upsampling algorithm. Among them, the interpolation-based upsampling algorithm is a relatively simple and fast algorithm for improving image resolution. Common interpolation algorithms include nearest neighbor interpolation (Nearest), bilinear interpolation (Bilinear), bicubic interpolation (Bicubic), etc. The nearest neighbor interpolation selects the pixel value of the nearest pixel point among the four points around the sampling point as the pixel value of the sampling point. The bilinear interpolation uses the four points around the sampling point to perform linear interpolation in the horizontal and vertical directions. For bicubic interpolation, it uses the Bicubic basis function to calculate the weights of the 16 pixels of 4×4 around the sampling point, and then convolves with these 16 pixels to obtain the pixel value of the sampling point. These interpolation algorithms are simple to implement and run fast, but because the interpolation algorithm is implemented based on local smoothness, it may produce unsatisfactory phenomena such as ringing effect and sawtooth effect for edge areas with more high-frequency components.

[0004] Reconstruction-based upsampling algorithms can generally be divided into image upsampling algorithms based on frequency domain reconstruction and upsampling algorithms based on spatial domain reconstruction. Image upsampling algorithms based on frequency domain reconstruction mainly improve the resolution of images by eliminating spectrum aliasing in the transform domain, such as using discrete Fourier transform (DFT) or discrete cosine transform (DCT) to map image data to the transform domain, and then using inverse transform to obtain high-resolution images after eliminating aliasing in the transform domain. Wavelet transform can also be used to estimate high-pass subband coefficients, restore high-frequency components, and then obtain high-resolution images through inverse wavelet transform. Image upsampling algorithms based on spatial domain reconstruction use prior information as constraints and converge to the optimal solution or local optimal solution through continuous iteration. They mainly include Projection Onto Conve Sets (POCS), Iterative Back Projection (IBP) and Maximum A Posteriori (MAP). These reconstruction-based upsampling algorithms can better restore the high-frequency components of the image boundary area, and their performance is better than the interpolation-based upsampling algorithms, but they are highly complex, and most of them require iterative calculations and take a long time to converge.

[0005] The learning-based upsampling algorithm uses a large amount of training data to learn a certain correspondence between low-resolution images and high-resolution images, which is used to determine the parameters between network layers, and then predicts the high-resolution image corresponding to the low-resolution image based on the learned mapping relationship. The key to the learning-based upsampling algorithm is to build a suitable learning model. Currently, deep learning based on convolutional neural networks (CNNs) is the main method. It completes upsampling through feature extraction, nonlinear mapping, reconstruction and other steps, and adds residual learning, recursive learning, adversarial learning and other methods to improve the performance of reconstructed images and reduce the complexity of the network. Compared with the interpolation-based upsampling algorithm, the deep learning-based upsampling algorithm has better performance and restores more high-frequency components, but it requires more training samples and longer training time. In order to have better performance, it needs multiple convolutional layers, and the computational complexity is much higher than the interpolation-based upsampling algorithm. Summary of the invention

[0006] Purpose of the invention: The present invention provides an upsampling implementation method based on a mean downsampled image. Based on a 4×4 mean downsampled image, the upsampling algorithm provided by the present invention has many advantages such as strong scalability, low complexity, fast processing speed, and excellent performance.

[0007] Technical solution: A method for implementing upsampling based on mean downsampling images. The source image to be processed in this upsampling is obtained by 4×4 mean downsampling. The upsampling method is characterized by:

[0008] (1) Read the pixel data of the source image in sequence, splice the newly input pixels with the pixels in the row buffer, and form 5 rows and 3 columns of adjacent pixels with the stored pixels. If the read pixel is located at the boundary, the boundary pixels need to be double-expanded to form 5 rows or 3 columns;

[0009] (2) Perform 3×3 step fitting in the horizontal and vertical directions on the middle three pixels of 5 rows and 3 columns of adjacent pixels, map one pixel of the source image to 4×4 pixels, a total of 16 pixels, and then perform weighted average of the pixels obtained by horizontal and vertical fitting to obtain the pixels after step fitting;

[0010] (3) Use the pixel value of the source pixel point to adjust the 16 pixel points corresponding to the step fitting as a whole, so that the pixel average value of the 16 points after fitting is equal to the source pixel value, and the mean adjustment is completed;

[0011] (4) The fitted pixels after step fitting + mean adjustment in this round are spliced ​​with the stored pixels after step fitting + mean adjustment to form a 6×6 adjacent pixel array;

[0012] (5) Perform 3×3 Gaussian smoothing on the middle 4×4 pixels of the 6×6 adjacent pixel array, a total of 16 pixels. After completion, perform a mean adjustment on the 16 pixels to obtain the fitting pixel after the final upsampling of a single source pixel.

[0013] (6) Traverse the entire source image, find the fitting pixel at each point, and complete the upsampling of the entire image.

[0014] The specific process of step (1) is as follows:

[0015] Read the pixels of the source image in sequence, and process the data in parallel according to the different RGB color components. For a single component, read the 4 rows of data corresponding to the column of the input pixel from the row buffer of 4 rows, and splice it with the input into a 5-row 1-column source pixel array, and write the input pixel data into the row buffer of the earliest row. When the third row of pixels of the source image is read, after expansion, valid 5 rows and 1 column of pixel data can be output at this time, which can fit the first row of pixels of the source image. In order to fit the last two rows of pixels of the source image, there is no data input at the input end at this time, and the last row of the source image needs to be double-expanded so that valid 5 rows and 1 column of source pixel data can be formed.

[0016] To form a 5-row 3-column adjacent pixel array for subsequent step fitting + mean adjustment, when the input pixel is the first column of the source image, the 3-column pixel data are all assigned to the first column of the spliced ​​source image, 5 rows and 1 column of pixels, and then every time 5 rows and 1 column of pixels are input, the 5-row 3-column pixel array is shifted left and the input 5 rows and 1 column of pixels are assigned to the 3rd column. To fit the penultimate and second columns of pixels, there is no data input at the input end, and the last column of the source image needs to be double-expanded. At this time, the equivalent input end inputs the last column of the source image twice in succession, thereby forming valid 5 rows and 3 columns of pixel data.

[0017] In step (2), after obtaining a valid 5-row 3-column adjacent pixel array, 3×3 step fitting is performed on the three middle source pixels respectively. The specific process of step fitting for each pixel is as follows:

[0018] Assume that the pixel values ​​of each pixel in the 3×3 source pixel array are a 0，0 , a 0，1 , a 0，2 , a 1，0 , a 1，1 , a 1，2 , a 2，0 , a 2，1 , a 2，2 First, use vertical linear interpolation to find the pixel values ​​at the left and right ends of the step function. The calculation formulas for the four rows in the horizontal direction are:

[0019]

[0020]

[0021]

[0022]

[0023] In order to find the coordinates of the jump point of the step function, the gradient change calculation formula of the four horizontal rows is:

[0024]

[0025]

[0026]

[0027]

[0028] The formula for the final trip point is:

[0029]

[0030] The coordinates of the pixels to be inserted in the horizontal direction are When the insertion point coordinates are less than or equal to the jump point xi, its pixel value is pixel_li. When the insertion point coordinates are greater than the jump point xi, its pixel value is pixel_ri. At this time, the fitting pixels of 16 points in the horizontal direction 4×4 are obtained to complete the horizontal pixel point a 1，1 Similarly, the same method as the horizontal step fitting is used to complete the source pixel a 1，1 Step fitting in the vertical direction is performed to find the fitting pixels of 16 points in the vertical direction (4×4), and then the fitting pixels in the horizontal and vertical directions are weighted averaged to obtain a 1，1 The pixel of the final step fit.

[0031] In step (3), 1，1 The average value of the 16 pixels obtained after step fitting is calculated, and then a is subtracted from the average value. 1，1 Represents the overall deviation after fitting, so that the average value of the 16 pixels after fitting is equal to a 1，1 If the adjusted pixel value is negative, set it to 0; if it exceeds 255, set it to 255 to complete the mean adjustment step.

[0032] In step (4), pixels in rows 4 to 9 of the 12 rows and 4 columns of fitting pixels obtained by step fitting + mean adjustment in steps (2) and (3) are taken out, the 6 rows and 9 columns of the registered pixel array are shifted 4 columns to the left and columns 6, 7, 8, and 9 are assigned to the newly obtained 6 rows and 4 columns of pixels, and then pixels in columns 1 to 6 are taken out to form a 6×6 adjacent fitting pixel array.

[0033] In step (5), the middle 4×4 pixels of the extracted 6×6 adjacent fitting pixel array are respectively subjected to 3×3 Gaussian smoothing, and the pixel values ​​of the adjacent 3×3 pixel array are respectively b 0，0 , b 0，1 , b 0，2 , b 1，0 , b 1，1 , b 1，2 , b 2，0 , b 2，1 , b 2，2 , when Gaussian smoothing is performed, the smoothing calculation expression of the middle point is:

[0034]

[0035] Since the boundary is double-extended, the pixels corresponding to all pixels in the source image after step fitting + mean adjustment can be Gaussian smoothed to obtain the fitted pixels after step fitting + mean adjustment + Gaussian smoothing. Finally, the number of smoothed pixels is mean-adjusted once to obtain the final up-sampled pixel after fitting a single source pixel.

[0036] In step (6), each pixel of the source image is read in turn, and each pixel is subjected to step fitting + mean adjustment + Gaussian smoothing + mean adjustment processing to complete the fitting of all source pixels and obtain the final upsampled image.

[0037] Compared with the prior art, the beneficial effects of the upsampling method are as follows: the upsampling method is based on the source image obtained by 4×4 mean downsampling, and completes the upsampling of the entire image through four steps of step fitting + mean adjustment + Gaussian smoothing + mean adjustment. Step fitting is used to simulate boundary jumps and increase high-frequency components. 3×3 Gaussian smoothing is used to make the boundary transition smoother. At the same time, a mean adjustment step is added, so that the source image can still be obtained after the upsampled image is mean downsampling. The method has many advantages such as strong scalability, low complexity, fast processing speed, and excellent performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a single color component upsampling framework diagram;

[0039] Figure 2 It is a schematic diagram of double expansion of boundary pixels;

[0040] Figure 3 A schematic diagram of the specific process of upsampling a single pixel;

[0041] Figure 4 It is a schematic diagram of solving left and right pixels in horizontal step fitting;

[0042] Figure 5 It is a schematic diagram of solving the fitting pixels in the horizontal step fitting;

[0043] Figure 6 is a 3×3 Gaussian smoothing template;

[0044] Figure 7 This is a comparison chart of the PSNR performance of the upsampling algorithm of the present invention and bilinear interpolation;

[0045] Figure 8 This is a performance comparison chart of the upsampling algorithm of the present invention and the bilinear interpolation SSIM; DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings and specific examples. It should be pointed out that the following examples are intended to further describe the present invention and do not have any limiting effect.

[0047] Figure 1This is a schematic diagram of the single color component upsampling framework. The source image used in this example is 960 pixels wide and 540 pixels high. Each pixel contains three components, RGB. The bit width of each component is 8 bits, and the value range is between 0 and 255. In the specific hardware implementation, the single color component upsampling module will be instantiated three times, and the color components will be separated and spliced ​​at the input and output ends, so as to split the pixels or splice them into complete RGB pixels. The operating frequency of the entire module is set to 100MHz.

[0048] Based on the above parameter settings, the single color component upsampling module reads 8 bits of pixel data from the input end each time. When reading to the rightmost end of the image, due to double boundary expansion, there are two clocks that do not need to read data from the input end, but generate data based on the internally stored pixels. After two clocks, the next row of source image pixel data is read from the input end. When all pixels of the source image are read, due to boundary expansion, two rows need to be generated using the internally stored data. Therefore, fitting a row of source pixels requires two additional clocks, a total of 962 clocks, and fitting an image requires two additional rows of clocks, a total of 962×542=521404 clocks. The specific expansion method of double boundary expansion is as follows: Figure 2 As shown, the black dots represent the pixels of the source image, the white dots represent the extended pixels, and the pixel value of the extended pixel is equal to the pixel value of the boundary row or column.

[0049] like Figure 1 As shown in the figure, the specific process of the single color component upsampling module is as follows: read the source image pixel data from the input end, and read the 4 rows of pixels of the corresponding column from the row buffer of 4 rows, so as to form 5 rows and 1 column of pixels, and store the input pixels in the earliest row buffer at the same time, completing the read-before-write. When the row counter is recorded to 2, the third row of the source image is read at this time, the valid signal of the 5 rows and 1 column pixels is pulled high, and the 1st and 2nd rows of the 5 rows and 1 column are assigned to the 3rd row of pixels, that is, the first row of the source image. When the row counter is recorded to 3, the fourth row of the source image is read at this time, and the 1st row of the 5 rows and 1 column is assigned to the 2nd row of pixels. When the row counter is recorded to 540, there is no data input at this time, and the 5th row of the 5 rows and 1 column is assigned to the 4th row of pixels. When the row counter is recorded to 541, the 4th and 5th rows of the 5 rows and 1 column are assigned to the 3rd row of pixels.

[0050] In order to form a 5-row 3-column pixel array for subsequent step fitting + mean adjustment, when the column counter reaches 0, the first column of the source image is read, and the 3-column pixel data in the 5-row 3-column register are assigned to the input 5-row 1-column pixels. When the column counter reaches 960, the 3rd column is assigned to the 2nd column pixel. When the column counter reaches 961, the 2nd and 3rd columns are assigned to the 1st column pixel. For the remaining 5-row 1-column pixels, the 5-row 3-column pixel register is shifted left and the input 5-row 1-column pixels are assigned to the 3rd column. When the row counter counts to 541 and the column counter reaches 961, all data input is completed, and the 5-row 1-column pixel valid signal is pulled low.

[0051] After obtaining a valid 5-row 3-column adjacent pixel array, to fit a single source pixel, the process is as follows Figure 3 As shown, firstly, the 3×3 pixel array is used to perform step fitting on the three middle pixels respectively. For the step fitting of a single pixel, the pixel values ​​of each pixel in the 3×3 source pixel array are assumed to be a 0，0 , a 0，1 , a 0，2 , a 1，0 , a 1，1 , a 1，2 , a 2，0 , a 2，1 , a 2，2 ,like Figure 4 Linear interpolation is used to find the pixel values ​​at the left and right ends of the step function. The calculation formulas for the four horizontal rows are:

[0052]

[0053]

[0054]

[0055]

[0056] In order to find the coordinates of the jump point of the step function, the gradient change calculation formula of the four horizontal rows is:

[0057]

[0058]

[0059]

[0060]

[0061] The formula for the final trip point is:

[0062]

[0063] like Figure 5 As shown, the coordinates of the pixels to be inserted in the horizontal direction are When the insertion point coordinates are less than or equal to the jump point xi, its pixel value is pixel_li. When the insertion point coordinates are greater than the jump point xi, its pixel value is pixel_ri. Thus, the fitting pixels of 16 points in the horizontal direction 4×4 are obtained to complete the pixel point a. 1，1 Step fitting in the horizontal direction. At the same time, complete a 1，1 A step fitting is performed in the vertical direction, and then the pixels fitted in the horizontal and vertical directions are weighted averaged to obtain a 1，1 The pixel of the final step fit.

[0064] After completing the step fitting of the three pixels, the mean value of the fitted pixels is adjusted once, and the 16 pixels obtained after fitting are averaged. Then, the average value minus the source pixel is used to represent the overall deviation after fitting, and then the deviation value is subtracted from the 16 pixel values ​​after fitting, so that the average value of the 16 pixels after fitting is equal to a. 1，1 If the adjusted pixel value is negative, set it to 0; if it exceeds 255, set it to 255 to complete the mean adjustment step.

[0065] After completing the step fitting + mean adjustment of the three source pixels, Figure 3 In the first picture, only 6 rows of fitting pixels are retained from 4 to 9. Figure 1 The 6 rows and 9 columns register in the 6 rows and 9 columns are shifted 4 columns to the left, and the 6 to 9 columns are assigned with the newly fitted 6 rows and 4 columns of pixels. Then, the 1 to 6 columns of pixels are taken out from the newly assigned 6 rows and 9 columns register to form a 6 rows and 6 columns of adjacent pixel arrays, that is, Figure 3 The second picture in .

[0066] After obtaining the 6×6 adjacent pixel array, the middle 4×4 pixels are all subjected to 3×3 Gaussian smoothing. The Gaussian smoothing template is as follows: Figure 6 As shown, suppose that each pixel point of the adjacent 3×3 pixel array is b 0，0 , b 0，1 , b 0，2 , b 1，0 , b 1，1 , b 1，2 , b 2，0 , b 2，1 , b 2，2 , when Gaussian smoothing is performed, the smoothing calculation expression of the middle point is:

[0067]

[0068] Finally, the smoothed pixel number is adjusted once for the mean to obtain the final upsampled pixel after fitting a single source pixel point, that is, Figure 3 The third picture in .

[0069] Read each pixel of the source image in turn, perform step fitting + mean adjustment + Gaussian smoothing + mean adjustment on each source pixel, complete the fitting of all source pixels, and obtain the final upsampled image. Figure 7 , 8 As shown, compared with bilinear interpolation, the upsampling algorithm provided by the present invention has obvious performance improvements in both PSNR and SSIM indicators, and consists of four simple steps, with many advantages such as strong scalability, low complexity, fast processing speed, and excellent performance.

Claims

1. A method for implementing upsampling based on mean downsampling images, wherein the source image to be processed in the upsampling is obtained by 4×4 mean downsampling. The upsampling method is characterized in that it comprises the following steps: (1) Read the pixel data of the source image in sequence, splice the newly input pixels with the pixels in the row buffer, and form 5 rows and 3 columns of adjacent pixels with the stored pixels. If the read pixel is located at the boundary, the boundary pixels need to be double-expanded to form 5 rows or 3 columns; (2) Perform 3×3 step fitting in the horizontal and vertical directions on the middle three pixels of 5 rows and 3 columns of adjacent pixels, map one pixel of the source image to 4×4 pixels, a total of 16 pixels, and then perform weighted average of the pixels obtained by horizontal and vertical fitting to obtain the pixels after step fitting; (3) Use the pixel value of the source pixel point to adjust the 16 pixel points corresponding to the step fitting as a whole, so that the pixel average value of the 16 points after fitting is equal to the source pixel value, and the mean adjustment is completed; (4) The fitted pixels after step fitting + mean adjustment in this round are spliced ​​with the stored pixels after step fitting + mean adjustment to form a 6×6 adjacent pixel array; (5) Perform 3×3 Gaussian smoothing on the middle 4×4 pixels of the 6×6 adjacent pixel array, a total of 16 pixels. After completion, perform a mean adjustment on the 16 pixels to obtain the fitting pixel after the final upsampling of a single source pixel. (6) Traverse the entire source image, find the fitting pixel at each point, and complete the upsampling of the entire image.

2. The double expansion of boundary pixels according to claim 1, characterized in that: When the read pixel is located at the boundary of the source image, the row or column needs to be expanded twice, and the pixel value in the expanded row or column is equal to the pixel value of the boundary row or column, so that each pixel point of the source image can be located in the middle of the 5×5 adjacent source pixel array.

3. The 3×3 horizontal step fitting according to claim 1, characterized in that: Assume that the pixel values ​​of each pixel in the 3×3 source pixel array are a 0，0 , a 0，1 , a 0，2 , a 1，0 , a 1，1 , a 1，2 , a 2，0 , a 2，1 , a 2，2 , use linear interpolation to find the pixel values ​​at the left and right ends of the step function. The calculation formulas for the four horizontal rows are: In order to find the coordinates of the jump point of the step function, the gradient change calculation formula of the four horizontal rows is: The formula for the final trip point is: The coordinates of the pixels to be inserted in the horizontal direction are When the insertion point coordinates are less than or equal to the jump point xi, its pixel value is pixel_li. When the insertion point coordinates are greater than the jump point xi, its pixel value is pixel_ri. At this time, the fitting pixels of 16 points in the horizontal direction 4×4 are calculated to complete the pixel point a. 1，1 Step fit in the horizontal direction.

4. The step fitting according to claim 1 or 3, characterized in that: The same method as the horizontal step fitting is used to complete the source pixel a 1，1 Step fitting in the vertical direction is performed to find the fitting pixels of 16 points in the vertical direction (4×4), and then the fitting pixels in the horizontal and vertical directions are weighted averaged to obtain a 1，1 The pixel of the final step fit.

5. The mean value adjustment according to claim 1, characterized in that: to a 1，1 The average value of the 16 pixels obtained after step fitting is calculated, and then a is subtracted from the average value. 1，1 represents the overall deviation after fitting, and then the overall deviation is subtracted from the 16-point pixel value after fitting, so that the average value of the 16-point pixel after fitting is equal to a 1，1 If the adjusted pixel value is negative, set it to 0; if it exceeds 255, set it to 255 to complete the mean adjustment step.

6. The 3×3 Gaussian smoothing according to claim 1, characterized in that: Assume that after completing step fitting + mean adjustment, there is an adjacent 3×3 pixel array b 0，0 , b 0，1 , b 0，2 , b 1，0 , b 1，1 , b 1，2 , b 2，0 , b 2，1 , b 2，2 , when Gaussian smoothing is performed, the calculation expression is: After obtaining the fitting pixels of step fitting + mean adjustment + Gaussian smoothing, the fitting pixels are further mean adjusted to obtain the final up-sampled pixels corresponding to the source pixels.

Images