An up-sampling implementation method based on mean value down-sampling image

Abstract

The application discloses an up-sampling implementation method belonging to the technical field of digital image up-sampling, and is characterized by the following steps: obtaining a mean down-sampled image; performing step fitting on the mean down-sampled image to obtain a step fitting image; performing mean adjustment on the step fitting image to obtain a mean adjustment image; performing Gaussian smoothing on the mean adjustment image to obtain a Gaussian smoothing image; and performing mean adjustment on the Gaussian smoothing image to obtain an up-sampled image.

Description

Technical Field

[0001] This invention belongs to the field of digital image upsampling technology, and in particular relates to a high-performance, high-throughput, and low-complexity upsampling implementation method based on mean downsampled images. Background Technology

[0002] In image processing, to reduce storage bandwidth, transmit / receive bandwidth, or alleviate the load on the pixel engine, the image can be downsampled to obtain a low-resolution image. This low-resolution image is then subjected to appropriate data processing, and finally, an upsampling module restores the image to its original resolution. In GPU image processing, pixel performance is often the bottleneck of system performance. Therefore, reducing the load on the pixel engine is crucial for improving the chip's image processing capabilities. Rendering a low-resolution image and then upsampling it to the desired size has become a widely used performance optimization technique.

[0003] Currently, image upsampling algorithms mainly include: interpolation-based upsampling algorithms, reconstruction-based upsampling algorithms, and learning-based upsampling algorithms. Among them, interpolation-based upsampling algorithms are relatively simple and fast algorithms for improving image resolution. Common interpolation algorithms include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. Nearest neighbor interpolation selects the pixel value of the nearest pixel among four points surrounding the sampling point as the sampling point pixel value. Bilinear interpolation uses linear interpolation of the four points surrounding the sampling point in the horizontal and vertical directions. For bicubic interpolation, it uses the Bicubic basis function to calculate the weights of 16 pixels in a 4×4 matrix surrounding the sampling point, and then convolves these 16 pixels to obtain the sampling point pixel value. These interpolation algorithms are simple to implement and fast to run. However, since interpolation algorithms are based on local smoothness, they may produce unsatisfactory phenomena such as ringing and jagged edges in edge regions with many high-frequency components.

[0004] Image upsampling algorithms based on reconstruction can generally be divided into image upsampling algorithms based on frequency domain reconstruction and image upsampling algorithms based on spatial domain reconstruction. Image upsampling algorithms based on frequency domain reconstruction mainly improve image resolution by eliminating spectral aliasing in the transform domain. For example, they can use Discrete Fourier Transform (DFT) or Discrete Cosine Transform (DCT) to map image data to the transform domain, eliminate aliasing in the transform domain, and then use inverse transform to obtain a high-resolution image. Alternatively, wavelet transform can be used to estimate high-pass subband coefficients, recover high-frequency components, and then use inverse wavelet transform to obtain a high-resolution image. Image upsampling algorithms based on spatial domain reconstruction use prior information as constraints and converge to the optimal solution or a local optimum through continuous iteration. These mainly include Projection Onto Conve Sets (POCS), Iterative Back Projection (IBP), and Maximum A posteriori (MAP). These reconstruction-based upsampling algorithms can better recover high-frequency components in image boundary regions and outperform interpolation-based upsampling algorithms. However, they are more complex and most require iterative calculations, resulting in longer convergence times.

[0005] Learning-based upsampling algorithms utilize large amounts of training data to learn a correspondence between low-resolution and high-resolution images. This correspondence is used to determine the parameters between network layers, and then the learned mapping is used to predict the corresponding high-resolution image from the low-resolution image. The key to learning-based upsampling algorithms lies in constructing a suitable learning model. Currently, deep learning based on Convolutional Neural Networks (CNNs) is dominant. CNNs perform upsampling through feature extraction, nonlinear mapping, and reconstruction, and incorporate residual learning, recursive learning, and adversarial learning to improve the performance of reconstructed images and reduce network complexity. Compared to interpolation-based upsampling algorithms, deep learning-based upsampling algorithms offer superior performance and restore more high-frequency components. However, they require more training samples and longer training times. To achieve better performance, they need multiple convolutional layers, resulting in significantly higher computational complexity than interpolation-based upsampling algorithms. Summary of the Invention

[0006] Purpose of the invention: This invention provides an upsampling method based on mean-downsampled images. Based on 4×4 mean-downsampled images, the upsampling algorithm provided by this invention has many advantages such as strong scalability, low complexity, fast processing speed, and excellent performance.

[0007] Technical solution: An upsampling method based on mean downsampling of images. In this upsampling, the source image to be processed is obtained by 4×4 mean downsampling. The upsampling method is characterized by:

[0008] (1) Read the pixel data of the source image in sequence, concatenate the newly input pixel with the pixel in the row buffer, and form 5 rows and 3 columns of adjacent pixels with the registered pixel. If the read pixel is located at the boundary, the boundary pixel needs to be double-expanded to form 5 rows or 3 columns.

[0009] (2) Perform 3×3 step fitting in the horizontal and vertical directions on the middle 3 pixels of the 5 rows and 3 columns of adjacent pixels, and map one pixel of the source image to 4×4 pixels. Then, perform weighted averaging on the pixels obtained by fitting in the horizontal and vertical directions to obtain the pixels after step fitting.

[0010] (3) Use the pixel values ​​of the source pixel to make an overall adjustment of the 16 pixels corresponding to the step fit, so that the average pixel value of the 16 fitted points is equal to the source pixel value, thus completing the mean adjustment.

[0011] (4) The fitted pixels after step fitting and mean adjustment in this round are spliced ​​with the registered pixels that have also been fitted and mean adjusted to form a 6×6 adjacent pixel array.

[0012] (5) Perform 3×3 Gaussian smoothing on the middle 4×4 of the 16 pixels in the 6×6 adjacent pixel array. After that, perform a mean adjustment on the 16 pixels to obtain the fitted pixel after the final upsampling of a single source pixel.

[0013] (6) Traverse the entire source image, find the fitted pixel at each point, and complete the upsampling of the entire image.

[0014] The specific process of step (1) is as follows:

[0015] The source image pixels are read sequentially, and the data is processed in parallel according to the different RGB color components. For a single component, the four rows of data corresponding to the input pixel are read from a four-row row buffer and concatenated with the input to form a 5x1 source pixel array. Simultaneously, the input pixel data is written to the row buffer of the earliest row. When the third row of pixels in the source image is read, after expansion, a valid 5x1 pixel array can be output, which can fit the first row of pixels in the source image. To fit the last two rows of pixels in the source image, since there is no input data at this point, the last row of the source image needs to be double-expanded to form a valid 5x1 source pixel array.

[0016] To form a 5x3 adjacent pixel array for subsequent step fitting and mean adjustment, when the input pixel is the first column of the source image, all three columns of pixel data are assigned the values ​​of the stitched first column (5x1) of the source image. Subsequently, for every 5x1 input pixels, the 5x3 pixel array is shifted left and the input 5x1 pixels are assigned to the third column. To fit the last two columns of pixels, since there is no input data at this point, the last column of the source image needs to be double-expanded. In this case, the equivalent input terminal receives the last column of the source image twice consecutively, thus forming a valid 5x3 pixel data set.

[0017] In step (2), after obtaining an effective 5x3 adjacent pixel array, a 3×3 step fit is performed on the three middle source pixels. The specific process for step fitting of each pixel is as follows:

[0018] Let the pixel values ​​of each pixel in the 3×3 source pixel array be a. 0，0 a 0，1 a 0，2 a 1，0 a 1，1 a 1，2 a 2，0 a 2，1 a 2，2 First, the pixel values ​​at both ends of the step function are calculated using vertical linear interpolation. The calculation formulas for the four horizontal rows are as follows:

[0019]

[0020]

[0021]

[0022]

[0023] To determine the coordinates of the jump point of the step function, the formula for calculating the gradient change in the four horizontal rows is as follows:

[0024]

[0025]

[0026]

[0027]

[0028] The formula for the final transition point is:

[0029]

[0030] The coordinates of the pixels to be inserted in the horizontal direction are as follows: When the coordinates of the insertion point are less than or equal to the jump point xi, its pixel value is taken as pixel_li; when the coordinates of the insertion point are greater than the jump point xi, its pixel value is taken as pixel_ri. In this case, the fitted pixels for 16 points in a 4×4 pattern in the horizontal direction are calculated, completing the pixel value a in the horizontal direction. 1，1 The step fitting is performed. Similarly, the same method as the horizontal step fitting is used to complete the fitting of the source pixel a. 1，1 For vertical step fitting, 16 fitted pixels (4×4 points) are obtained in the vertical direction. Then, a weighted average of the fitted pixels in the horizontal and vertical directions is calculated to obtain 'a'. 1，1 The final step fit of the pixels.

[0031] In step (3), for a 1，1 The average value of the 16 pixels obtained after step fitting is calculated, and then 'a' is subtracted from the average value. 1，1 This represents the overall deviation after fitting, thus making the average value of the 16 pixels after fitting equal to a. 1，1 If the adjusted pixel value is negative, set it to 0; if it exceeds 255, set it to 255, thus completing the mean adjustment step.

[0032] In step (4), the pixels in rows 4 to 9 of the 12 rows and 4 columns of fitted pixels obtained by step fitting and mean adjustment in steps (2) and (3) are taken out. The 6 rows and 9 columns of registered pixel array are moved 4 columns to the left and the 6th, 7th, 8th and 9th columns are assigned the newly obtained 6 rows and 4 columns of pixels. Then, the pixels in columns 1 to 6 are taken out to form a 6×6 adjacent fitted pixel array.

[0033] In step (5), the middle 4×4 pixels of the extracted 6×6 adjacent fitted pixel array are subjected to 3×3 Gaussian smoothing. Let the pixel values ​​of the adjacent 3×3 pixel arrays be b. 0，0 b 0，1 b 0，2 b 1，0 b 1，1 b 1，2 b 2，0 b 2，1 b 2，2 When performing Gaussian smoothing, the expression for smoothing the intermediate points is:

[0034]

[0035] Because the boundary is double-expanded, Gaussian smoothing can be performed on all pixels in the source image corresponding to the step fit + mean adjustment, to obtain the fitted pixels after step fit + mean adjustment + Gaussian smoothing. Finally, the number of smoothed pixels is mean-adjusted once to obtain the final upsampled pixels after fitting a single source pixel.

[0036] In step (6), each pixel of the source image is read sequentially, and step fitting + mean adjustment + Gaussian smoothing + mean adjustment are performed on each pixel to complete the fitting of all source pixels and obtain the final upsampled image.

[0037] Compared with existing technologies, the advantages of this upsampling method are as follows: Based on the source image obtained by 4×4 mean downsampling, this upsampling method completes the upsampling of the entire image through four steps: step fitting, mean adjustment, Gaussian smoothing, and mean adjustment. Step fitting is used to simulate boundary jumps and increase high-frequency components. 3×3 Gaussian smoothing is used to make the boundary transition smoother. At the same time, the mean adjustment step is added so that the upsampled image can still be obtained as the source image after mean downsampling. It has many advantages such as strong scalability, low complexity, fast processing speed, and excellent performance. Attached Figure Description

[0038] Figure 1 A framework diagram for upsampling a single color component;

[0039] Figure 2 A schematic diagram of the double expansion of boundary pixels;

[0040] Figure 3 A schematic diagram illustrating the specific process of upsampling a single pixel;

[0041] Figure 4 This is a schematic diagram illustrating the solution for the left and right pixels in a horizontal step fitting process.

[0042] Figure 5 This is a schematic diagram illustrating the solution of fitted pixels in a horizontal step fit.

[0043] Figure 6 Use a 3×3 Gaussian smoothing template;

[0044] Figure 7 This is a comparison chart of the PSNR performance of the upsampling algorithm and bilinear interpolation in this invention;

[0045] Figure 8 This is a performance comparison chart between the upsampling algorithm of this invention and the bilinear interpolation SSIM. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and specific examples. It should be noted that the following examples are intended to further describe the present invention and are not intended to limit it in any way.

[0047] Figure 1This is a schematic diagram of a single color component upsampling framework. In this example, the source image used is 960 pixels wide and 540 pixels high. Each pixel contains three components: RGB. Each component has a bit width of 8 bits and a value range between 0 and 255. In the specific hardware implementation, the single color component upsampling module will be instantiated three times. At the same time, the color components will be separated and spliced ​​at the input and output ends, thereby splitting the pixels or splicing them into complete RGB pixels. The entire module is set to operate at a frequency of 100MHz.

[0048] Based on the above parameter settings, the single color component upsampling module reads 8 bits of pixel data from the input end each time. When reading the rightmost edge of the image, due to double boundary expansion, two clock cycles are generated based on the internally stored pixel data instead of reading data from the input end. After two clock cycles, the next row of source image pixel data is read from the input end. When all pixels of the source image have been read, due to boundary expansion, two rows need to be generated using the internally stored data. Therefore, fitting one row of source pixels requires two additional clock cycles, totaling 962 clock cycles. Fitting an entire image requires two additional rows of clock cycles, totaling 962 × 542 = 521404 clock cycles. The specific expansion method of double boundary expansion is as follows: Figure 2 As shown, black dots represent pixels in the source image, and white dots represent extended pixels. The pixel value of an extended pixel is equal to the pixel value of the boundary row or column.

[0049] like Figure 1 As shown, the specific process of the single color component upsampling module is as follows: It reads source image pixel data from the input terminal and reads the corresponding four rows of pixels from the four-row row buffer, thus forming a five-row, one-column pixel array. Simultaneously, the input pixels are stored in the earliest row buffer, completing a read-then-write operation. When the row counter reaches 2, the third row of the source image is read, the valid signal for the five-row, one-column pixel array is pulled high, and rows 1 and 2 of the five-row, one-column array are assigned to the third row of pixels, i.e., the first row of the source image. When the row counter reaches 3, the fourth row of the source image is read, and row 1 of the five-row, one-column array is assigned to the second row of pixels. When the row counter reaches 540, there is no data input at the input terminal, and row 5 of the five-row, one-column array is assigned to the fourth row of pixels. When the row counter reaches 541, rows 4 and 5 of the five-row, one-column array are assigned to the third row of pixels.

[0050] To form a 5x3 pixel array for subsequent step fitting and mean adjustment, when the column counter reaches 0, the first column of the source image is read, and the pixel data in the 5x3 register is assigned to the input 5x1 pixel. When the column counter reaches 960, the third column is assigned the second column pixel, and when the column counter reaches 961, the second and third columns are assigned the first column pixel. For every additional 5x1 pixel input, the 5x3 pixel register is shifted left and the input 5x1 pixel is assigned to the third column. When the row counter reaches 541 and the column counter reaches 961, all data input is complete, and the valid signal for the 5x1 pixel array is pulled low.

[0051] After obtaining an effective 5x3 adjacent pixel array, the process for fitting a single source pixel is as follows: Figure 3 As shown, firstly, a step fit is performed on the three middle pixels using a 3×3 pixel array. For step fit on a single pixel, let the pixel value of each pixel in the 3×3 source pixel array be a. 0，0 a 0，1 a 0，2 a 1，0 a 1，1 a 1，2 a 2，0 a 2，1 a 2，2 ,like Figure 4 The pixel values ​​at both ends of the step function are obtained using linear interpolation. The calculation formulas for the four horizontal rows are as follows:

[0052]

[0053]

[0054]

[0055]

[0056] To determine the coordinates of the jump point of the step function, the formula for calculating the gradient change in the four horizontal rows is as follows:

[0057]

[0058]

[0059]

[0060]

[0061] The formula for the final transition point is:

[0062]

[0063] like Figure 5 As shown, the coordinates of the pixels to be inserted in the horizontal direction are respectively When the coordinates of the insertion point are less than or equal to the jump point xi, its pixel value is taken as pixel_li; when the coordinates of the insertion point are greater than the jump point xi, its pixel value is taken as pixel_ri. This process yields the fitted pixels for 16 points in a 4×4 horizontal direction, completing the pixel value for point a. 1，1 Horizontal step fitting. Simultaneously, perform a calculation using the same method. 1，1 A step fit is performed in the vertical direction, and then a weighted average is taken of the pixels fitted in the horizontal and vertical directions to obtain a. 1，1 The final step fit of the pixels.

[0064] After completing the step fit for 3 pixels, the mean of each fitted pixel is adjusted. The average of the 16 fitted pixels is then taken. The overall deviation of the fitted pixel is represented by subtracting the source pixel value from the average value. This deviation is then subtracted from the fitted pixel values ​​to make the average of the 16 fitted pixels equal to 'a'. 1，1 If the adjusted pixel value is negative, set it to 0; if it exceeds 255, set it to 255, thus completing the mean adjustment step.

[0065] After completing the step fit and mean adjustment of the three source pixels, i.e. Figure 3 The first image in the dataset retains only rows 4 to 9 (6 rows in total) of fitted pixels. Figure 1 The 6x9 register in the original array is shifted 4 columns to the left, and columns 6 through 9 are assigned the newly fitted 6x4 pixel values. Then, pixels from columns 1 through 6 are retrieved from the newly assigned 6x9 register to form a 6x6 adjacent pixel array. Figure 3 The second picture in the series.

[0066] After obtaining a 6×6 adjacent pixel array, perform 3×3 Gaussian smoothing on the middle 4×4 pixels. The Gaussian smoothing template is as follows: Figure 6 As shown, let each pixel in an adjacent 3×3 pixel array be b. 0，0 b 0，1 b 0，2 b 1，0 b 1，1 b 1，2 b 2，0 b 2，1 b 2，2 When performing Gaussian smoothing, the expression for smoothing the intermediate points is:

[0067]

[0068] Finally, the smoothed pixel count is adjusted by mean to obtain the final upsampled pixel after fitting a single source pixel. Figure 3 The third image in the series.

[0069] Each pixel of the source image is read sequentially, and a step fit + mean adjustment + Gaussian smoothing + mean adjustment is performed on each source pixel to complete the fitting of all source pixels, resulting in the final upsampled image. For example... Figure 7 , 8 As shown, compared with bilinear interpolation, the upsampling algorithm provided by this invention has significant performance improvements in both PSNR and SSIM metrics. It consists of four simple steps and has many advantages such as strong scalability, low complexity, fast processing speed, and excellent performance.

Claims

1. A method for implementing up-sampling of a down-sampled image based on mean value, wherein the source image to be processed is obtained by 4x4 mean value down-sampling, and the method comprises the following steps: (1) reading pixel data of the source image in sequence, splicing the newly input pixel with the pixel in the row buffer, and combining the pixel to form 5 rows and 3 columns of adjacent pixels, if the read pixel is located at the boundary, a complete 5 rows and 3 columns of adjacent pixels cannot be read, at this time, the boundary pixel needs to be expanded twice to splice a complete 5 rows and 3 columns of pixels; (2) performing 3x3 step fitting in horizontal and vertical directions on the middle 3 pixel points of the 5 rows and 3 columns of adjacent pixels, mapping one pixel point of the source image to 4x4, i.e. 16 pixel points, and then performing weighted average on the pixels obtained by fitting in horizontal and vertical directions to obtain the pixel after step fitting; (3) adjusting the 16 pixel points corresponding to the pixel after step fitting as a whole by using the pixel value of the source pixel point, so that the average value of the 16 points after fitting is equal to the source pixel value, and the mean value adjustment is completed; (4) splicing the fitting pixel after step fitting and mean value adjustment in this round with the pixel registered and also subjected to step fitting and mean value adjustment to form a 6x6 adjacent pixel array; (5) performing 3x3 Gaussian smoothing on the middle 4x4, i.e. 16 pixel points of the 6x6 adjacent pixel array, and then performing mean value adjustment on the 16 pixel points to obtain the final fitting pixel of the single source pixel point after up-sampling; (6) traversing the entire source image to obtain the fitting pixel of each point, and completing the up-sampling of the entire image.

2. The method of claim 1, wherein the method is implemented based on a mean value down-sampled image. When the read pixel is located at the boundary of the source image, the row or column needs to be expanded twice, and the pixel value in the expanded row or column is equal to the pixel value of the boundary row or column, so that each pixel point of the source image can be located in the middle of the 5x5 adjacent source pixel array.

3. The method of claim 1, wherein: Let the pixel values of the pixels in a 3x3 source pixel array be a 0，0 , a 0，1 , a 0，2 , a 1，0 , a 1，1 , a 1，2 , a 2，0 , a 2，1 , a 2，2 The pixel values at the left and right ends of the step function are calculated using linear interpolation, and the calculation formulas for the four rows in the horizontal direction are as follows: To obtain the coordinates of the jump point of the step function, the gradient change calculation formula of the four rows in the horizontal direction is: The formula of the final jump point is: The coordinates of the pixel points to be inserted in the horizontal direction are respectively When the insertion point coordinate is less than or equal to the jump point xi, the pixel value is taken as pixel_li, and when the insertion point coordinate is greater than the jump point xi, the pixel value is taken as pixel_ri. At this time, the fitting pixels of 4x4=16 points in the horizontal direction are obtained, and the pixel point a 1,1 Step fitting in the horizontal direction.

4. The method of claim 1, wherein: The source pixel point a is obtained by using the same method as the horizontal step fitting 1，1 The vertical step fitting is performed to obtain 16 fitting pixels in the vertical direction, and then the fitting pixels in the horizontal direction and the vertical direction are weighted and averaged to obtain a 1,1 The final step fitting pixel.

5. The method of claim 1, wherein: a 1,1 The average value of the 16 pixel points obtained after step fitting is calculated, and the average value is subtracted from a 1，1 The overall deviation after fitting is represented, and the overall deviation is subtracted from the 16 pixel values after fitting, so that the average value of the 16 pixel points after fitting is equal to a 1，1 If the adjusted pixel value is negative, it is set to 0, and if it exceeds 255, it is set to 255, completing the mean adjustment step.

6. The method of claim 1, wherein: Set after completing the step fitting + mean adjustment, there are adjacent 3x3 pixel array b 0,0 , b 0,1 , b 0,2 , b 1,0 , b 1，1 , b 1,2 , b 2,0 , b 2,1 , b 2，2 , when Gaussian smoothing, the calculation expression is: After obtaining the fitting pixel after step fitting, mean value adjustment and Gaussian smoothing, the fitting pixel is subjected to mean value adjustment again to obtain the final up-sampled pixel corresponding to the source pixel point.

Images