An image interpolation method based on edge repair and adaptive edge processing

Abstract

The application relates to the field of image processing algorithms and discloses an image interpolation method based on edge repairing and adaptive edge processing, which comprises the following steps: performing an edge expansion operation on a low-resolution image, so that each pixel point can be interpolated; performing traversal on the low-resolution image with a 4*4 matrix as a window after the edge expansion, and obtaining edge features Feature and Ratio of a local image in the window; adopting different super sampling methods for images of different edge types according to different values of Feature and Ratio; and finally limiting the amplitude of the output result to obtain an enlarged high-resolution image; the technical scheme in the application can be used in various super-resolution image processing occasions, such as game image rendering, can render a low-resolution image, and then can super sample the rendered result into a high-resolution image by using the method in the application, so that the game rendering time and the equipment pressure are greatly reduced.

Description

Technical Field

[0001] The present invention relates to the field of image processing algorithms, and in particular to an image interpolation method based on edge restoration and adaptive edge processing. Background Art

[0002] In our daily lives, we encounter vast amounts of image data. This data, captured by various cameras, contains numerous small objects, such as minute facial details in large scenes captured by cameras, small abnormalities in medical images, and distant pedestrians and vehicles captured by surveillance cameras. Therefore, accurately identifying these small objects is crucial for analyzing and processing this image data. However, these objects typically appear on the order of a dozen or even dozens of pixels in size, making their identification challenging.

[0003] Image interpolation is a technique that can enlarge a low-resolution image into a high-resolution image through interpolation, while effectively restoring its features. This technique not only improves image perception but also contributes to other computer vision tasks. Therefore, enlarging a low-resolution image into a high-resolution image while preserving or even enhancing the original image's features is a challenging problem with great practical application value. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides an image interpolation method based on edge restoration and adaptive edge processing to solve the problems of poor performance and complex algorithms of existing image supersampling.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] An image interpolation method based on edge restoration and adaptive edge processing, comprising:

[0007] Step 1: Expand the low-resolution image;

[0008] Step 2: Use a matrix of set size as a window to traverse the low-resolution image after edge expansion and calculate the edge feature parameters of the local image in the window;

[0009] Step 3: Calculate the average value of the local image edge feature parameters in the X direction ave within the window x And the average value ave in the Y direction y , and then normalized to the variable criterion, different interpolation kernel influence factors a are selected according to the value range of the variable criterion, and the local image in the window is interpolated;

[0010] Step 4: Limit the interpolation result to eliminate outliers and obtain a high-resolution image considering adaptive edge interpolation.

[0011] As a further description of the above technical solution, when expanding the low-resolution image in step one, the low-resolution image is expanded by two rows and two columns, so that a complete high-resolution image can be output when traversing the low-resolution image using a 4*4 matrix as a window; in step two, the matrix size is set to a 4*4 matrix.

[0012] As a further description of the above technical solution, edge types can be roughly divided into two types: left and right edges and up and down edges. Assume that f(x,y) is the pixel value of the coordinate point (x,y), and use edge feature parameters Feature to describe the edge. Because the edges considered are divided into two types, left and right (X direction) edges and up and down (Y direction) edges, the corresponding Features are also divided into Features x and Feature y : Where f(x,y) is the pixel value at the center of the 3*3 sliding window. The larger the Feature, the stronger the edge of the image. Each 3*3 sliding window can have one Feature. x and Feature y , so a 4*4 window can obtain a total of four features x :Feature xA 、Feature xB 、Feature xC 、Feature xD , and four Features y :Feature yA 、Feature yB 、Feature yC 、Feature yD .

[0013] As a further description of the above technical solution, the average value of the edge feature parameters in the 4*4 window in the X direction The average value in the Y direction Then normalize it into a variable:

[0014] As a further description of the above technical solution, when the variable criterion falls within different ranges, the interpolation kernel The impact factor a takes different values. When criterion < 0.1, a = 0.22; when criterion ∈ [0.1, 0.2), a = 0.19; when criterion ∈ [0.2, 0.3), a = 0.17; when criterion ≥ 0.3, a = 0.165.

[0015] As a further description of the above technical solution, by bringing the value of a obtained by edge recognition and classification into the lanczos2 interpolation kernel, and bringing the weight calculation formula into the interpolation kernel, the weight contribution of the sixteen down-sampling points of each super-sampling point to the X direction and Y direction of the super-sampling point is calculated to obtain the weight matrix [X1, X2, X3, X4], [Y1, Y2, Y3, Y4]; where X1 = lanczos2 (1 + x1, a), X2 = lanczos2 (x1, a), X3=lanczos2(1-x1,a),X4=lanczos2(2-x1,a),Y1=lanczos2(1+y1,a),Y2=lanczos2(y1,a),Y3=lanczos2(1-y1,a),Y4=lanczos2(2-y1,a),where x1 and y1 are the distances between the centers of the supersampling point and the subsampling point in the X and Y directions respectively; the distance between two adjacent subsampling points is defined as 1;

[0016] Single channel value at each supersampling point where f i, is the weight contribution of the down-sampling point (i, j) to the super-sampling point, f i,j =X i Y j ×R′ i,j , R′ i,j is the color value of the color channel corresponding to the downsampling point (i, j).

[0017] The above method can be used to calculate the single-channel value of the first supersampling point in the window, and then calculate the single-channel value (such as the R channel) of each supersampling point in the window, and then calculate the values ​​of other channels (such as the G channel and the B channel) of each supersampling point in the window; then use a 4*4 window to traverse the entire low-resolution image to interpolate the high-resolution image.

[0018] Among them, X i Y j To calculate f i,j Normalization is required before: each time a single channel value of a supersampling point is obtained, the pixel values ​​of the sixteen downsampling points need to be weighted and summed, that is, the weights of the sixteen downsampling point pixel values ​​relative to the supersampling value need to be obtained, that is, the sixteen X i Y j, but for the lanczos2 interpolation method, sixteen X i Y j The sum is not 1, so it needs to be normalized, that is:

[0019] As a further description of the above technical solution, while calculating the edge feature parameter Feature in step 2, the edge feature parameter Ratio is also calculated. Before proceeding to step 4, it is first determined whether to perform cliff edge repair on the local image in the window based on the value of Ratio: when the average difference between two rows in the 4*4 window divided by the average difference between the other two rows is less than the cliff ratio CLIFF, the local image in the window is determined to be a cliff edge, and the edge of the local image needs to be repaired: the data of the downsampling points f(3,2) and f(3,3) are copied to the next two rows of the oversampled image to complete the edge repair.

[0020] As a further description of the above technical solution, when the interpolation result is limited and output and outliers are excluded in step 4, if the interpolation result is greater than the maximum value of the downsampling window, then the interpolation result is set to be equal to the maximum value of the downsampling value; if the interpolation result is less than the minimum value of the downsampling window, then the interpolation result is set to be equal to the minimum value of the downsampling window.

[0021] Compared with the prior art, the beneficial technical effects of the present invention are:

[0022] The technical solution of the present invention can be used in various super-resolution image processing scenarios, such as game image rendering. A low-resolution image can be rendered, and then the rendered result can be super-sampled into a high-resolution image using the method of this invention, thereby greatly reducing the game rendering time and device pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of an image interpolation method based on edge restoration and adaptive edge processing provided by an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of edge recognition types provided by an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of a gradient operator provided in an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of regional blocks provided in an embodiment of the present invention;

[0027] Figure 5 An error map without cliff edge repair provided by an embodiment of the present invention;

[0028] Figure 6An error map containing cliff edge repair provided by an embodiment of the present invention;

[0029] Figure 7 Schematic diagram of the position of super-sampling pixel numbers. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] Example 1

[0032] like Figure 1 As shown, an image interpolation method based on edge restoration and adaptive edge processing includes the following steps:

[0033] S1. Expand the edge of the low-resolution image.

[0034] S2. Take the expanded low-resolution image as input, traverse the image with a 4*4 matrix as a window, and calculate the edge feature parameters Feature and Ratio of the local image in the window; the edge types can be roughly divided into two types, the edge of the left and right trend and the edge of the up and down trend. Assume that f(x,y) is the pixel value of the coordinate point (x,y), and use the edge feature parameter Feature to describe the edge; Figure 2 As shown, considering that the edge is divided into left and right edges and upper and lower edges, the corresponding edge feature parameters Feature are also divided into Feature x and Feature y . Where f(x,y) is the center pixel value of the 3*3 sliding window, such as Figure 3 As shown. The larger the Feature, the stronger the edge of the image. Each 3*3 sliding window can have one Feature. x and Feature y , so a 4*4 window can obtain a total of 4 features x and 4 Features y ,like Figure 4 shown.

[0035] S3. Calculate the feature average value ave in the X and Y directions based on the values ​​of the four features calculated in a 4*4 window x and ave y, and then normalized to the variable criterion. Finally, different interpolation kernel influence factors a are selected according to the value range of criterion to perform interpolation operations on the image in the window.

[0036] Specifically, the edge type judgment variables in the 4*4 window are solved in the X direction and the Y direction. The average value of the edge feature parameter in the X direction is: The average value of the edge feature parameters in the Y direction is: Then normalize it into a variable:

[0037] When the variable criterion falls in different ranges, the interpolation kernel The impact factor a takes different values. When criterion < 0.1, a = 0.22; when criterion ∈ [0.1, 0.2), a = 0.19; when criterion ∈ [0.2, 0.3), a = 0.17; when criterion ≥ 0.3, a = 0.165.

[0038] By substituting the value of a obtained by edge recognition and classification into the lanczos2 interpolation kernel and substituting the weight calculation formula into the interpolation kernel, the weight contribution of the sixteen down-sampling points of each super-sampling point to the super-sampling point in the X and Y directions is calculated, and the weight matrices [X1, X2, X3, X4] and [Y1, Y2, Y3, Y4] are obtained; where X1 = lanczos2(1+x1, a), X2 = lanczos2(x1, a), X3 = lanczos2(1-x1, a), X4 = lanczos2(2-x1, a), Y1 = lanczos2(1+y1, a), Y2 = lanczos2(y1, a), Y3 = lanczos2(1-y1, a), Y4 = lanczos2(2-y1, a), where x1 and y1 are the distances between the centers of the super-sampling point and the down-sampling point in the X and Y directions, respectively. The distance between two adjacent downsampling points is defined as 1.

[0039] Single channel value at each supersampling point where f i, is the weight contribution of the down-sampling point (i, j) to the super-sampling point, f i,j =X i Y j ×R′ i,j , R′ i,j is the color value of the color channel corresponding to the downsampling point (i, j).

[0040] In this way, the single-channel value of the first super-sampled point can be calculated, and then the single-channel value (for example, R channel) of the super-sampled point in the entire super-sampled image can be calculated, and then the values ​​of other channels (for example, G channel and B channel) of the super-sampled point in the entire super-sampled image can be calculated.

[0041] For example, the single channel value R of the first point of the supersampled image 1,1 Calculated by the following formula: R 1,1 =f 1,1 +f 1,2 +f 1,3 +f 1,4 +f 2,1 +f 2,2 +f 2,3 +f 2,4 +f 3,1 +f 3,2 +f 3,3 +f 3,4 +f 4,1 +f 4,2 +f 4,3 +f 4,4 .

[0042] Among them, X i Y j To calculate f i,j Normalization is required before: each time a single channel value of a supersampling point is obtained, the pixel values ​​of the sixteen downsampling points need to be weighted and summed, that is, the weights of the sixteen downsampling point pixel values ​​relative to the supersampling value need to be obtained, that is, the sixteen X i Y j , but for the lanczos2 interpolation method, sixteen X i Y j The sum is not 1, so it needs to be normalized, that is:

[0043] S4. Determine whether to perform cliff edge repair on the image based on the value of Ratio. When the average difference between the lower (upper) two rows of the 4*4 window divided by the average difference between the upper (lower) two rows is less than the cliff ratio CLIFF, it is determined to be a cliff edge and an edge repair operation is performed on it. The data of the downsampled points f(3,2) and f(3,3) are copied to the next two rows of the supersampled image to complete the edge repair.

[0044] S5. Limit the output of the super-resolution sampling result to eliminate outliers. To remove outliers, after obtaining the interpolation result, if the interpolation output value is greater than the maximum value of the downsampling window, then set the outlier value equal to the maximum value of the downsampling value. If the interpolation output value is less than the minimum value of the downsampling window, then set it equal to the minimum value of the downsampling window. Finally, the image interpolation work based on edge restoration and adaptive edge processing is completed for the entire image.

[0045] Example 2

[0046] In this embodiment, it is assumed that a 4*4 sliding window matrix is Then, the image interpolation method of the present invention can be used to obtain an output result through the sliding window.

[0047] You can get 4 Features x and 4 Features y :

[0048] Further, according to step S2, calculate Then normalize it into a variable: You can calculate ave x =0.0667;ave y =0.4549; criterion=0.2608.

[0049] When criterion∈[0.2,0.3), a=0.17, we can get Furthermore, the weights X1, X2, X3, X4, Y1, Y2, Y3, and Y4 of each supersampled pixel are calculated. Taking the first supersampled point as an example, x1 = y1 = 1 / 8, the calculations yield: X1 = -0.0493, X2 = 0.1434, X3 = 0.9753, X4 = -0.1118, Y1 = -0.0493, Y2 = 0.1434, Y3 = 0.9753, and Y4 = -0.1118.

[0050] like Figure 7 As shown in the figure, the super-sampled pixels obtained by supersampling have been marked with serial numbers 1 to 16. It is worth mentioning that when downsampling, the sampling method is mean downsampling, and the final pixel coordinates are the center points. Assuming that the distance between two adjacent downsampling points is 1, the distance between the 16 points obtained by supersampling and the downsampling pixel points is not a perfect multiple of 1 / 4 ( Figure 7The distance deviation is not an integer multiple of the side length of the small and medium squares, but a distance deviation of 1 / 8 (the distance of half a small square). Therefore, the distance deviation of half a small square needs to be considered when bringing the distance into the lanczos2 interpolation kernel function.

[0051] Calculate the normalized weights corresponding to all downsampling points: Get sixteen normalized weight values: The single-channel color value of the first super-sampled pixel can be obtained by multiplying and summing the sixteen normalized weight values ​​and the image pixel value of the down-sampled point:

[0052] Thus, the interpolation value of each super sampling point of the 4*4 window is calculated, and the result is By using a 4*4 window to traverse the entire low-resolution image, a magnified high-resolution image can be interpolated. Example 2 shows the calculation process of magnifying an image sixteen times, for example, magnifying a 1K image to a 4K image.

[0053] Example 3

[0054] In this embodiment, the value of Ratio is used to determine whether to perform cliff edge repair on the image. When the average difference between the lower (upper) two rows of the 4*4 window divided by the average difference between the upper (lower) two rows is less than the cliff ratio CLIFF, it is determined to be a cliff edge and an edge repair operation is performed on it. The downsampled data f(3,2) and f(3,3) are copied to the next few rows of the supersampled image to complete the edge repair. If the cliff edge is not performed on the entire image, the edge error diagram of the image is as follows: Figure 5 , with large white and black errors (grey errors are larger in the image itself). After performing cliff edge repair on the entire image, the edge error of the image is significantly reduced, and the effect is better, such as Figure 6 shown.

[0055] In order to further determine which rows of data are most effective for cliff edge restoration, the present invention also conducted experiments on cliff edge restoration. The algorithm effect was evaluated by downsampling the high-resolution original image and then interpolating it. The PSNR and SSIM were compared with the original image. The experimental results are shown in the following table.

[0056]

[0057] Where PSNR_N and SSIM_N represent the PSNR and SSIM results of directly overwriting N rows of data from pic_EAUP with the data from pic_downscale (the uncovered parts are still interpolated). Testing with the official dataset found that copying the downsampled data to overlay the two rows with the smallest oversampled derivatives worked best.

[0058] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image interpolation method based on edge restoration and adaptive edge processing, characterized in that: include: Step 1: Expand the low-resolution image; Step 2: Use a matrix of set size as a window to traverse the low-resolution image after edge expansion and calculate the edge feature parameters of the local image in the window; Step 3: Calculate the average value of the local image edge feature parameters in the X direction within the window and the average value in the Y direction , and then normalized to the variable criterion, different interpolation kernel influence factors a are selected according to the value range of the variable criterion, and the local image in the window is interpolated; Step 4: Limit the interpolation result to eliminate outliers and obtain a high-resolution image considering adaptive edge interpolation; Interpolation Kernel ; When interpolating the local image in the window in step 3, the weight calculation formula is brought into the interpolation kernel to calculate the weight contribution of the sixteen down-sampling points of each super-sampling point to the X and Y directions of the super-sampling point, and the weight matrix is ​​obtained. 、 ;in, , , , , , , , ,in and are the distances between the centers of the supersampling point and the downsampling point in the X and Y directions respectively; Single channel value at each supersampling point ,in is the downsampling point The weighted contribution to the super-sampling points, , is the downsampling point The color value of the corresponding color channel.

2. The image interpolation method based on edge restoration and adaptive edge processing according to claim 1, characterized in that: When expanding the low-resolution image in step 1, the low-resolution image is expanded by two rows and two columns; the matrix size set in step 2 is matrix.

3. The image interpolation method based on edge restoration and adaptive edge processing according to claim 1, characterized in that: In step 2, The matrix is ​​used as a window to traverse the low-resolution image and is embedded in In the matrix Sliding window calculation of edge feature parameters of the local image in the window Feature: edge feature parameters in the X direction , edge feature parameters in the Y direction ,in for The center pixel value of the sliding window; each The sliding window has a and a , then each The window corresponds to four : 、 、 、 , and four : 、 、 、 .

4. The image interpolation method based on edge restoration and adaptive edge processing according to claim 3, characterized in that: In step 3, the average value of the edge feature parameter Feature in the X direction , the average value in the Y direction ;variable .

5. The image interpolation method based on edge restoration and adaptive edge processing according to claim 4, characterized in that: when hour, ;when hour, ;when hour, ;when hour, .

6. The image interpolation method based on edge restoration and adaptive edge processing according to claim 3, characterized in that: In step 2, while calculating the edge feature parameter Feature, the edge feature parameter Ratio is calculated. Before proceeding to step 4, the value of Ratio is used to determine whether to perform cliff edge repair on the local image in the window: If the average difference between two rows in the window divided by the average difference between the other two rows is less than the cliff ratio CLIFF, the local image in the window is judged to be a cliff edge and edge repair is required for the local image: and The data is copied to the next two rows of the supersampled image to complete the edge repair.

7. The image interpolation method based on edge restoration and adaptive edge processing according to claim 1, characterized in that: In step 4, when the interpolation result is limited and output and outliers are excluded, if the interpolation result is greater than the maximum value of the downsampling window, then the interpolation result is set to be equal to the maximum value of the downsampling value; if the interpolation result is less than the minimum value of the downsampling window, then the interpolation result is set to be equal to the minimum value of the downsampling window.

Images