A polarization image interpolation method based on edge perception

Abstract

The present invention discloses a polarization image interpolation method based on edge perception, which aims to reduce defects and restore lost polarization information by using an interpolation method. First, the polarization image is pre-processed for denoising to eliminate the problem of low image pixel quality caused by noise. Then, the edge perception weight is calculated for the denoised polarization image, a guide map is generated, and the image is processed by weighted guided filtering using a LOG operator, and then the error is estimated in the diagonal, horizontal and vertical directions. Finally, the intensity error in each direction is used to determine the specific interpolation method to obtain the missing polarization information of each pixel. The present invention fully considers the connection between the flat area and the edge area, improves the accuracy of the reconstructed polarization image, and uses the edge perception weight for processing to obtain a high-quality polarization image.

Description

Technical Field

[0001] The invention relates to a polarization image interpolation method based on edge perception, and belongs to the technical field of polarization image interpolation algorithms. Background Art

[0002] The three basic properties of light are intensity, wavelength, and polarization. Humans perceive the intensity and wavelength of light as brightness and color, respectively. However, polarization is invisible to the human eye. But the polarization properties of light provide a lot of valuable information. Polarization imaging is a new imaging technology developed on the basis of traditional imaging technology, which can capture the polarization properties of objects. The corresponding polarization image sensing system has been verified in various applications including biomedicine, such as 3D shape reconstruction and material classification. Recent progress in nanotechnology and nanomanufacturing has also made it possible. In order to make full use of this polarization information to identify materials, the processing of polarization images has become the focus of research. The use of interpolation algorithms to process polarization images is also widely used in various fields such as military, medicine, communications, satellites, remote sensing, and astronomical research and observation, and has achieved good results.

[0003] Split focal plane (DoFP) polarization image sensors have the significant advantage of being able to record all polarization information of a target object within one frame. However, this comes at the expense of spatial resolution (IFOV), since each filter can only record one polarization information of the target, and therefore, a pixel should rely on its neighbors to estimate the other three intensity measurements. Since each pixel has a different IFoV, errors are inevitably introduced in the evaluation process. However, if an interpolation method is used, the error can be significantly reduced. In order to compensate for this shortcoming, an interpolation algorithm is proposed to recover the missing polarization information at each pixel, and the interpolation method is used to restore part of the lost spatial resolution and improve the accuracy of the captured polarization information.

[0004] The interpolation algorithm can be regarded as a computational processing process for regenerating image information. It recovers continuous image information data with better resolution from the original limited discrete data with lower resolution. The interpolation algorithm mainly uses the grayscale values ​​of the image at adjacent positions to calculate the grayscale value of the unknown image, thereby realizing image restoration, scaling and reorganization, etc., and can obtain better and clearer images. The use of interpolation algorithms for polarization image processing is of great significance for improving the contrast, resolution, quality, etc. of polarization images. At the same time, it is also of great significance in the recognition and restoration of polarization image information.

[0005] The interpolation algorithm focuses on smoothing features such as image details to achieve better observation effects. Interpolation algorithms are also a key step in image processing for three-dimensional reconstruction technology. Therefore, the quality of image processing by interpolation algorithms plays a vital role in image reconstruction. Therefore, by using interpolation algorithms to calculate the target image, image information can be restored. Restore, reconstruct, reduce and enlarge the information actually required, and display or restore the detailed information or incomplete partial information of the image. Currently available interpolation algorithms include non-adaptive techniques, such as bilinear interpolation, bicubic interpolation, and bicubic spline interpolation. These three interpolation methods are based on non-adaptive linear filters that are spatially invariant. They are essentially low-pass filters that distort high-frequency details and smooth detail information. In some images, due to the smoothing of intensity images, jagged artifacts will be generated at the edges of objects. These jagged artifacts will reduce the imaging quality and should be eliminated. There are also some adaptive techniques, such as gradient-based interpolation methods and intensity-correlation-based interpolation methods. Gradient-based interpolation methods interpolate missing polarization components in four basic gradient directions to obtain more accurate edge features; intensity correlation-based interpolation methods use intensity correlation methods to directly detect edges in polarized images. This method fills in missing intensities along a single direction based on judgments in two orthogonal directions, and potentially useful estimates may be lost due to this binary decision. So these methods are not the best choice. Summary of the invention

[0006] In order to solve the problems existing in the prior art, the present invention provides a polarization image interpolation method based on edge perception, which generates a guide map in an edge-aware manner and effectively interpolates missing pixel values ​​without crossing the edge.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] A polarization image interpolation method based on edge perception, the specific steps are as follows:

[0009] Step 1, denoising and upsampling the polarization image to obtain polarization images in four directions of 0°, 45°, 90° and 135°;

[0010] Step 2: Calculate edge-aware weights and generate edge-aware guidance graphs;

[0011] Step 3: interpolate the polarization images in the four directions in the diagonal, horizontal and vertical directions respectively to obtain four images with different polarization directions.

[0012] Furthermore, the edge-aware guidance graph G in step 2 is:

[0013]

[0014] Among them, (i, j) represents pixels, v represents the horizontal direction, h represents the vertical direction, and X k (i,j)=X k1 (i,j)-X k2 (i,j),

[0015]

[0016]

[0017] represents the filtering operation, I d represents the denoised polarization image, and ε represents a very small positive value.

[0018] Further, step 3 is specifically as follows:

[0019] Step 3.1, diagonal direction

[0020] Calculate the pixel error ΔD in the diagonal direction of all pixels 45 and ΔD 135 , if the ΔD corresponding to a certain pixel 135 / ΔD 45 ≥λ1, then the four adjacent pixels at 135° are interpolated with bicubic spline for the pixel point; otherwise, the four adjacent pixels at 45° are interpolated with bicubic spline for the pixel point; λ1 indicates setting the first threshold;

[0021] Step 3.2: Horizontal and vertical directions

[0022] Step 3.2.1: Based on the denoised polarization image I d , get I h ,I v :

[0023]

[0024]

[0025] Among them, (i,j) represents a pixel,

[0026] Step 3.2.2: Use the edge-aware guided graph and the weighted guided filtering method based on the LOG operator to filter the I h ,I v Filter to obtain horizontal and vertical images ΔI h , ΔI v ;

[0027] Step 3.2.3: Calculate the horizontal and vertical pixel errors ΔD0 and ΔD 90 , if the ΔD0 / ΔD corresponding to a certain pixel 90 >λ2, then the four adjacent pixels in the horizontal direction of the pixel are interpolated by bicubic spline; if ΔD0 / ΔD 90 <λ2, then the four adjacent pixels in the vertical direction of the pixel are interpolated by bicubic spline; if ΔD0 / ΔD 90 =λ2, then four adjacent pixels are used in the horizontal and vertical directions to apply bicubic spline interpolation to the pixel, and λ2 represents setting the second threshold.

[0028] Furthermore, BM3D denoising is performed on the polarization image.

[0029] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0030] 1. Before applying the interpolation algorithm to the polarization image, the image is pre-processed for denoising to avoid the degradation of image quality caused by noise, which results in the inability to obtain accurate polarization information when applying the subsequent interpolation algorithm;

[0031] 2. Design edge perception weights for images, and then combine them with the weighted guided filtering of the LOG operator to process the image, which not only ensures the resolution of the image but also highlights the edge details;

[0032] 3. Based on the calculated intensity errors in the diagonal, horizontal and vertical directions, the specific method of interpolating image pixels is determined, which fully considers the relationship between the intensity correlation between image pixels and the relationship between flat areas and edge areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the edge-aware polarization image interpolation method;

[0034] Figure 2 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0036] If the image noise is not considered before implementing the respective interpolation algorithms, the reconstructed polarization image will be severely degraded. Since the presence of noise will seriously reduce the quality of the image and lead to the reconstruction of false polarization information, the image collected by the DoFP polarimeter must be denoised before the subsequent interpolation. Traditional grayscale domain and color domain denoising methods can be extended to DoFP images. These traditional methods include average filtering, median filtering, Wiener filtering, and wavelet threshold denoising. However, due to the inherent physical properties of the DoFP polarimeter, it is unrealistic to directly apply the grayscale denoising strategy to the images taken by the DoFP polarization image sensor. Therefore, the BM3D (Block-Matching and 3D filtering) denoising algorithm is used in this invention to preprocess the DoFP image first to obtain the denoised image I d .

[0037] The DoFP imaging sensor captures the intensity and polarization information of the scene, samples the scene through 0°, 45°, 90° and 135° polarization filters, and aligns the four sub-sampled images. The image intensity and polarization are then calculated using linear polarization filters at 0°, 45°, 90° and 135°. The Stokes vector is composed of four parameters: S0, S1, S2 and S3. Its four dimensions are the average values ​​of the light wave intensity, and three of the parameters are independent of each other. The standard method of representing light polarization is to use the first three Stokes parameters to describe the polarization characteristics of light. The formula is as follows:

[0038]

[0039] According to the Stokes formula, for the horizontal area, the image I is estimated according to formulas (2) to (5): d The intensity value of pixel (i,j) in:

[0040]

[0041]

[0042]

[0043]

[0044] in Representative use I 0° and I 90° Estimated strength, Representative use I 45° and I 135° The estimated intensity, I0(i,j) represents I 0° The pixel value of pixel (i, j) in 90(i-1,j) and I 90 (i+1,j) represent I 90° The pixel values ​​of pixels (i-1, j) and (i+1, j) in I 135 (i,j-1) represents I 135° The pixel value of pixel (i, j-1) in 45 (i-1,j-1) and I 45 (i+1,j-1) represent I 90° The pixel values ​​of pixels (i-1, j-1) and (i+1, j-1) in . and Represent the mean and difference of two estimates, respectively.

[0045] like Figure 1 As shown in the figure, in order to fully consider the edge information, we propose a new edge-aware weight, which effectively generates a guide map to filter the DoFP image by integrating a new edge detector. The image is processed by a weighted guided filtering method based on the LOG operator to ensure the resolution of the image and highlight the edge details. In addition, by combining pixel information from different regions to generate an effective guide wave image G, the absolute error of the subsequent calculation is reduced. For the denoised image I d The error is calculated to obtain image I h ,I v , to I h ,I v The guided wave image G is used for LOG operator weighted guided filtering, and the generated image I G The edge of the pixel has a low correlation with the neighborhood of the pixel, that is, it has a high intensity error. According to the above formula, we can define the pixel I G The absolute error ΔI(i,j) between the true value and the estimated value of (i,j).

[0046]

[0047] Since the polarization direction of each pixel is different and its neighborhood is 3×3 pixels, traditional edge detection methods cannot detect accurate edges for DoFP images. Fortunately, we can detect edges based on gradient correlation. Edges in an image are defined as where the spatial gradient exceeds a given threshold, usually identifying the area in the image where one object ends and another begins. These objects are usually different in the sensor parameters that produce them. For example, in a color sensor, there is a high gradient between two objects of different colors or intensities. In a polarization sensor, there is a high gradient between objects with different linear polarization angles or degrees. These gradients will mean that there are edges in the 4 synthetic images, namely in the 0°, 45°, 90° and 135° images. We extend the bicubic spline interpolation method to include gradient selective features. To this extent, the intensity error is calculated from the gradients of all four images in the vertical, horizontal and two diagonal directions near the low-resolution pixel, and the interpolation is performed based on the error. The intensity error formula is as follows:

[0048]

[0049]

[0050] According to the error of the gradient value in the 45-degree and 135-degree directions, if the value in the 45-degree direction is equal to the value in the 135-degree direction, the intensity value of the target pixel is the average of the intensity values ​​in the diagonal direction. In addition, single-direction interpolation is performed in the 45-degree and 135-degree directions. For example, if the error value of the pixel in the 90° polarization image in the 45-degree direction is large, we can obtain the intensity measurement in the 0° direction by implementing bicubic spline interpolation on four adjacent pixels in the 135-degree direction. After the diagonal interpolation, the horizontal and vertical areas are interpolated using bicubic spline interpolation.

[0051] The edge-aware polarization image interpolation algorithm described in the present invention focuses on estimating the horizontal and vertical directions of the polarization image, and then averages them according to the weight of each direction to obtain the guide map G, and then processes the image in combination with the weighted guided filtering of the LOG operator to reduce the intensity error. Figure 2 As shown, the specific implementation includes the following steps:

[0052] Step 1: Perform BM3D denoising preprocessing on the image Img to obtain the processed image I d ;

[0053] Step 2: I d Up-sampling is performed to obtain polarization images I at four angles: 0°, 45°, 90°, and 135°. 0° ,I 45° ,I 90° and I 135° ;

[0054] Step 3: Obtain the mind map G by calculating the edge perception weight;

[0055] Step 3.1: Evaluate the intensity difference in the horizontal and vertical directions to determine the weight of the interpolation direction of the edge-aware guided image. is the mean of the pixel intensity estimates) normalized to the pixel value range and calculated as:

[0056] X k (i,j)=X k1 (i,j)-X k2 (i,j) (9)

[0057]

[0058] in represents the filtering operation, and k represents the horizontal (v) and vertical (h) directions. k is the filter kernel in the k direction, expressed as:

[0059]

[0060]

[0061] Step 3.2: Calculate the directional strength difference

[0062]

[0063]

[0064] Each filter H k It is expressed as:

[0065]

[0066]

[0067] Step 3.3: Calculate the horizontal and vertical weights W k (i,j):

[0068]

[0069] in:

[0070]

[0071]

[0072] M kis a Laplace operator of size 3*3, ε is a very small positive value, and |·| is an absolute value operator;

[0073] Step 3.4, X k (i, j) is weighted averaged by pixel to generate edge-aware map G:

[0074]

[0075] Step 4: interpolation method;

[0076] Step 4.1: Interpolate in the diagonal direction to obtain the missing pixel values ​​in the opposite polarization direction: Calculate the pixel error ΔD in the diagonal direction of all pixels in the entire image according to the intensity error formula 45 and ΔD 135 If ΔD 135 / ΔD 45 >=λ1,(λ1=1.2), then take the 4 adjacent pixels in the 135 direction for this pixel and do bicubic spline interpolation. If ΔD 135 / ΔD 45 <λ1, then the four adjacent pixels in the 45-degree direction of the pixel are interpolated by bicubic spline;

[0077] Step 4.2: Interpolation in the horizontal and vertical directions, first calculate the absolute error I h (i,j),I v (i,j):

[0078]

[0079]

[0080]

[0081] Use the map G generated in step 3.4 and the input image I h ,I v The output image ΔI is obtained by filtering with weighted guided filtering using the LOG operator. h , ΔI v ;

[0082] Step 4.3: Calculate the pixel errors ΔD0 and ΔD in the horizontal and vertical directions of all pixels in the entire image according to the intensity error formula. 90 If ΔD0 / ΔD 90 >λ2, (λ2=1.5), then the four adjacent pixels in the horizontal direction of the pixel are interpolated by bicubic spline. If ΔD0 / ΔD 90 <λ2, then the four adjacent pixels in the vertical direction of the pixel are interpolated by bicubic spline.90 =λ2, then bicubic spline interpolation is applied to the target pixel using four adjacent pixels in the horizontal and vertical directions respectively. Finally, four images with different polarization directions are obtained.

[0083] In order to more clearly demonstrate the advantages of the present invention, the comparison results between the present invention and other solutions are shown in Table 1, and the quantitative evaluation standard RMSE is used to evaluate five DoFP images.

[0084] Table 1 Comparison of the present invention and other methods in terms of RMSE

[0085] The present invention Based on intensity correlation Using edge-aware residual interpolation <![CDATA[I 0° ]]> 0.014654 3.98584 3.05672 <![CDATA[I 45° ]]> 0.01535 4.26544 3.438 <![CDATA[I 90° ]]> 0.014741 4.08778 3.14634 <![CDATA[I 135° ]]> 0.014491 3.97132 3.23152 <![CDATA[S0]]> 0.008127 2.18952 1.38528 DoLP 0.026952 0.02896 0.02966

[0086] It can be seen from Table 1 that the present invention is compared with other solutions in terms of RMSE. 0° ,I 45° ,I 90° ,I 135° The results of S0 and DoLP are obviously better than those of other schemes, and the RMSE result of the present invention can be reduced by up to 99%.

[0087] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technology can understand and think of any changes or substitutions within the technical scope disclosed by the present invention, which should be included in the scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A polarization image interpolation method based on edge perception, characterized in that: The specific steps are as follows: Step 1, denoising and upsampling the polarization image to obtain polarization images in four directions of 0°, 45°, 90° and 135°; Step 2: Calculate edge-aware weights and generate edge-aware guidance graphs; Step 3, interpolating the polarization images in the four directions in the diagonal, horizontal and vertical directions respectively to obtain four images with different polarization directions; The edge-aware guidance graph G in step 2 is: Among them, (i, j) represents pixels, v represents the horizontal direction, h represents the vertical direction, and X k (i,j)=X k1 (i,j)-X k2 (i,j), represents the filtering operation, I d represents the denoised polarization image, and ε represents a very small positive value; Step 3 is as follows: Step 3.1, diagonal direction Calculate the pixel error ΔD in the diagonal direction of all pixels 45 and ΔD 135 , if the ΔD corresponding to a certain pixel 135 / ΔD 45 ≥λ1, then the four adjacent pixels at 135° are interpolated with bicubic spline for the pixel point; otherwise, the four adjacent pixels at 45° are interpolated with bicubic spline for the pixel point; where λ1 represents the setting of the first threshold; Step 3.2: Horizontal and vertical directions Step 3.2.1: Based on the denoised polarization image I d , get I h ,I v : Among them, (i,j) represents a pixel, Step 3.2.2: Use the edge-aware guided graph and the weighted guided filtering method based on the LOG operator to filter the I h ,I v Filter to obtain horizontal and vertical images ΔI h , ΔI v ; Step 3.2.3: Calculate the horizontal and vertical pixel errors ΔD0 and ΔD 90 , if the ΔD0 / ΔD corresponding to a certain pixel 90 >λ2, then the four adjacent pixels in the horizontal direction of the pixel are interpolated by bicubic spline; if ΔD0 / ΔD 90 <λ2, then the four adjacent pixels in the vertical direction of the pixel are interpolated by bicubic spline; if ΔD0 / ΔD 90 =λ2, then bicubic spline interpolation is applied to the pixel using four adjacent pixels in the horizontal direction and the vertical direction respectively; wherein λ2 represents setting the second threshold.

2. The edge-aware polarization image interpolation method according to claim 1, characterized in that: Perform BM3D denoising on polarization images.

Images