Focal plane polarization demosaicking method based on local gradient and channel correlation

Abstract

The disclosed focal plane polarization demosaicking method based on local gradient and channel correlation belongs to the field of polarization imaging and image processing. The method comprises the following steps: obtaining a sparse image by downsampling a polarization mosaic image; calculating the local gradient of a current pixel by using the pixels of other adjacent polarization directions, and designing a smoothness weight to optimize the bilinear interpolation process, thereby generating an initial demosaicked image; calculating the relationship between the minimum cost function and the normalized cross-correlation coefficient by using the normalized cross-correlation coefficient and guided filtering, and designing a polarization channel correlation weight with symmetry characteristics; and applying the weight in the polarization channel difference model to further optimize the demosaicking result. The method can more accurately reflect the distribution characteristics of intensity, linear polarization degree and polarization angle in actual application scenarios, improve the polarization demosaicking effect, and reduce the instantaneous field of view error of the focal plane polarization mosaic image.

Description

TECHNICAL FIELD

[0001] The application relates to a focal plane polarization demosaicking method, in particular to a focal plane polarization demosaicking method based on local gradient and channel correlation, and belongs to the field of polarization imaging and image processing. BACKGROUND

[0002] Polarization is one of the main physical characteristics of electromagnetic waves. The introduction of polarization information in conventional imaging technology can effectively broaden the dimension of acquired information. Polarization imaging technology has broad application prospects in the fields of military security, biomedicine, material science, environmental monitoring, etc., and provides important technical support for target detection and identification, disease diagnosis and drug research, material characterization and defect detection, etc.

[0003] The focal plane polarization imaging system integrates a micro-polarization plate array with a super-pixel structure on the surface of the detector, simultaneously acquires the radiation of the scene in different polarization directions, and has no moving parts, compact structure and firm reliability, and has become a research hotspot in the field of polarization imaging technology in recent years. However, since each pixel in the super-pixel structure can only acquire the radiation in a single polarization direction, the original image acquired by the focal plane polarization imaging system is a mosaic image, and there is a instantaneous field of view error, which affects the reconstruction of the polarization image and the calculation of the polarization information.

[0004] Although the current polarization image demosaicking method has continuously improved accuracy, the calculation cost also increases, and complex polarization image demosaicking processing requires expensive hardware and a long processing time, which is contrary to the original intention of using the focal plane polarization imaging system. The method based on polarization channel correlation has attracted more and more attention due to its interpolation accuracy and processing efficiency, but this kind of method usually relies on some prior knowledge, but these prior knowledge is often difficult to obtain or lacks universality. Therefore, how to reasonably use the polarization channel correlation and design a polarization demosaicking method with universality is a key problem worthy of study. SUMMARY

[0005] The current focal plane polarization demosaicking method based on polarization channel correlation generally assumes that the intensity, linear polarization degree and polarization angle of the scene are uniformly distributed, and ignores the complexity of the scene, so that the polarization demosaicking effect is poor in actual application. The purpose of the present application is to provide a focal plane polarization demosaicking method based on local gradient and channel correlation, which uses local gradient to optimize bilinear interpolation to generate an initial demosaicked image, designs a polarization channel correlation weight with symmetry characteristics through a normalized cross-correlation coefficient and a guided filter, and applies the weight in a polarization channel difference model to further optimize the demosaicking result and reduce the instantaneous field of view error of the focal plane polarization mosaic image.

[0006] The purpose of the present application is achieved by the following technical solutions.

[0007] The disclosed focal plane polarization demosaicking method based on local gradient and channel correlation calculates the local gradient of the current pixel using the pixels of adjacent other polarization directions, and designs a smoothness weight to optimize the bilinear interpolation process, thereby generating an initial demosaicked image. By normalizing the cross-correlation coefficient and guided filtering, the relationship between the minimum cost function and the normalized cross-correlation coefficient is calculated, and a polarization channel correlation weight with symmetry characteristics is designed. The weight is applied in the polarization channel difference model to further optimize the demosaicking result. Compared with the existing focal plane polarization demosaicking method based on polarization channel correlation, the application can more accurately reflect the distribution characteristics of intensity, linear polarization degree and polarization angle in actual application scenarios, improve the polarization demosaicking effect, and reduce the instantaneous field of view error of the focal plane polarization mosaic image.

[0008] The disclosed focal plane polarization demosaicking method based on local gradient and channel correlation comprises the following steps:

[0009] Step 1, for the polarization mosaic image, a sparse image is obtained by downsampling.

[0010] For the polarization mosaic image, a sparse image is obtained by downsampling according to formula (1).

[0011] I α =I⊙M α (1)

[0012] Wherein, I represents the original mosaic image, I α represents the sparse image of the alpha polarization channel, M α represents the binary downsampling matrix, and represents the Hadamard product.

[0013]

[0014] Step 2, for the polarization mosaic image, the local gradient of the current pixel is calculated using the pixels of adjacent other polarization directions, and a smoothness weight is designed to optimize the bilinear interpolation process, thereby generating an initial demosaicked image.

[0015] Step 2.1, the gradients of the current pixel in the horizontal, vertical and diagonal directions are calculated using the pixels of adjacent other polarization directions according to formula (3), and linear weighting is performed to determine the local gradient of the current pixel.

[0016]

[0017] Wherein, LG(i,j) represents the local gradient at (i,j).

[0018] Step 2.2. Design the smoothness weight to optimize the bilinear interpolation, to avoid the blur phenomenon generated by the bilinear interpolation method in the edge and detail area.

[0019]

[0020] where W LG (i,j) represents the smoothness weight at (i,j), ranging from [0,1], LG max represents the maximum gradient in the image.

[0021] Step 2.3. After obtaining the smoothness weight, optimize the bilinear interpolation.

[0022]

[0023] where, represents the initial demosaicing image of the alpha polarization channel generated by the local gradient method, F represents the bilinear interpolation filter, represents the convolution operation.

[0024] Step 3. According to the initial demosaicing image obtained in step 2, calculate the normalized cross-correlation coefficient of different polarization channels, use guided filtering to calculate the relationship between the minimum cost function and the normalized cross-correlation coefficient, and design polarization channel correlation weight with symmetry characteristics.

[0025] Step 3.1. There is correlation between different channels of the polarization image, which is used to further optimize the demosaicing result. According to formula (6), the normalized cross-correlation processing is performed on the initial demosaicing images of different polarization channels, to obtain the normalized cross-correlation coefficient measuring the similarity of two images.

[0026]

[0027] where, represents the normalized cross-correlation coefficient measuring the similarity of the initial demosaicing images of the alpha and beta polarization channels, ranging from [-1,1], -1 represents complete negative correlation, 1 represents complete positive correlation, 0 represents no linear correlation, and the larger the absolute value is, the higher the similarity is. and respectively represent the average intensity value of the initial demosaicing images of the alpha and beta polarization channels, represents the covariance of the initial demosaicing images of the alpha and beta polarization channels, and respectively represent the standard deviation of the initial demosaicing images of the alpha and beta polarization channels.

[0028] Step 3.2. In the guided filtering, the input image I inthe pixel (i, j) of the output image I out is a linear transformation of the guide image I guide , then

[0029]

[0030] where a(i, j) and b(i, j) are linear coefficients within the filter window ω, and represent the mean of the input image I out and the guide image I guide within the filter window ω, respectively.

[0031] a(i, j) and b(i, j) within each filter window are obtained by minimizing the cost function E(a(i, j), b(i, j)) according to equation (8).

[0032]

[0033] where λ is a regularization term to prevent a(i, j) from being too large, and is also a smoothing threshold of the guide filter. The purpose of using the guide filter in polarization image demosaicking is to obtain more accurate interpolation results without smoothing the input image, so λ is 0.

[0034] By linearly solving equation (8), the closed-form solutions of a(i, j) and b(i, j) are obtained.

[0035]

[0036] where represent the mean of the input image I in and the guide image I guide within the filter window ω, respectively. represent the variance of the guide image I guide within the filter window ω. and represent the mean of the input image I in and the guide image I guide within the filter window ω, respectively.

[0037] By substituting equations (9) and (10) into equation (8), the minimum cost function and within the filter window ω.

[0038]

[0039] where represent the variance of the input image I in within the filter window ω.

[0040] the initial demosaiced images of the alpha and beta polarization channels and the minimum cost function and the relationship within the filter window ω is

[0041]

[0042] Step 3.3, the similarity measure of the initial demosaiced images of the alpha and beta polarization channels within the filter window ω is defined as

[0043]

[0044] Step 3.4, the alpha and beta polarization channel correlation weight is defined as

[0045]

[0046] Step 4, using the initial demosaiced images obtained in Step 2, the difference images of different polarization channels are calculated.

[0047] using the initial demosaiced images obtained in Step 2 the difference image Δ of the alpha and beta polarization channel images at the alpha polarization channel sampling point is calculated according to formula (15) α,β .

[0048]

[0049] Step 5, after obtaining the polarization channel difference image according to Step 4, the bilinear interpolation is optimized according to the local gradient method of Step 2 to obtain the polarization channel difference estimation image.

[0050] using the difference images Δ of the alpha and beta polarization channels obtained in Step 4 α,β , the bilinear interpolation process is optimized according to the local gradient method of formula (3)-(5) and the smoothness weight of Step 2 to obtain the difference estimation images of the alpha and beta polarization channels

[0051] Step 6, the initial demosaiced images obtained in Step 2 and the difference estimation images of different polarization channels obtained in Step 5 are added to obtain the polarization channel estimation image.

[0052] the initial demosaiced image of the beta polarization channel obtained in Step 2 and the difference estimation images of the alpha and beta polarization channels obtained in Step 5​​​ add, to obtain the alpha polarization channel image estimated by the beta polarization channel

[0053]

[0054] Step 7, using the polarization channel correlation weight obtained in step 3, the polarization channel estimation image obtained in step 6 is weighted and averaged to obtain the final demosaicing image.

[0055] According to formula (17), using the correlation weight of alpha and beta polarization channels obtained in step 3 the alpha polarization channel image estimated by the beta polarization channel obtained in step 6 weighted and averaged, further optimize the demosaicing result, reduce the instantaneous field error, and obtain the final demosaicing image I α .

[0056]

[0057] Beneficial effects:

[0058] 1. The existing focal plane polarization demosaicing method based on polarization channel correlation generally assumes that the intensity, linear polarization degree and polarization angle of the scene are uniformly distributed, and ignores the complexity of the scene, so that the polarization demosaicing effect in actual application is poor. The focal plane polarization demosaicing method disclosed in the present application is based on local gradient and channel correlation, which uses local gradient to optimize bilinear interpolation to generate an initial demosaicing image; the polarization channel correlation weight with symmetry characteristics is designed by normalizing the cross-correlation coefficient and guided filtering; the weight is introduced into the polarization channel difference model to further optimize the demosaicing result and reduce the instantaneous field error of the focal plane polarization demosaicing image.

[0059] 2. The focal plane polarization demosaicing method disclosed in the present application is based on local gradient and channel correlation, which calculates the normalized cross-correlation coefficient of different polarization channels according to the obtained initial demosaicing image; the relationship between the minimum cost function and the normalized cross-correlation coefficient is calculated by guided filtering; the polarization channel correlation weight with symmetry characteristics is designed; the weight is introduced into the polarization channel difference model to further optimize the demosaicing result and reduce the instantaneous field error of the focal plane polarization demosaicing image.

[0060] 3. The focal plane polarization demosaicing method disclosed in the present application is based on local gradient and channel correlation, after obtaining sparse images of different polarization channels, the local gradient of the current pixel is calculated by using pixels of adjacent other polarization directions; the bilinear interpolation process is optimized by designing the smoothness weight, so as to generate an initial demosaicing image and improve the initial demosaicing effect. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 Flow chart of the focal plane polarization demosaicking method based on local gradient and channel correlation of the present application.

[0062] Figure 2 Polarization demosaicking result of a real scene image of the present application.

[0063] Wherein: 1-raw polarization mosaic image, 2-single polarization channel sparse image, 3-initial demosaicking image, 4-polarization channel correlation weight, 5-smoothness weight, 6-0° polarization channel sparse image, 7-polarization channel difference image, 8-polarization channel difference estimation image, 9-estimated 0° polarization channel image of other polarization channels, 10-final 0° polarization channel demosaicking image, 11-focal plane polarization mosaic image of a real scene, 12-demosaicking image of four polarization channels, 13-Stokes vector image, 14-linear polarization degree and polarization angle image. DETAILED DESCRIPTION

[0064] In order to better illustrate the purpose and advantages of the present application, the content of the application is further illustrated below in combination with the drawings and examples.

[0065] Example 1:

[0066] As shown in the figure, the focal plane polarization demosaicking method based on local gradient and channel correlation of the present embodiment has the following specific implementation steps: Figure 1 Step 1: Taking a four-channel (0°, 45°, 90° and 135°) polarization mosaic image as an example, for the raw polarization mosaic image 1, according to formulas (1) and (2), a single polarization channel sparse image 2 is obtained by downsampling.

[0067] Step 2: For the raw polarization mosaic image 1, according to formulas (3)-(5), the local gradient of the current pixel is calculated using the pixels of adjacent other polarization directions, and the smoothness weight 5 is designed to optimize the bilinear interpolation process, and the initial demosaicking image 3 is obtained.

[0068] Step 3: Using the initial demosaicking image obtained in step 2, according to formulas (6)-(14), the normalized cross-correlation coefficients of different polarization channels are calculated, the relationship between the minimum cost function and the normalized cross-correlation coefficients is calculated using guided filtering, and the polarization channel correlation weight 4 with symmetry characteristics is designed.

[0069] Step 4: Taking the 0° polarization channel sparse image 6 as an example, using the initial demosaicking image 3 obtained in step 2, according to formula (15), the difference image 7 of different polarization channels is calculated.

[0070] Step 4: Taking the 0° polarization channel sparse image 6 as an example, using the initial demosaicking image 3 obtained in step 2, according to formula (15), the difference image 7 of different polarization channels is calculated.

[0071] Step 5, using the polarization channel difference image 7 obtained in step 4, the bilinear interpolation is optimized according to the local gradient method of step 2, to obtain the polarization channel difference estimation image 8.

[0072] Step 6, according to formula (16), the initial demosaicing image 3 obtained in step 2 and the difference estimation image 8 of different polarization channels obtained in step 5 are added to obtain the 0° polarization channel image 9 estimated by other polarization channels (45°, 90° and 135°).

[0073] Step 7, using the polarization channel correlation weight 4 obtained in step 3, according to formula (17), the polarization channel estimation image 9 obtained in step 6 is weighted and averaged to obtain the final demosaicing image 10.

[0074] As shown in Figure 2 , it is a polarization demosaicing result of a practical scene image disclosed in the embodiment. A four-polarization direction pixel interval undersampled polarization mosaic image 11 is obtained by a focal plane polarization imaging system; since a single pixel only responds to the intensity of the current polarization direction, four interpolated single-channel polarization images 12 need to be generated by a demosaicing method; the Stokes vector image 13 and the linear polarization degree and polarization angle image 14 of the target scene are calculated. The polarization mosaic image acquisition device is a PHX050S visible light focal plane polarization imaging system of Canada LUCID company, the resolution is 2448×2048, the image bit number is 8bit, and the polarization direction is 0°, 45°, 90° and 135°. 100 images of the same static scene are continuously collected and averaged to reduce the influence of time noise.

[0075] The focal plane polarization demosaicing method based on local gradient and channel correlation disclosed in the embodiment is used in the process of generating four polarization channel demosaicing images 12 from the original polarization mosaic image 11. After obtaining the four polarization channel demosaicing images 12, the Stokes vector image 13, the linear polarization degree and the polarization angle image 14 of the scene are calculated according to formula (18) and (19).

[0076] S0=(I0+I 45 +I 90 +I 135 ) / 2(18a)

[0077] S1=I0-I 90 (18b)

[0078] S2=I 45 -I 135 (18c)

[0079] Wherein, I0, I 45 , I 90 and I135 The radiation intensity images representing the four polarization directions (0°, 45°, 90° and 135°) respectively. S0 represents the total radiation intensity of the scene, S1 represents the horizontal or vertical linear polarization component, and S2 represents the ±45° linear polarization component. Since the circular polarization component is extremely rare in nature, the fourth component of the Stokes vector is usually 0.

[0080]

[0081] where DoLP and AoP represent the linear polarization degree and the polarization angle images respectively.

[0082] Since S1 and S2 are obtained by differentiating the images of different polarization directions, the accuracy of the demosaicing method directly affects the calculation of the linear polarization degree and the polarization angle. Therefore, the key to the focal plane polarization image processing lies in the effectiveness of the demosaicing effect. By demosaicing the focal plane polarization image, the instantaneous field error of the focal plane polarization mosaic image is reduced.

[0083] The above specific description further details the purpose, technical solutions and beneficial effects of the application. It should be understood that the above description is only a specific embodiment of the application and is not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A focal plane polarization demosaic method based on local gradient and channel correlation, characterized in that: Includes the following steps, Step 1: For polarized mosaic images, obtain sparse images through downsampling; Step 2: For the polarized mosaic image, calculate the local gradient of the current pixel using pixels with other adjacent polarization directions, and design smoothness weights to optimize the bilinear interpolation process, thereby generating the initial demosaic image. Step 2 is implemented as follows: Step 2.1: Calculate the gradient of the current pixel in the horizontal, vertical and diagonal directions using the pixels of adjacent other polarization directions according to equation (3), and perform linear weighting to determine the local gradient of the current pixel. Where LG(i,j) represents the local gradient at (i,j); Step 2.2: Design smoothness weights to optimize bilinear interpolation and avoid the blurring phenomenon produced by the bilinear interpolation method in edge and detail areas; Among them, W LG (i,j) represents the smoothness weight at (i,j), ranging from [0,1]. LG max This represents the maximum gradient in the image; Step 2.3: After obtaining the smoothness weights, optimize the bilinear interpolation; in, This represents the initial demosaic image of the α-polarization channel generated using the local gradient method, where F represents the bilinear interpolation filter. This represents the convolution operation; Step 3: Based on the initial demosaic image obtained in Step 2, calculate the normalized cross-correlation coefficients of different polarization channels, use guided filtering to calculate the relationship between minimizing the cost function and the normalized cross-correlation coefficients, and design polarization channel correlation weights with symmetry characteristics. Step 3 is implemented as follows: Step 3.1: There is a correlation between different channels of the polarization image; according to Equation (6), the initial demosaic images of different polarization channels are subjected to normalized cross-correlation processing to obtain the normalized cross-correlation coefficient that measures the similarity between the two images; in, The normalized cross-correlation coefficient represents the similarity between the initial demosaiced images of the α and β polarization channels. It ranges from [-1, 1], where -1 indicates a perfect negative correlation, 1 indicates a perfect positive correlation, and 0 indicates no linear correlation. The larger the absolute value, the higher the similarity. and These represent the average intensity values ​​of the initial demosaic images for the α and β polarization channels, respectively. The covariance of the initial demosaic image for the α and β polarization channels is represented. and α and β represent the standard deviations of the initial demosaic images for the α and β polarization channels, respectively; Step 3.2: In guided filtering, use the input image I in Within a filtering window ω centered at pixel (i,j), the output image I out It is the guide image I guide A linear transformation of , then Where a(i,j) and b(i,j) are the linear coefficients within the filter window ω. and Representing the output image I out and guiding image I guide The intensity value within the filter window ω; According to equation (8), a(i,j) and b(i,j) within each filtering window are obtained by minimizing the cost function E(a(i,j),b(i,j)); Here, λ is a regularization term to prevent the solved a(i,j) from being too large, and it is also the smoothing threshold of the guided filter. The purpose of using guided filtering in polarization image demosaic is to obtain more accurate interpolation results without the need to smooth the input image, so λ is 0. By solving formula (8) linearly, closed-form solutions for a(i,j) and b(i,j) are obtained; in, Indicates input image I in and guiding image I guide Covariance within the filter window ω Indicates the guiding image I guide The variance within the filter window ω, and Representing the input image I in and guiding image I guide The mean value within the filter window ω; Substituting equations (9) and (10) into equation (8), we obtain the minimum cost function. and The relationship within the filter window ω; in, Indicates input image I in The variance within the filter window ω; Initial demosaic images of the α and β polarization channels and Minimum cost function, using the input and guide images as input and guide images. and The relationship within the filter window ω is as follows Step 3.3: Based on the symmetry of the similarity measure of the initial demosaic images of the α and β polarization channels, measure the similarity of the initial demosaic images of the α and β polarization channels within the filtering window ω. Defined as Step 3.4: Based on similarity measurement Weight the correlation between α and β polarization channels Defined as Step 4: Using the initial demosaiced image obtained in Step 2, calculate the difference images for different polarization channels; Step 5: After obtaining the polarization channel difference image according to Step 4, optimize the bilinear interpolation according to the local gradient method in Step 2 to obtain the polarization channel difference estimation image. Step 6: Add the initial demosaic image obtained in Step 2 and the differential estimation images of different polarization channels obtained in Step 5 to obtain the polarization channel estimation image. Step 7: Using the polarization channel correlation weights obtained in Step 3, perform a weighted average on the polarization channel estimation image obtained in Step 6 to obtain the final de-mosaic image.

2. The focal plane polarization demosaicing method based on local gradient and channel correlation as described in claim 1, characterized in that: Step 1 is implemented as follows: For polarized mosaic images, sparse images are obtained by downsampling according to equation (1); I α =I⊙M α (1) Where I represents the original mosaic image, I α M represents a sparse image of the α polarization channel. α Represents a binary downsampling matrix, and ⊙ represents the Hadamard product; 3. The focal plane polarization demosaic method based on local gradient and channel correlation as described in claim 1, characterized in that: Step 4 is implemented as follows: The initial depixelated image is obtained using step 2. Then, according to equation (15), the difference image Δ between the α and β polarization channel images at the sampling point of the α polarization channel is calculated. α,β ; 4. The focal plane polarization demosaic method based on local gradient and channel correlation as described in claim 1, characterized in that: Step 5 is implemented as follows: The differential image Δ of the α and β polarization channels is obtained using step 4. α,β Then, following the local gradient method and smoothness weight optimization of the bilinear interpolation process in steps 2 (3)-(5), the difference estimation images of the α and β polarization channels are obtained.

5. The focal plane polarization demosaic method based on local gradient and channel correlation as described in claim 1, characterized in that: Step 6 is implemented as follows: The initial demosaic image of the β polarization channel obtained in step 2 is obtained according to equation (16). The difference estimation images of the α and β polarization channels obtained in step 5 Add them together to obtain the α-polarization channel image estimated using the β-polarization channel.

6. The focal plane polarization demosaicing method based on local gradient and channel correlation as described in claim 1, characterized in that: Step 7 is implemented as follows: According to equation (17), the correlation weights of the α and β polarization channels obtained in step 3 are used. The α-polarization channel image estimated from the β-polarization channel obtained in step 6 A weighted average is performed to reduce instantaneous field-of-view error, resulting in the final de-mosaic image I. α ;

Images