High-quality multispectral demosaicing method based on residual guidance

Abstract

The invention discloses a high-quality multispectral demosaicing method based on residual error guidance, belongs to the technical field of image restoration, and aims to solve the problems that a large amount of high-frequency information is not effectively processed and RGB image chromaticity channel restoration is insufficient. The method comprises the following steps: step 1, designing nine-channel multispectral optical filter array arrangement; step 2, interpolating a dense sampling channel image; 3, preliminarily reconstructing a sparse sampling channel image; and 4, further restoring the sparse sampling channel image by using a residual interpolation method. According to the demosaicing method based on residual guidance, when pixels of other channels are reconstructed, interpolation is carried out on residual values, and the problems of key detail reservation, fuzziness and detail loss are emphasized. And the mutual influence between different spectrum channels is considered, so that more accurate color restoration is realized, and the visual perception and the practical application value of the image are improved. The reconstructed image is excellent in the aspect of recovering the chromaticity channel of the RGB image, and the real effect of the reconstructed image is closer to that of the original image.

Description

Technical Field

[0001] The invention relates to a high-quality multispectral de-mosaicing method based on residual guidance, belonging to the technical field of image restoration. Background Art

[0002] The residual-guided high-quality multispectral demosaicing method is an advanced image processing technique specially designed to deal with the mosaic effect in multispectral images and enhance the image clarity and color restoration.

[0003] By placing a filter array with a mosaic pattern in front of the image sensor, all spectral information of the object can be obtained after a single exposure. The image directly obtained from the filter array is called the original image, and the process of estimating the lost information of the remaining bands at each sampling point is called the multispectral demosaicing process. Commonly used multispectral demosaicing methods include interpolation methods, frequency domain methods, sparse representation-based methods, and deep learning-based methods.

[0004] See the journal "Proceedings of SPIE", the article published by Monno et al. is titled "Multispectral demosaicking using guided filter", which designs a five-band multispectral filter array (Multispectral Filter Array, MSFA) with a specific pattern. Based on the guided filter method (Guided Filter, GF), the authors use adaptive Gaussian upsampling to interpolate the G-band original image, and use it as a guided image to interpolate the remaining bands. This method reconstructs the image not smooth enough, and the high-frequency information is not effectively processed. The corresponding insertion error increases, which is not conducive to the restoration of the chromaticity channels of the red, green, and blue (RGB) images. Summary of the invention

[0005] In order to solve the problem that a large amount of high-frequency information is not effectively processed and the chromaticity channel of the RGB image is not sufficiently restored, the present invention proposes a high-quality multispectral demosaicing method based on residual guidance.

[0006] The solution to the technical problem of the present invention is:

[0007] A high-quality multispectral demosaicing method based on residual guidance, the method comprises the following steps:

[0008] Step 1: Design a nine-channel multispectral filter array arrangement: Design a nine-channel multispectral filter array arrangement based on a binary tree algorithm, which includes a dense sampling channel with a sampling rate of 1 / 2 and eight sparse sampling channels with a sampling rate of 1 / 16;

[0009] Step 2: Interpolate densely sampled channel images: Use low-cost edge-aware demosaicing technology to accurately estimate the interpolated densely sampled channel images by calculating gradients and using the weighted sum of the average values ​​of horizontal and vertical neighboring pixels;

[0010] Step 3: Preliminary reconstruction of the sparse sampling channel image: subtract the interpolated dense sampling channel image from the measured sparse sampling channel image to obtain a sparse color difference map, then restore the sparse color difference map using guided filtering to obtain a complete color difference map, and then add the complete color difference map to the interpolated dense sampling channel image to obtain a preliminary reconstruction result of the sparse sampling channel image;

[0011] Step 4: Use the residual interpolation method to further restore the sparsely sampled channel image: subtract the initially reconstructed sparsely sampled channel image from the measured sparsely sampled channel image to obtain a sparse residual map, and use a Gaussian low-pass filter to interpolate the residual to obtain a complete residual map, and add the complete residual map to the initially reconstructed sparsely sampled channel image to obtain the final restored sparsely sampled channel image.

[0012] Beneficial effects of the present invention:

[0013] The present invention is based on a residual-guided demosaicing method. When reconstructing pixels of other channels, the residual values ​​are interpolated, focusing on retaining key details and avoiding the blurring and detail loss problems that may be caused by traditional methods. The mutual influence between different spectral channels is also considered to achieve more accurate color restoration and improve the visual experience and practical application value of the image. The experimental results are better than the GF demosaicing method in all three indicators, effectively improving the smoothness of the guided image and reducing the insertion error caused by high-frequency information, verifying the effectiveness and superiority of the method. In visual evaluation, the image reconstructed by the present invention performs well in restoring the chromaticity channels of the red, green and blue RGB images, and is closer to the real effect of the original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG1 is a flow chart of a high-quality multispectral demosaicing method based on residual guidance according to the present invention;

[0015] Figure 2 (a) is a binary tree split graph. Figure 2 (b) is the nine-channel MSFA map;

[0016] FIG3 : A schematic diagram of the arrangement of known pixels and pixels to be estimated in the dense sampling channel of the present invention;

[0017] FIG4 : Schematic diagram of the single-band multispectral image reconstruction process of the present invention;

[0018] Figure 5: Comparison of the two algorithms in reconstructing images on the dataset. DETAILED DESCRIPTION

[0019] The present invention is described in detail below with reference to the accompanying drawings.

[0020] like Figure 1 As shown, a high-quality multispectral demosaicing method based on residual guidance includes the following steps:

[0021] Step 1: Design a nine-channel multispectral filter array arrangement;

[0022] The nine-channel multispectral filter array is designed based on the binary tree algorithm, which includes a sampling rate of densely sampled channels and eight The sampling rate is sparsely sampled channels to meet the requirements of spectral consistency, spatial uniformity and periodicity.

[0023] like Figure 2 As shown in (a), the spatial sampling rate of the root node of the binary tree is 1. First, binary splitting starts from the root node. In each splitting process, the sampling rate of the child channel is half of the parent channel, that is, the spatial sampling rate of the spectral band at the nth level node is The resulting nine-channel multispectral filter array is Figure 2 As shown in (b), the densely sampled channels are on the first-level nodes with a sampling rate of 1 / 2, and the remaining eight sparsely sampled channels are on the fourth-level nodes with a sampling rate of 1 / 16.

[0024] Step 2, interpolating densely sampled channel images;

[0025] A low-cost edge-aware demosaicing technique is used to accurately estimate the interpolated densely sampled channel image by calculating gradients and utilizing the weighted sum of the horizontal and vertical neighboring pixel averages, optimizing image details and reducing interpolation errors.

[0026] A low-cost edge-aware demosaicing method is used to interpolate dense sampling channels containing rich texture information of the target. For each pixel to be estimated in the dense channel, a pixel with a size of The local neighborhood window of Figure 3 As shown, let the four neighboring pixels above, below, left and right of the center of the local window be known pixels:

[0027] (1)

[0028] in, and Represents the pixel to be estimated Coordinate location.

[0029] First, calculate the pixels The horizontal and vertical gradients are calculated as follows:

[0030] (2)

[0031] in, represents the horizontal gradient, Represents the vertical gradient.

[0032] Then, let and are the average values ​​of adjacent pixels in the horizontal and vertical directions, namely:

[0033] (3)

[0034] Finally, the estimated value It can be expressed as the following weighted sum form:

[0035] (4)

[0036] in,

[0037] (5)

[0038] Step 3: Preliminary reconstruction of sparsely sampled channel images;

[0039] The interpolated densely sampled channel image is subtracted from the measured sparsely sampled channel image to obtain a sparse color difference map, and then the sparse color difference map is restored using guided filtering to obtain a complete color difference map. The complete color difference map is then added to the interpolated densely sampled channel image to obtain a preliminary reconstruction result of the sparsely sampled channel image.

[0040] like Figure 4 In the workflow shown in Figure 1, the interpolated densely sampled channel image (denoted by G) is used as a guide image to reconstruct the sparsely sampled channel image (denoted by Y) with mosaic input. The local window at the center In, the estimation of sparsely sampled channel images is a local linear transformation of the guidance image, namely:

[0041] (6)

[0042] in, It is a window Any pixel within . It is a window A set of linear coefficients, which are assumed to be constant, can be determined by minimizing the following cost function:

[0043] (7)

[0044] is a binary mask, is the smoothing factor.

[0045] Formula (7) is a linear ridge regression. Two coefficients can be calculated through linear regression:

[0046] (8)

[0047] in, and Is the guide image G window Intrinsic pixel mean and variance, Display Window The number of pixels inside. Indicates the output window The mean value of the pixel value of the internal sampling point. The output at is expressed as a weighted average:

[0048] (9)

[0049] Among them, the window In pixels Central, It is a window Any pixel within In pixels Centered window A set of linear coefficients. Therefore, the formula can be rewritten as:

[0050] (10)

[0051] in, and Indicates that all the points that contain this Window of The coefficient average is:

[0052] (11)

[0053] Step 4, using the residual interpolation method to further restore the sparsely sampled channel image;

[0054] The sparse residual image is obtained by subtracting the initially reconstructed sparse sampling channel image from the measured sparse sampling channel image, and the residual is interpolated using a Gaussian low-pass filter to obtain a complete residual image. The complete residual image is added to the initially reconstructed sparse sampling channel image to obtain the final restored sparse sampling channel image.

[0055] The residual interpolation generated by the color difference interpolation is used to further enhance the sparsely sampled channel image initially reconstructed by the guided filter. The advantage of the residual interpolation algorithm is that the residual is smoother than the color difference, so a higher accuracy is obtained. Figure 4 As shown, the sparsely sampled channel image is reconstructed using residual interpolation It is divided into the following steps:

[0056] First, calculate the guided filter unsampled estimate Compared with the actual measured value Perform subtraction, defined as the residual .

[0057] Then, the residual is interpolated using Gaussian low-pass filtering.

[0058] (12)

[0059] in, is the spatial domain representation of the Gaussian low-pass filter, and are the row and column coordinates in the spatial domain, respectively; It is the standard deviation of the Gaussian function, which determines the width of the filter, that is, the filter's attenuation rate of the frequency. The larger the standard deviation, the wider the filter, the more low-frequency components are retained, and the less high-frequency components are suppressed.

[0060] For a given residual image , which is the result after Gaussian low-pass filtering It can be expressed as:

[0061] (13)

[0062] in, Represents a convolution operation.

[0063] Finally, the initially reconstructed sparsely sampled channel image is added to the interpolated residual to obtain the final restored sparsely sampled channel image .

[0064] The present invention uses multiple indicators to evaluate the image demosaicing effect:

[0065] Three main image quality evaluation indicators are used: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM) and Spectral Angle Mapper (SAM) to objectively evaluate the effect of the demosaicing algorithm.

[0066] The three evaluation indicators are calculated in the standard red green blue (sRGB) color space to quantitatively analyze the similarity between the reconstructed image and the original image. PSNR is measured by calculating the difference between the corresponding pixels of the real image and the reconstructed image and taking the average; while SSIM and SAM evaluate the similarity between images from the structural and spectral perspectives respectively. The higher the values ​​of these indicators, the better the reconstruction effect of the demosaicing algorithm. Corresponding to each color channel, the calculation method of PSNR is as follows:

[0067] (12)

[0068] In the formula, Indicates the maximum pixel intensity of the real image. For 8-bit image data, =255. Similar to PSNR, sRGB SSIM is also calculated for each color channel separately, and its calculation formula is:

[0069] (13)

[0070] The image and images The mean values ​​of , , and their variances are , , the covariance is

[0071] . and The image dynamic range Determined constants:

[0072] (14)

[0073] In the formula and Usually the default values ​​are 0.01 and 0.03 respectively.

[0074] SAM is tested by With reference spectrum Determine their similarity between:

[0075] (15)

[0076] In the formula, =Number of frequency bands.

[0077] Example:

[0078] Matlab2023b software was used to simulate and analyze the two algorithms on the CAVE dataset, and the algorithms were compared from the two aspects of quality evaluation indicators of reconstructed images and visual evaluation indicators. The quality evaluation indicators include PSNR, SSIM, and SAM. As shown in Table 1, the PSNR and SSIM of this method are improved by 0.73 dB and 0.0092 compared with the GF method, and the test spectrum of the image reconstructed by this method on the SAM indicator is also shown. With reference spectrum The similarity between them also increased by 0.009.

[0079] Table 1

[0080] PSNR↑ / dB PSNR↑ / dB SSIM↑ / dB SSIM↑ / dB SAM↓ / dB SAM↓ / dB CAVE GF Pro GF Pro GF Pro Balloons 41.94 44.08 0.9910 0.9941 0.052 0.049 Beads 25.15 26.99 0.8673 0.9362 0.202 0.143 CD 34.98 37.47 0.9744 0.9794 0.095 0.099 Beers 39.11 42.26 0.9883 0.9902 0.031 0.031 Clay 35.67 37.19 0.9786 0.9874 0.118 0.114

[0081] The above results show that the demosaiced image quality of the present invention on the two datasets is generally better than that of the GF method. Figure 5 As shown, a visual comparison of the demosaicing error maps of the test images in the range of 0 to 50 nm by different methods is shown. Through the comparison, it can be clearly seen that the present invention shows significant advantages in demosaicing error, and the details and accuracy of the restored images are better than the GF method, which verifies the superior performance in processing spectral information.

Claims

1. A high-quality multispectral demosaicing method based on residual guidance, characterized by: The method comprises the following steps: Step 1: Design a nine-channel multispectral filter array arrangement: Design a nine-channel multispectral filter array arrangement based on a binary tree algorithm, which includes a sampling rate of densely sampled channels and eight Sparsely sampled channels with sampling rate; Step 2: Interpolate densely sampled channel images: Use low-cost edge-aware demosaicing technology to accurately estimate the interpolated densely sampled channel images by calculating gradients and using the weighted sum of the average values ​​of horizontal and vertical neighboring pixels; Step 3: Preliminary reconstruction of the sparse sampling channel image: subtract the interpolated dense sampling channel image from the measured sparse sampling channel image to obtain a sparse color difference map, then restore the sparse color difference map using guided filtering to obtain a complete color difference map, and then add the complete color difference map to the interpolated dense sampling channel image to obtain a preliminary reconstruction result of the sparse sampling channel image; Step 4: Use the residual interpolation method to further restore the sparsely sampled channel image: subtract the initially reconstructed sparsely sampled channel image from the measured sparsely sampled channel image to obtain a sparse residual map, and use a Gaussian low-pass filter to interpolate the residual to obtain a complete residual map, and add the complete residual map to the initially reconstructed sparsely sampled channel image to obtain the final restored sparsely sampled channel image.

2. A high-quality multispectral demosaicing method based on residual guidance according to claim 1, characterized in that: The specific steps of step 2 of interpolating densely sampled channel images are as follows: A low-cost edge-aware demosaicing method is used to interpolate dense sampling channels containing rich texture information of the target. For each pixel to be estimated in the dense channel, a pixel with a size of The local neighborhood window is , and the four neighboring pixels above, below, left and right of the center of the local window are set as known pixels. (1) in, and Represents the pixel to be estimated Coordinate location; First, calculate the pixels The horizontal and vertical gradients are calculated as follows: (2) in, represents the horizontal gradient, represents the vertical gradient; Then, let and are the average values ​​of adjacent pixel values ​​in the horizontal and vertical directions, namely: (3) Finally, the estimated value It can be expressed as the following weighted sum form: (4) in, (5)。 3. The high-quality multispectral demosaicing method based on residual guidance according to claim 1, characterized in that: The specific steps of step 3 to preliminarily reconstruct the sparse sampling channel image are: The interpolated densely sampled channel image G is used as a guide image to reconstruct the sparsely sampled channel image Y with mosaic input; according to GIF theory, the pixel The local window at the center In, the estimation of sparsely sampled channel images is a local linear transformation of the guidance image, namely: (6) in, It is a window Any pixel within; It is a window A set of linear coefficients, which are assumed to be constant, can be determined by minimizing the following cost function: (7) is a binary mask, is the smoothing factor; Formula (7) is a linear ridge regression. Two coefficients can be calculated through linear regression: (8) in, and Is the guide image G window Intrinsic pixel mean and variance, Display Window The number of pixels inside. Indicates the output window The mean value of the pixel value at the internal sampling point; guide the filter at a certain pixel point The output at is expressed as a weighted average: (9) Among them, the window In pixels Central, It is a window Any pixel within In pixels Centered window A set of linear coefficients; therefore, the formula can be rewritten as: (10) in, and Indicates that all the points that contain this Window of The average value of the coefficient is: (11)。 4. The high-quality multispectral demosaicing method based on residual guidance according to claim 1, characterized in that: Step 4: Reconstruct the sparsely sampled channel image using residual interpolation It is divided into the following steps: First, calculate the guided filter unsampled estimate Compared with the actual measured value Perform subtraction, defined as the residual ; Then, the residual is interpolated using Gaussian low-pass filtering. (12) in, is the spatial domain representation of the Gaussian low-pass filter, and are the row and column coordinates in the spatial domain, respectively; It is the standard deviation of the Gaussian function, which determines the width of the filter, that is, the attenuation rate of the filter to the frequency; For a given residual image , which is the result after Gaussian low-pass filtering It can be expressed as: (13) in, Represents the convolution operation; Finally, the initially reconstructed sparsely sampled channel image is added to the interpolated residual to obtain the final restored sparsely sampled channel image .

Images