A High-Quality Multispectral Demosaicking Method Based on Residual Guidance

Abstract

A high-quality multi-spectral demosaicking method based on residual guidance, belonging to the field of image restoration technology. To solve the problems of ineffective processing of a large amount of high-frequency information and insufficient restoration of the chrominance channels of RGB images, the method includes the following steps: Step 1, design the arrangement of a nine-channel multi-spectral filter array; Step 2, interpolate and densely sample the channel images; Step 3, preliminarily reconstruct the sparsely sampled channel images; Step 4, use the residual interpolation method to further restore the sparsely sampled channel images. The demosaicking method based on residual guidance of the present invention interpolates the residual values when reconstructing the pixels of other channels, pays attention to retaining key details, and solves the problems of blurring and detail loss. It also considers the mutual influence between different spectral channels to achieve more accurate color restoration and improve the visual perception and practical application value of the images. The images reconstructed by the present invention perform excellently in restoring the chrominance channels of the red, green, and blue RGB images and are closer to the true effects of the original images.

Description

Technical Field

[0001] The present invention relates to a high-quality multi-spectral demosaicking method based on residual guidance, belonging to the technical field of image restoration. Background Art

[0002] The high-quality multi-spectral demosaicking method based on residual guidance is an advanced image processing technology, specifically designed to process the mosaic effect in multi-spectral images and enhance the clarity and color restoration of images.

[0003] By placing a filter array with a mosaic pattern in front of the image sensor, all spectral information of the object can be obtained through a single exposure. The image directly obtained from the filter array is called the original image, and the process of estimating the missing information in the remaining bands at each sampling point is called the multi-spectral demosaicking process. Common multi-spectral demosaicking methods include interpolation methods, frequency domain methods, sparse representation-based methods, and deep learning-based methods, etc.

[0004] Referring to the journal "Proceedings of SPIE", the article named "Multispectral demosaicking using guided filter" published by Monno et al. designed a specific pattern of five-band multi-spectral filter array (Multispectral Filter Array, MSFA). Based on the guided filter method (Guided Filter, GF), the author used adaptive Gaussian upsampling interpolation for the G-band original image and used it as the guided image to interpolate the remaining bands. The reconstructed image of this method is not smooth enough, the high-frequency information is not effectively processed, and the corresponding interpolation error becomes more, which is not conducive to the recovery of the chrominance channels of the Red Green Blue (RGB) image. Summary of the Invention

[0005] In order to solve the problems that a large amount of high-frequency information is not effectively processed and the recovery of the chrominance channels of the RGB image is insufficient, the present invention proposes a high-quality multi-spectral demosaicking method based on residual guidance.

[0006] The solution of the present invention to solve the technical problems is as follows:

[0007] A high-quality multi-spectral demosaicking method based on residual guidance, the method comprising the following steps:

[0008] Step 1, designing the arrangement of a nine-channel multi-spectral filter array: designing the arrangement of a nine-channel multi-spectral filter array based on the binary tree algorithm, which includes a dense sampling channel with a 1 / 2 sampling rate and eight sparse sampling channels with a 1 / 16 sampling rate;

[0009] Step 2, Interpolation Dense Sampling Channel Image: Adopt a low-cost edge-aware demosaicking technique to accurately estimate the interpolation dense sampling channel image by calculating gradients and using the weighted sum of the average values of horizontal and vertical neighboring pixels;

[0010] Step 3, Preliminary Reconstruction of Sparse Sampling Channel Image: Subtract the interpolated dense sampling channel image from the measured sparse sampling channel image to obtain a sparse color difference map, then use guided filtering to restore the sparse color difference map to obtain a complete color difference map, and then add the complete color difference map to the interpolated dense sampling channel image to obtain the preliminary reconstruction result of the sparse sampling channel image;

[0011] Step 4, Further Restoration of Sparse Sampling Channel Image Using Residual Interpolation Method: Subtract the preliminarily reconstructed sparse sampling channel image from the measured sparse sampling channel image to obtain a sparse residual map, and use a Gaussian low-pass filter to interpolate the residual to obtain a complete residual map, and then add the complete residual map to the preliminarily reconstructed sparse sampling channel image to obtain the finally restored sparse sampling channel image.

[0012] Advantages of the Invention:

[0013] Based on the residual-guided demosaicking method, when reconstructing pixels of other channels, the invention interpolates the residual values, pays attention to retaining key details, and avoids the blurring and detail loss problems that may be caused by traditional methods. It also considers the mutual influence between different spectral channels to achieve more accurate color restoration, improve the visual perception and practical application value of the image. The experimental results are better than those of the GF demosaicking method in three indicators, effectively improving the smoothness of the guiding image and reducing the interpolation error caused by high-frequency information, verifying the effectiveness and superiority of the method. In visual evaluation, the images reconstructed by the invention perform excellently in restoring the chrominance channels of the red-green-blue (RGB) image and are closer to the real effect of the original image. Description of the Drawings

[0014] Figure 1: Flowchart of a high-quality multi-spectral demosaicking method based on residual guidance according to the invention;

[0015] Figure 2 (a) is a binary tree splitting diagram, Figure 2 (b) is a nine-channel MSFA diagram;

[0016] Figure 3: Schematic diagram of the arrangement of known pixels and pixels to be estimated in the dense sampling channels described in the invention;

[0017] Figure 4: Schematic diagram of the single-band multi-spectral image reconstruction process described in the invention;

[0018] Figure 5: Comparison of reconstructed images of two algorithms on the dataset. Detailed Embodiment

[0019] The present invention will be described in detail below with reference to the accompanying drawings.

[0020] As Figure 1 shown, a high-quality multi-spectral demosaicking method based on residual guidance includes the following steps:

[0021] Step 1: Design the arrangement of a nine-channel multi-spectral filter array;

[0022] Design the arrangement of a nine-channel multi-spectral filter array based on the binary tree algorithm, which includes a dense sampling channel with a sampling rate and eight

[0023] sparse sampling channels with a Figure 2 sampling rate to meet the requirements of spectral consistency, spatial uniformity, and periodicity. As shown in (a), assuming that the spatial sampling rate of the root node of the binary tree is 1, start binary splitting from the root node first. In each splitting process, the sampling rate of the sub-channel is half of that of the parent channel, that is, the spatial sampling rate of the spectral band at the nth-level node is Figure 2 . The finally obtained nine-channel multi-spectral filter array, as shown in

[0024] (b), has the dense sampling channel at the first-level node with a sampling rate of 1 / 2, and the remaining eight sparse sampling channels at the fourth-level node with a sampling rate of 1 / 16.

[0025] Step 2: Interpolate the image of the dense sampling channel;

[0026] Adopt a low-cost edge-aware demosaicking technique to accurately estimate the interpolated image of the dense sampling channel by calculating the gradient and using the weighted sum of the horizontal and vertical neighboring pixel averages, optimize the image details, and reduce the interpolation error. Use the low-cost edge-aware demosaicking method to interpolate the dense sampling channel containing rich texture information of the target. For each pixel to be estimated in the dense channel, select a local neighborhood window with a size of Figure 3 centered on it. As shown in

[0027] (1)

[0028] Among them, and represent the coordinate positions of the pixel point to be estimated.

[0029] First, calculate the horizontal and vertical gradients at the pixel , and the calculation formula is as follows:

[0030] (2)

[0031] Among them, represents the horizontal gradient, represents the vertical gradient.

[0032] Then, let and be the average values of adjacent pixel values in the horizontal and vertical directions respectively, that is:

[0033] (3)

[0034] Finally, the estimated value can be expressed in the following weighted sum form:

[0035] (4)

[0036] Among them,

[0037] (5)

[0038] Step 3, preliminarily reconstruct the sparse sampling channel image;

[0039] Use the subtracted dense sampling channel image after interpolation and the measured sparse sampling channel image to obtain the sparse color difference map, then use guided filtering to restore the sparse color difference map to obtain the complete color difference map, and then perform an addition operation on the complete color difference map and the dense sampling channel image after interpolation to obtain the preliminary reconstruction result of the sparse sampling channel image.

[0040] As Figure 4 shown in the workflow, the dense sampling channel image after interpolation (denoted as G) is used as the guiding image to reconstruct the sparse sampling channel image with mosaic input (denoted as Y). According to the theory of guided image filter (GIF), within the local window centered on the pixel the estimation of the sparse sampling channel image is a local linear transformation of the guiding image, that is:

[0041] (6)

[0042] Among them, is any pixel point within the window . is a set of linear coefficients assumed to be constant within the window , which can be determined by minimizing the following cost function:

[0043] (7)

[0044] is a binary mask, is the smoothing factor.

[0045] Equation (7) is linear ridge regression, and two coefficients can be calculated through linear regression:

[0046] (8)

[0047] where, and are the mean and variance of the pixels in the guiding image G window respectively, represents the window the number of pixels within, represents the output window the mean of the pixel values of the sampling points within. The output of the guided filter at a certain pixel point is expressed in the form of a weighted average:

[0048] (9)

[0049] where, the window is centered on the pixel and is the window any pixel within, is centered on the pixel and is the window a set of linear coefficients within. Therefore, the equation can be rewritten as:

[0050] (10)

[0051] where, and represent the average of the coefficients of all windows that contain this point i.e.:

[0052] (11)

[0053] Step 4, use the residual interpolation method to further restore the sparse sampling channel image;

[0054] Subtract the preliminarily reconstructed sparse sampling channel image from the measured sparse sampling channel image to obtain a sparse residual map, and use a Gaussian low-pass filter to interpolate the residual to obtain a complete residual map. Add the complete residual map to the preliminarily reconstructed sparse sampling channel image to obtain the finally restored sparse sampling channel image.

[0055] Using the residual interpolation generated from chromatic aberration interpolation, the sparsely sampled channel image preliminarily reconstructed by guided filtering is further enhanced. The advantage of the residual interpolation algorithm is that the residual is smoother than the chromatic aberration, so higher accuracy is obtained. As Figure 4 shown, reconstructing the sparsely sampled channel image using residual interpolation is divided into the following steps:

[0056] First, calculate the estimated value of the guided filter without sampling and the actual measured value to perform a subtraction operation, which is defined as the residual .

[0057] Then, use Gaussian low-pass filtering to interpolate the residual,

[0058] (12)

[0059] where is the spatial domain representation of the Gaussian low-pass filter, and are the row and column coordinates in the spatial domain respectively; is the standard deviation of the Gaussian function, which determines the width of the filter, that is, the attenuation speed of the filter for frequency. The larger the standard deviation, the wider the filter, the more low-frequency components are retained, and the less high-frequency components are suppressed.

[0060] For a given residual image , its result after Gaussian low-pass filtering can be expressed as:

[0061] (13)

[0062] where represents the convolution operation.

[0063] Finally, add the preliminarily reconstructed sparsely sampled channel image and the interpolated residual to obtain the finally restored sparsely sampled channel image .

[0064] The present invention uses multiple indicators to evaluate the demosaicing effect of the image:

[0065] Three main image quality evaluation indicators are adopted: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Spectral Angle Mapper (SAM) to objectively evaluate the effect of the demosaicing algorithm.

[0066] The above three evaluation metrics are calculated in the Standard Red Green Blue (sRGB) color space to quantitatively analyze the similarity between the reconstructed image and the original image. PSNR is measured by calculating the difference between the corresponding pixels of the real image and the reconstructed image and taking the average value; while SSIM and SAM evaluate the similarity between images from the structural and spectral perspectives respectively. The higher the values of these metrics, the better the reconstruction effect of the demosaicing algorithm. Corresponding to each color channel, the calculation method of PSNR is as follows:

[0067] (12)

[0068] In the formula, represents the maximum pixel intensity of the real image. For 8-bit image data, = 255. Similar to PSNR, sRGB SSIM is also calculated for each color channel respectively, and its calculation formula is:

[0069] (13)

[0070] where the means of image and image are , respectively, the variances are , respectively, and the covariance is

[0071] . and are constants determined by the image dynamic range :

[0072] (14)

[0073] In the formula, and are usually default values, taking 0.01 and 0.03 respectively.

[0074] And SAM determines the similarity between the test spectrum and the reference spectrum :

[0075] (15)

[0076] In the formula, = number of frequency bands.

[0077] Example:

[0078] The two algorithms are simulated and analyzed on the CAVE dataset using Matlab 2023b software, and the algorithms are compared in terms of the quality evaluation metrics and visual evaluation metrics of the reconstructed images. The quality evaluation metrics include PSNR, SSIM, and SAM. As shown in Table 1, this method improves by 0.73 dB and 0.0092 compared to the GF method in terms of the two metrics of PSNR and SSIM, and it also shows the test spectrum of the reconstructed image by this method in terms of the SAM metric and the reference spectrum The similarity between them has also increased by 0.009 respectively.

[0079] Table 1

[0080] PSNR↑ / dB PSNR↑ / dB SSIM↑ / dB SSIM↑ / dB SAM↓ / dB SAM↓ / dB CAVE GF Pro GF Pro GF Pro Balloons 41.94 44.08 0.9910 0.9941 0.052 0.049 Beads 25.15 26.99 0.8673 0.9362 0.202 0.143 CD 34.98 37.47 0.9744 0.9794 0.095 0.099 Beers 39.11 42.26 0.9883 0.9902 0.031 0.031 Clay 35.67 37.19 0.9786 0.9874 0.118 0.114

[0081] The above results show that the demosaicked image quality of the present invention on the two datasets is generally better than that of the GF method. As Figure 5 shown, the visual comparison of the demosaicking error maps of the test images in the range of 0 to 50 nm by different methods. Through comparison, it can be clearly seen that the present invention shows significant advantages in terms of demosaicking error, and the details and accuracy of the restored image are better than those of the GF method, verifying the superior performance in processing spectral information.

Claims

1. A high-quality multispectral demosaicing method based on residual guidance, characterized by: The method comprises the following steps: Step 1: Design a nine-channel multispectral filter array arrangement: Design a nine-channel multispectral filter array arrangement based on a binary tree algorithm, which includes a sampling rate of densely sampled channels and eight Sparsely sampled channels with sampling rate; Step 2: Interpolate densely sampled channel images: Use low-cost edge-aware demosaicing technology to accurately estimate the interpolated densely sampled channel images by calculating gradients and using the weighted sum of the average values ​​of horizontal and vertical neighboring pixels; Step 3: Preliminary reconstruction of the sparse sampling channel image: subtract the interpolated dense sampling channel image from the measured sparse sampling channel image to obtain a sparse color difference map, then restore the sparse color difference map using guided filtering to obtain a complete color difference map, and then add the complete color difference map to the interpolated dense sampling channel image to obtain a preliminary reconstruction result of the sparse sampling channel image; Step 4: Use the residual interpolation method to further restore the sparsely sampled channel image: subtract the initially reconstructed sparsely sampled channel image from the measured sparsely sampled channel image to obtain a sparse residual map, and use a Gaussian low-pass filter to interpolate the residual to obtain a complete residual map, and add the complete residual map to the initially reconstructed sparsely sampled channel image to obtain the final restored sparsely sampled channel image.

2. A high-quality multispectral demosaicing method based on residual guidance according to claim 1, characterized in that: The specific steps of step 2 of interpolating densely sampled channel images are as follows: A low-cost edge-aware demosaicing method is used to interpolate dense sampling channels containing rich texture information of the target. For each pixel to be estimated in the dense channel, a pixel with a size of The local neighborhood window is , and the four neighboring pixels above, below, left and right of the center of the local window are set as known pixels. (1) in, and Represents the pixel to be estimated Coordinate location; First, calculate the pixels The horizontal and vertical gradients are calculated as follows: (2) in, represents the horizontal gradient, represents the vertical gradient; Then, let and are the average values ​​of adjacent pixel values ​​in the horizontal and vertical directions, namely: (3) Finally, the estimated value It can be expressed as the following weighted sum form: (4) in, (5)。 3. The high-quality multispectral demosaicing method based on residual guidance according to claim 1, characterized in that: The specific steps of step 3 to preliminarily reconstruct the sparse sampling channel image are: The interpolated densely sampled channel image G is used as a guide image to reconstruct the sparsely sampled channel image Y with mosaic input; according to GIF theory, the pixel The local window at the center In, the estimation of sparsely sampled channel images is a local linear transformation of the guidance image, namely: (6) in, It is a window Any pixel within; It is a window A set of linear coefficients, which are assumed to be constant, can be determined by minimizing the following cost function: (7) is a binary mask, is the smoothing factor; Formula (7) is a linear ridge regression. Two coefficients can be calculated through linear regression: (8) in, and Is the guide image G window Intrinsic pixel mean and variance, Display Window The number of pixels inside. Indicates the output window The mean value of the pixel value at the internal sampling point; guide the filter at a certain pixel point The output at is expressed as a weighted average: (9) Among them, the window In pixels Central, It is a window Any pixel within In pixels Centered window A set of linear coefficients; therefore, the formula can be rewritten as: (10) in, and Indicates that all the points that contain this Window of The average value of the coefficient is: (11)。 4. The high-quality multispectral demosaicing method based on residual guidance according to claim 1, characterized in that: Step 4: Reconstruct the sparsely sampled channel image using residual interpolation It is divided into the following steps: First, calculate the guided filter unsampled estimate Compared with the actual measured value Perform subtraction, defined as the residual ; Then, the residual is interpolated using Gaussian low-pass filtering. (12) in, is the spatial domain representation of the Gaussian low-pass filter, and are the row and column coordinates in the spatial domain, respectively; It is the standard deviation of the Gaussian function, which determines the width of the filter, that is, the attenuation rate of the filter to the frequency; For a given residual image , which is the result after Gaussian low-pass filtering It can be expressed as: (13) in, Represents the convolution operation; Finally, the initially reconstructed sparsely sampled channel image is added to the interpolated residual to obtain the final restored sparsely sampled channel image .

Images