A High-Quality Multispectral Demosaicking Method Based on Edge Priors

Abstract

A multi-spectral demosaicing method based on edge prior belongs to the field of image restoration technology. To solve the problem of artifacts and noise near the edges of the reconstructed image caused by insufficient restoration of the edge texture information of the guiding image, the method includes the following steps: First, design a nine-channel spectral filter array with SSF; Second, construct a dense sampling channel image estimation model; Third, calculate the gradient to obtain the estimated values in the horizontal and vertical directions; Fourth, determine the contribution weights of the estimated values and use the S-shaped function to smooth the mixed estimated values; Fifth, reconstruct the sparse sampling channel image through guided filtering. The present invention uses Taylor polynomials to estimate the pixel values of the dense channels, and calculates the horizontal and vertical first and second order gradients of the unsampled pixel points in the dense sampling channel image to adapt to the characteristics of different spectral channels. Using the S-shaped function as the contribution weight function, multiple estimated values are smoothly mixed to optimize the pixel value estimation and improve the image reconstruction quality.

Description

Technical Field

[0001] The present invention relates to a multispectral demosaicking method based on edge prior, belonging to the technical field of image restoration. Background Art

[0002] The multispectral demosaicking method is an emerging image processing technology aimed at solving the common mosaic phenomenon in multispectral images to improve image quality and color restoration. Using the edge information of the image as important prior knowledge, the characteristics of local regions are analyzed to guide the demosaicking process.

[0003] When using an image sensor with a mosaic pattern filter array, the full spectral information of an object can be captured in a single exposure. The image directly generated by the filter array is called the raw image, and the multispectral demosaicking process is carried out by estimating the missing band information of each sampling point. Currently, the methods of multispectral demosaicking include interpolation method, frequency domain method, sparse representation-based method, and deep learning-based method. It should be noted that a reasonable demosaicking method combined with the MSFA single sensor imaging system can effectively output a high-fidelity multispectral image.

[0004] Referring to the journal "Proceedings of SPIE", the article named "Multispectral demosaicking using guided filter" published by Monno et al. designed a specific pattern five-band multispectral filter array (MSFA). Based on the guided filter method (GF), the authors used adaptive Gaussian upsampling to interpolate the G-band raw image and used it as the guided image to interpolate the remaining bands. Due to the insufficient recovery of the edge texture information of the guided image during the image reconstruction process, this method results in the generation of edge artifacts and noise. Summary of the Invention

[0005] In order to solve the problem of artifacts and noise existing near the edge of the reconstructed image caused by insufficient recovery of the edge texture information of the guided image, the present invention proposes a multispectral demosaicking method based on edge prior.

[0006] The solution of the present invention to solve the technical problem is as follows:

[0007] A multispectral demosaicking method based on edge prior, the method comprising the following steps:

[0008] Step 1: Design a nine-channel spectral filter array with a specific spectral sensitivity function (SSF): Design the layout of the nine-channel multispectral filter array based on the binary tree algorithm, which includes a dense sampling channel with a certain sampling rate and eight sparse sampling channels with a certain sampling rate, as well as nine filters with specific spectral sensitivity functions (SSF) to achieve uniform sampling and meet specific spectral imaging requirements.

[0009] Step 2: Build an image estimation model for the dense sampling channel: Use the Taylor polynomial to estimate the pixel values of the unsampled points in the dense sampling channel image as the weighted sum of the pixel values of its neighboring sampled points.

[0010] Step 3: Calculate the gradients to obtain the estimated values in the horizontal and vertical directions: Calculate the first-order and second-order horizontal and vertical gradients of the unsampled pixel points in the dense sampling channel image to obtain the estimated values in the horizontal and vertical directions.

[0011] Step 4: Determine the contribution weights of the estimated values and use the S-shaped function to smooth the mixed estimated values: Determine the contribution weights of the estimated values through horizontal change 、horizontal upper-side change 、horizontal lower-side change 、vertical change 、vertical left-side change and vertical right-side change to determine the contribution weights of the estimated values, and smooth the mixed estimated values through the S-shaped function.

[0012] Step 5: Reconstruct the sparse sampling channel image through guided filtering: Use the reconstructed dense channel image as the guidance map, and apply the guided filtering technique to reconstruct the original image of the sparse sampling channel to optimize the image quality.

[0013] Advantages of the present invention:

[0014] The present invention uses the Taylor polynomial to estimate the pixel values of the dense channel, and calculates the first-order and second-order horizontal and vertical gradients of the unsampled pixel points in the dense sampling channel image to adapt to the characteristics of different spectral channels. It also uses the S-shaped function as the contribution weight function to smoothly mix multiple estimated values, optimize the pixel value estimation, and improve the image reconstruction quality. The experimental results are better than the guided image filtering (GF) demosaicking method in three metrics, verifying the effectiveness and advantages of this method. In terms of visual evaluation, the images reconstructed by this method are particularly excellent in reducing edge artifacts and maintaining image details, and are closer to the real effect of the original image. Description of the Drawings

[0015] Figure 1: Flowchart of a multispectral demosaicking method based on edge prior of the present invention;

[0016] Figure 2(a) is a binary tree splitting diagram, Figure 2 (b) is a nine-channel MSFA diagram, Figure 2 (c) is the spectral sensitivity function of a nine-channel filter;

[0017] Figure 3: Pixel arrangement of dense channels in a nine-channel multispectral filter array;

[0018] Figure 4: Two-stage workflow of guided filter demosaicing;

[0019] Figure 5: Comparison of example scene reconstruction images (a) real image (b) local real image (c) local GF method reconstruction image (d) local image reconstructed by this method. Specific implementation manner

[0020] The present invention is described in detail below with reference to the accompanying drawings.

[0021] As Figure 1 shown, a multispectral demosaicing method based on edge prior includes the following steps:

[0022] Step 1: Design a nine-channel spectral filter array with a specific spectral sensitivity function (Spectral Sensitivity Function, SSF);

[0023] Design the layout of a nine-channel multispectral filter array based on the binary tree algorithm, which includes a dense sampling channel with a sampling rate and eight sparse sampling channels with a sampling rate, as well as nine filters with a specific spectral sensitivity function (SSF) to achieve uniform sampling and meet specific spectral imaging requirements.

[0024] As Figure 2 (a) shown, assume that the spatial sampling rate of the root node of the binary tree is 1. First, start binary splitting from the root node. In each splitting process, the sampling rate of the sub-channel is half of that of the parent channel, that is, the spatial sampling rate of the spectral band at the th level node is . Finally, the obtained nine-channel multispectral filter array, as Figure 2 (b) shown, the dense sampling channel is located at the first layer of the binary tree, while the low sampling density channels are located at the fourth layer of the binary tree. The sampling rate of each channel is .

[0025] Use Fabry-Perot resonator technology to design nine filters with a specific spectral sensitivity function (SSF), as Figure 2 (c) shown. The spectral sensitivity function of the filter can be expressed as:

[0026] (1)

[0027] Among them, is defined as the wavelength, is the mean or expected value, is the standard deviation.

[0028] Step 2: Construct a dense sampling channel image estimation model;

[0029] Use the Taylor polynomial to estimate the pixel value of the unsampled point in the dense sampling channel image as a weighted sum of the pixel values of its neighboring sampled points.

[0030] The unsampled pixels in the dense sampling channel image The pixel value of is estimated as a weighted sum of the pixel values of its neighboring sampled points (in the image), which is expressed by the Taylor polynomial as follows:

[0031] (2)

[0032] Where is the starting point, represents the th derivative of the function, represents the residual error. When the directional derivative or gradient value is given, the sampled points are used to estimate the unsampled points. Estimate the pixel values at the center position of the neighborhood from the horizontal and vertical directions respectively, as follows:

[0033] (3)

[0034] and represent the horizontal first and second order gradients, and represent the vertical first and second order gradients, 、 are the horizontal and vertical estimated values at the center position respectively. Finally, use the weighted sum of these horizontal and vertical estimated values as the missing pixel value at the center position , that is

[0035] (4)

[0036] Where 、 represent the contribution weights of the direction estimation, .

[0037] Step 3: Calculate the gradients to obtain the estimated values in the horizontal and vertical directions.

[0038] Calculate the horizontal and vertical first-order and second-order gradients of the unsampled pixels of the densely sampled channel image to obtain estimated values ​​in both horizontal and vertical directions;

[0039] The densely sampled channel image middle , , , do not belong to the same spectral channel, such as Figure 3 As shown. In pixels The horizontal first and second order gradients at and ) and vertical first- and second-order gradients (denoted by and ) is calculated by the following formula:

[0040] (5)

[0041] in, , , and represents the contribution weight of the second-order gradient estimate of the neighborhood pixels, , ,Pixel , , and The first-order gradient and pixel , , and The second-order gradient calculation method is as follows:

[0042] (6)

[0043] (7)

[0044] Combining formulas (3), (4) and (5), we can obtain the estimated values ​​in the horizontal and vertical directions.

[0045] Step 4: Determine the estimated value contribution weight and use the S-type function to smooth the mixed estimated value;

[0046] By level changes , horizontal upper side changes , horizontal lower side changes , vertical change , vertical left change And vertical right changes To determine the estimated value contribution weights and smooth the mixed estimates through the S-type function.

[0047] Level changes , horizontal upper - side change , horizontal lower - side change , vertical change , vertical left - side change and vertical right - side change are defined as:

[0048] (8)

[0049] Among them, the above - mentioned change amounts are all dimensionless and applicable to image data of different magnitudes. is the maximum pixel value of the image, is the minimum pixel value of the image.

[0050] Then the estimated value contribution weights , , , , , are defined as:

[0051] (9)

[0052] (10)

[0053] (11)

[0054] Among them, the sigmoid function is used as the contribution weight function to smoothly mix multiple estimated values, where the sum of the weights is 1, thereby optimizing the pixel value estimation in the multi - spectral demosaicing process and improving the image reconstruction quality.

[0055] The form of the sigmoid function is as follows:

[0056] (12)

[0057] Among them, is a positive real number. This sigmoid function is non - exponential in calculation.

[0058] Step five, reconstruct the sparse - sampled channel image through guided filtering;

[0059] Use the reconstructed dense - channel image as the guidance map, apply the guided - filtering technology to reconstruct the original image of the sparse - sampled channel, and optimize the image quality.

[0060] As Figure 4 shown, use the reconstructed high - quality dense - channel image as the guidance map to reconstruct the original images of the remaining sparse - sampled notch channels. For a two - dimensional image, the guidance image G and the output image Y satisfy a linear relationship within a two - dimensional local window as follows:

[0061] (13)

[0062] Among them, is a local neighborhood window centered on pixels and is an arbitrary pixel point within the window. is a set of linear coefficients assumed to be constant within the window.

[0063] First, define the following cost function:

[0064] (14)

[0065] Among them, is a binary mask, with a value of 1 at the sampled points in the original image Y and 0 at other positions, is a regularization parameter used to adjust the effect of the filter.

[0066] Then, minimizing the above cost function gives:

[0067] (15)

[0068] Among them, and are the mean and variance of the pixels within the window of the guidance image G, represents the number of pixel points within the window . represents the mean of the sampled point pixels within the window of the output image Y, that is:

[0069] (16)

[0070] Finally, the estimated value of the output image Y can be expressed as follows:

[0071] (17)

[0072] Among them, and represent the average value of the coefficients of all windows containing this point , that is:

[0073] (18)

[0074] Example:

[0075] The simulation analysis of the two algorithms was carried out on the TT31 dataset using Matlab 2023b software, and the comparison was made from two aspects: the quality evaluation and visual evaluation of the reconstructed images. In terms of quality evaluation, evaluation metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Spectral Angle Mapper (SAM) were adopted. According to the data in Table 1, this technology improved by 2.13 dB and 0.0189 respectively in terms of PSNR and SSIM compared with the traditional guided filtering (GF) method. At the same time, it also showed higher spectral similarity in the SAM index, with an increase of 0.010.

[0076] Table 1

[0077] PSNR↑ / dB PSNR↑ / dB SSIM↑ / dB SSIM↑ / dB SAM↑ / dB SAM↑ / dB TT31 GF Pro GF Pro GF Pro Butterfly 38.16 40.73 0.9878 0.9906 0.025 0.021 Butterfly3 36.39 39.88 0.9790 0.9889 0.061 0.045 Colorchart 35.94 39.86 0.9821 0.9908 0.054 0.044 CD 32.38 35.90 0.9767 0.9895 0.079 0.058 Flower 32.50 36.71 0.9623 0.9834 0.050 0.039

[0078] These results indicate that this method is generally superior to the GF method in terms of the quality of demosaicked images. As Figure 5 shown, the examples of reconstructed images of different methods include the real image, the locally enlarged real image, the locally enlarged reconstructed image of the GF method, and the locally enlarged reconstructed image of the present invention. It can be seen from the figure that this method has significant advantages in restoring the edge texture information of the guidance map. This method can improve the reconstruction quality of multi-spectral images, effectively reduce noise and artifacts while maintaining image details, and its overall performance exceeds that of the GF method.

Claims

1. A multi-spectral demosaicking method based on edge prior, characterized in that, The method comprises the following steps: Step 1: Design a nine-channel spectral filter array with SSF: Based on the binary tree algorithm, a nine-channel multispectral filter array is designed, which includes a dense sampling channel with a sampling rate of 1 / 2 and eight sparse sampling channels with a sampling rate of 1 / 16, as well as nine filters with specific SSF to achieve uniform sampling and meet specific spectral imaging requirements; Step 2: Construct a densely sampled channel image estimation model: use Taylor polynomials to estimate the pixel value of an unsampled point in a densely sampled channel image as the weighted sum of the pixel values ​​of its neighboring sampling points; Step 3: Calculate the gradient to obtain the estimated values ​​in the horizontal and vertical directions: Calculate the horizontal and vertical first-order and second-order gradients of the unsampled pixels of the densely sampled channel image to obtain the estimated values ​​in the horizontal and vertical directions; Step 4, determine the contribution weights of the estimated values and smooth the mixed estimated values using the S-shaped function: Determine the contribution weights of the estimated values through the horizontal change υ h , the horizontal upper-side change , the horizontal lower-side change , the vertical change υ v , the vertical left-side change , and the vertical right-side change , and smooth the mixed estimated values through the S-shaped function; Step 5: Reconstruct the sparsely sampled channel image through guided filtering: Use the reconstructed dense channel image as a guide map, apply guided filtering technology to reconstruct the original image of the sparsely sampled channel, and optimize the image quality; The specific steps of step 2 to construct a dense sampling channel image estimation model are as follows: The pixel value of the unsampled point in the densely sampled channel image is estimated as the weighted sum of the pixel values ​​of its neighboring sampled points using Taylor polynomials; The pixel value of an unsampled pixel (i, j) in a densely sampled channel image is estimated as the weighted sum of the pixel values ​​of its neighboring sampling points in the image, expressed as follows using the Taylor polynomial: where x0 is the starting point, and f n (x0) represents the nth derivative of the function, and R n (x) represents the residual error; when the directional derivative or gradient value is given, sampling points are used to estimate the unsampled points; the pixel values at the neighborhood center positions are estimated separately from the horizontal and vertical directions, as follows: and represent the horizontal first- and second-order gradients, and represent the vertical first- and second-order gradients, which are the horizontal and vertical estimates at the central position (i, j), respectively; finally, the weighted sum of these horizontal and vertical estimates is used as the missing pixel value at the central position (i, j), i.e., Among them, ω h , ω v represent the contribution weights of direction estimation, and ω v +ω h = 1; Step 3: Calculate the horizontal and vertical first-order and second-order gradients of the unsampled pixels of the densely sampled channel image to obtain estimated values ​​in the horizontal and vertical directions; In the dense sampling channel image I, (i,j), (i+2,j), (i-2,j), and (i,j+2) do not belong to the same spectral channel. The horizontal first- and second-order gradients at pixel (i,j) are represented by and and the vertical first- and second-order gradients are represented by and and are calculated by the following formula: Among them, and represent the contribution weights of the second-order gradient estimation of neighboring pixels. The calculation methods of the first-order gradients at pixels (i - 1, j - 1), (i - 1, j + 1), (i + 1, j - 1), and (i + 1, j + 1) and the second-order gradients at pixels (i + 1, j), (i - 1, j), (i, j + 1), and (i, j - 1) are as follows: Combining formulas (3), (4) and (5), we can obtain the estimated values ​​in both horizontal and vertical directions.

2. The multispectral demosaicing method based on edge prior according to claim 1, characterized in that The step 4 is specifically as follows: Define the horizontal change υ h , the horizontal upper side change and the horizontal lower side change as well as the vertical change υ v , the vertical left side change and the vertical right side change as follows: Where, I max is the maximum pixel value of the image; I min is the minimum pixel value of the image; then the estimated value contribution weights ω h and ω v and are defined as: Among them, a sigmoid function is used as a contribution weight function to smoothly mix multiple estimated values, where the sum of the weights is 1, thereby optimizing the pixel value estimation in the multispectral demosaicing process and improving the image reconstruction quality; The S-type function is in the following form: Here, k is a positive real number; the sigmoid function is non-exponential.

3. A multi-spectral demosaicing method based on edge prior according to claim 1, wherein The step five is specifically as follows: The reconstructed high-quality image of the dense channel is used as the guide map, and the original image of the remaining sparsely sampled notch channels is reconstructed using the guide map; for a two-dimensional image, the guide image G and the output image Y satisfy a linear relationship within a two-dimensional local window, as follows: Among them, ω k is a local neighborhood window centered on pixel k, and i is any pixel point within the window; (a k , b k ) is a set of linear coefficients assumed to be constant within the window; First, define the following cost function: Among them, M i is a binary mask, with a value of 1 at the sampled points within the original image Y and a value of 0 at other positions. ε is a regularization parameter used to adjust the effect of the filter; Then, minimizing the above cost function can be obtained: Among them, μ k and are the mean and variance of the pixels in the guiding image G window ω k , and |ω| represents the number of pixel points in the window ω k ; represents the mean of the sampled point pixels in the output image Y window ω k , that is: Finally, the estimated value of the output image Y can be expressed as follows: Among them, and represent the average value of the coefficients of all windows ω i that contain the point i, that is:

Images