High-quality multispectral demosaicing method based on edge prior

Abstract

The invention relates to a multispectral demosaicing method based on edge prior, belongs to the technical field of image restoration, and aims to solve the problem that artifacts and noise exist near the edge of a reconstructed image due to insufficient guided image edge texture information restoration, and the method comprises the following steps: 1, designing a nine-channel spectrum filtering array with SSF; 2, constructing a dense sampling channel image estimation model; 3, calculating the gradient to obtain estimated values in horizontal and vertical directions; 4, determining the contribution weight of the estimated value, and smoothing the mixed estimated value by using an S-type function; and 5, reconstructing a sparse sampling channel image through guided filtering. According to the method, a Taylor polynomial is used for estimating dense channel pixel values, horizontal and vertical first-order and second-order gradients of unsampled pixel points in dense sampling channel images are calculated so as to adapt to characteristics of different spectrum channels, an S-type function is used as a contribution weighting function, multiple estimation values are smoothly mixed, pixel value estimation is optimized, and the method has the advantages of being high in robustness and high in robustness. And the image reconstruction quality is improved.

Description

Technical Field

[0001] The invention relates to a multi-spectral de-mosaicing method based on edge prior, belonging to the technical field of image restoration. Background Art

[0002] Multispectral demosaicing is an emerging image processing technology that aims to solve the common mosaic phenomenon in multispectral images to improve image quality and color reproduction. The edge information of the image is used as important prior knowledge to analyze the characteristics of the local area to guide the demosaicing process.

[0003] When using an image sensor with a mosaic pattern filter array, the full spectral information of the object can be captured in a single exposure. This image generated directly by the filter array is called a raw image, and the multispectral demosaicing process is performed by estimating the other missing band information of each sampling point. At present, the methods of multispectral demosaicing include interpolation, frequency domain, sparse representation-based methods, and deep learning-based methods. It is worth noting that a reasonable demosaicing method combined with the MSFA single-sensor imaging system can effectively output high-fidelity multispectral images.

[0004] See the journal "Proceedings of SPIE", the article published by Monno et al. is titled "Multispectral demosaicking using guided filter", which designs a five-band multispectral filter array (Multispectral Filter Array, MSFA) with a specific pattern. Based on the guided filter method (Guided Filter, GF), the authors use adaptive Gaussian upsampling to interpolate the G-band original image, and use it as a guided image to interpolate the remaining bands. This method does not fully restore the edge texture information of the guided image during the image reconstruction process, resulting in edge artifacts and noise. Summary of the invention

[0005] In order to solve the problem of artifacts and noise near the edge of a reconstructed image caused by insufficient restoration of edge texture information of a guide image, the present invention proposes a multispectral demosaicing method based on edge prior.

[0006] The solution to the technical problem of the present invention is:

[0007] A multispectral demosaicing method based on edge prior, the method comprising the following steps:

[0008] Step 1: Design a nine-channel spectral filter array with a specific spectral sensitivity function (SSF): Design a nine-channel multi-spectral filter array based on a binary tree algorithm, which includes a sampling rate of densely sampled channels and eight The spectral imaging system has sparsely sampled channels with a sampling rate of 100,000 and nine filters with specific spectral sensitivity functions (SSFs) to achieve uniform sampling and meet specific spectral imaging needs.

[0009] Step 2: Construct a densely sampled channel image estimation model: Use Taylor polynomials to estimate the pixel value of an unsampled point in a densely sampled channel image as the weighted sum of the pixel values ​​of its neighboring sampling points.

[0010] Step 3: Calculate the gradient to obtain estimated values ​​in the horizontal and vertical directions: Calculate the horizontal and vertical first-order and second-order gradients of the unsampled pixels of the densely sampled channel image to obtain estimated values ​​in the horizontal and vertical directions.

[0011] Step 4: Determine the estimated value contribution weight and use the S-type function to smooth the mixed estimate: , horizontal upper side changes , horizontal lower side changes , vertical change , vertical left change And vertical right changes To determine the estimated value contribution weights and smooth the mixed estimates through the S-type function.

[0012] Step 5: Reconstruct the sparsely sampled channel image through guided filtering: Use the reconstructed dense channel image as a guide map, apply guided filtering technology to reconstruct the original image of the sparsely sampled channel, and optimize the image quality.

[0013] Beneficial effects of the present invention:

[0014] The present invention uses Taylor polynomials to estimate the pixel values ​​of dense channels, and calculates the horizontal and vertical first-order and second-order gradients of unsampled pixels in densely sampled channel images to adapt to the characteristics of different spectral channels. It also uses S-type functions as contribution weight functions to smoothly mix multiple estimated values, optimize pixel value estimation, and improve image reconstruction quality. The experimental results are superior to the guided image filtering (GF) demosaicing method in all three indicators, verifying the effectiveness and advantages of this method. In terms of visual evaluation, the image reconstructed by this method performs particularly well in reducing edge artifacts and maintaining image details, and is closer to the real effect of the original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG1 is a flow chart of a multispectral demosaicing method based on edge priors according to the present invention;

[0016] Figure 2 (a) is a binary tree split graph. Figure 2 (b) is the nine-channel MSFA map. Figure 2 (c) is the spectral sensitivity function of the nine-channel filter;

[0017] Figure 3: Pixel arrangement of dense channels in a nine-channel multispectral filter array;

[0018] Figure 4: Two-stage workflow for guided filtering demosaicing;

[0019] Figure 5: Comparison of reconstructed images of example scenes (a) real image (b) local image of real image (c) local image reconstructed by GF method (d) local image reconstructed by this method. DETAILED DESCRIPTION

[0020] The present invention is described in detail below with reference to the accompanying drawings.

[0021] like Figure 1 As shown, a multispectral demosaicing method based on edge prior includes the following steps:

[0022] Step 1: Design a nine-channel spectral filter array with a specific spectral sensitivity function (SSF);

[0023] The nine-channel multispectral filter array is designed based on the binary tree algorithm, which includes a sampling rate of densely sampled channels and eight The spectral imaging system has sparsely sampled channels with a sampling rate of 100,000 and nine filters with specific spectral sensitivity functions (SSFs) to achieve uniform sampling and meet specific spectral imaging needs.

[0024] like Figure 2 As shown in (a), the spatial sampling rate of the root node of the binary tree is set to 1. First, binary splitting starts from the root node. In each splitting process, the sampling rate of the child channel is half of that of the parent channel. The spatial sampling rate of the spectral bands at the level node is The resulting nine-channel multispectral filter array is Figure 2 As shown in (b), the densely sampled channels are located at the first level of the binary tree, while the low-sampling density channels are located at the fourth level of the binary tree. The sampling rate of each channel is .

[0025] Nine filters with specific spectral sensitivity functions (SSFs) were designed using Fabry-Perot cavity technology, such as Figure 2 (c) The spectral sensitivity function of the filter can be expressed as:

[0026] (1)

[0027] in, Defined as wavelength, is the mean or expected value, is the standard deviation.

[0028] Step 2: construct a densely sampled channel image estimation model;

[0029] Taylor polynomials are used to estimate the pixel value of an unsampled point in a densely sampled channel image as the weighted sum of the pixel values ​​of its neighboring sampled points.

[0030] Unsampled pixels in densely sampled channel images The pixel value of is estimated as the weighted sum of the pixel values ​​of its neighboring sampling points (in the image), expressed as follows using Taylor polynomials:

[0031] (2)

[0032] in is the starting point, Represents a function Derivatives, Represents the residual error. When the directional derivative or gradient value is given, the sampled points are used to estimate the unsampled points. The pixel value at the center of the neighborhood is estimated from the horizontal and vertical directions respectively, as follows:

[0033] (3)

[0034] and represents the horizontal first-order and second-order gradients, and represents the vertical first-order and second-order gradients, , The center position Finally, the weighted sum of the estimated values ​​in the horizontal and vertical directions is used as the center position The missing pixel value at

[0035] (4)

[0036] in, , represents the contribution weight of the direction estimation, .

[0037] Step 3: Calculate the gradient to get the estimated value in the horizontal and vertical directions.

[0038] Calculate the horizontal and vertical first-order and second-order gradients of the unsampled pixels of the densely sampled channel image to obtain estimated values ​​in both horizontal and vertical directions;

[0039] The densely sampled channel image middle , , , do not belong to the same spectral channel, such as Figure 3 As shown. In pixels The horizontal first and second order gradients at and ) and vertical first- and second-order gradients (denoted by and ) is calculated by the following formula:

[0040] (5)

[0041] in, , , and represents the contribution weight of the second-order gradient estimate of the neighborhood pixels, , ,Pixel , , and The first-order gradient and pixel , , and The second-order gradient calculation method is as follows:

[0042] (6)

[0043] (7)

[0044] Combining formulas (3), (4) and (5), we can obtain the estimated values ​​in the horizontal and vertical directions.

[0045] Step 4: Determine the estimated value contribution weight and use the S-type function to smooth the mixed estimated value;

[0046] By level changes , horizontal upper side changes , horizontal lower side changes , vertical change , vertical left change And vertical right changes To determine the estimated value contribution weights and smooth the mixed estimates through the S-type function.

[0047] Level changes , horizontal upper side changes , horizontal lower side changes , vertical change , vertical left change And vertical right changes Defined as:

[0048] (8)

[0049] Among them, the above-mentioned changes are dimensionless and applicable to image data of different magnitudes. is the maximum pixel value of the image, is the minimum pixel value of the image.

[0050] The estimated value contribution weight , , , , , Defined as:

[0051] (9)

[0052] (10)

[0053] (11)

[0054] The sigmoid function is used as the contribution weight function to smoothly mix multiple estimated values, where the sum of the weights is 1, thereby optimizing the pixel value estimation in the multispectral demosaicing process and improving the image reconstruction quality.

[0055] The S-type function is in the following form:

[0056] (12)

[0057] in, is a positive real number. The sigmoid function is non-exponential.

[0058] Step 5, reconstructing the sparsely sampled channel image through guided filtering;

[0059] The reconstructed dense channel image is used as the guide map, and the guided filtering technique is applied to reconstruct the original image of the sparsely sampled channel to optimize the image quality.

[0060] like Figure 4 As shown in Figure 1, the reconstructed high-quality image of the dense channel is used as the guide map, and the original image of the remaining sparsely sampled notch channels is reconstructed using the guide map. For a two-dimensional image, the guide image G and the output image Y satisfy a linear relationship within a two-dimensional local window, as follows:

[0061] (13)

[0062] in, In pixels The local neighborhood window centered at is any pixel within the window. is a set of linear coefficients assumed to be constant within the window.

[0063] First, define the following cost function:

[0064] (14)

[0065] in, is a binary mask, where the value at the sampled point in the original image Y is 1, and the value at other locations is 0. is a regularization parameter used to adjust the effect of the filter.

[0066] Then, minimizing the above cost function can be obtained:

[0067] (15)

[0068] in, and Is the guide image G window Intrinsic pixel mean and variance, Display window The number of pixels inside. Represents the output image Y window The mean value of the inner sampling point pixels, that is:

[0069] (16)

[0070] Finally, the output image Y estimate It can be expressed as follows:

[0071] (17)

[0072] in, and Indicates that all the points that contain this Window The average value of the coefficients is:

[0073] (18)

[0074] Example:

[0075] The two algorithms were simulated and analyzed on the TT31 dataset using Matlab2023b software, and compared from the aspects of quality evaluation and visual evaluation of the reconstructed image. In terms of quality evaluation, evaluation indicators such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Spectral Angle Mapper (SAM) were used. According to the data in Table 1, the PSNR and SSIM indicators of this technology are improved by 2.13 dB and 0.0189 respectively compared with the traditional guided filtering (GF) method, and the SAM indicator also shows a higher spectral similarity, which is improved by 0.010.

[0076] Table 1

[0077] PSNR↑ / dB PSNR↑ / dB SSIM↑ / dB SSIM↑ / dB SAM↑ / dB SAM↑ / dB TT31 GF Pro GF Pro GF Pro Butterfly 38.16 40.73 0.9878 0.9906 0.025 0.021 Butterfly3 36.39 39.88 0.9790 0.9889 0.061 0.045 Colorchart 35.94 39.86 0.9821 0.9908 0.054 0.044 CD 32.38 35.90 0.9767 0.9895 0.079 0.058 Flower 32.50 36.71 0.9623 0.9834 0.050 0.039

[0078] These results show that this method generally outperforms the GF method in terms of demosaic image quality. Figure 5 As shown in the figure, examples of reconstructed images of different methods include real images, partially enlarged real images, locally enlarged reconstructed images of the GF method, and locally enlarged reconstructed images of the present invention. As can be seen from the figure, this method has significant advantages in restoring the edge texture information of the guide image. This method can improve the reconstruction quality of multispectral images, effectively reduce noise and artifacts while maintaining image details, and its overall performance exceeds that of the GF method.

Claims

1. A multispectral demosaicing method based on edge prior, characterized in that: The method comprises the following steps: Step 1: Design a nine-channel spectral filter array with SSF: Design a nine-channel multi-spectral filter array based on a binary tree algorithm, which includes a sampling rate of densely sampled channels and eight Sparsely sampled channels with different sampling rates, and nine filters with specific SSFs to achieve uniform sampling and meet specific spectral imaging needs; Step 2: Construct a densely sampled channel image estimation model: use Taylor polynomials to estimate the pixel value of an unsampled point in a densely sampled channel image as the weighted sum of the pixel values ​​of its neighboring sampling points; Step 3: Calculate the gradient to obtain the estimated values ​​in the horizontal and vertical directions: Calculate the horizontal and vertical first-order and second-order gradients of the unsampled pixels of the densely sampled channel image to obtain the estimated values ​​in the horizontal and vertical directions; Step 4: Determine the estimated value contribution weight and use the S-type function to smooth the mixed estimate: , horizontal upper side changes , horizontal lower side changes , vertical change , vertical left change And vertical right changes To determine the contribution weight of the estimate, and smooth the mixed estimate through the S-type function; Step 5: Reconstruct the sparsely sampled channel image through guided filtering: Use the reconstructed dense channel image as a guide map, apply guided filtering technology to reconstruct the original image of the sparsely sampled channel, and optimize the image quality.

2. A multispectral demosaicing method based on edge prior according to claim 1, characterized in that: The specific steps of step 2 to construct a dense sampling channel image estimation model are as follows: The pixel value of the unsampled point in the densely sampled channel image is estimated as the weighted sum of the pixel values ​​of its neighboring sampled points using Taylor polynomials; Unsampled pixels in densely sampled channel images The pixel value of is estimated as the weighted sum of the pixel values ​​of its neighboring sampling points in the image, which can be expressed as follows using Taylor polynomials: (2) in is the starting point, Represents a function Derivatives, Represents the residual error; when the directional derivative or gradient value is given, the sampled points are used to estimate the unsampled points; the pixel values ​​at the center of the neighborhood are estimated from the horizontal and vertical directions, respectively, as follows: (3) and represents the horizontal first-order and second-order gradients, and represents the vertical first-order and second-order gradients, , The center position Finally, the weighted sum of the estimated values ​​in the horizontal and vertical directions is used as the center position The missing pixel value at (4) in, , represents the contribution weight of the direction estimation, .

3. The multispectral demosaicing method based on edge prior according to claim 2, characterized in that: Step 3: Calculate the horizontal and vertical first-order and second-order gradients of the unsampled pixels of the densely sampled channel image to obtain estimated values ​​in the horizontal and vertical directions; The densely sampled channel image middle , , , do not belong to the same spectral channel, in the pixel The horizontal first and second order gradients at and ) and the vertical first- and second-order gradients are represented by and It is calculated by the following formula: (5) in, , , and represents the contribution weight of the second-order gradient estimate of the neighborhood pixels, , ,Pixel , , and The first-order gradient and pixel , , and The second-order gradient calculation method is as follows: (6) (7) Combining formulas (3), (4) and (5), we can obtain the estimated values ​​in the horizontal and vertical directions.

4. The multispectral demosaicing method based on edge prior according to claim 3, characterized in that: The step 4 is specifically as follows: Change the level , horizontal upper side changes , horizontal lower side changes , vertical change , vertical left change And vertical right changes Defined as: (8) in, is the maximum pixel value of the image; is the minimum pixel value of the image; The estimated value contribution weight , , , , , Defined as: (9) (10) (11) Among them, a sigmoid function is used as a contribution weight function to smoothly mix multiple estimated values, where the sum of the weights is 1, thereby optimizing the pixel value estimation in the multispectral demosaicing process and improving the image reconstruction quality; The S-type function is in the following form: (12) in, is a positive real number; the sigmoid function is non-exponential.

5. The multispectral demosaicing method based on edge prior according to claim 3, characterized in that: The step five is specifically as follows: The reconstructed high-quality image of the dense channel is used as the guide map, and the original image of the remaining sparsely sampled notch channels is reconstructed using the guide map; for a two-dimensional image, the guide image G and the output image Y satisfy a linear relationship within a two-dimensional local window, as follows: (13) in, In pixels The local neighborhood window centered at is any pixel in the window; is a set of linear coefficients assumed to be constant within the window; First, define the following cost function: (14) in, is a binary mask, where the value at the sampled point in the original image Y is 1, and the value at other locations is 0. is the regularization parameter used to adjust the effect of the filter; Then, minimizing the above cost function can be obtained: (15) in, and Is the guide image G window Intrinsic pixel mean and variance, Display window The number of internal pixels; Represents the output image Y window The mean value of the inner sampling point pixels, that is: (16) Finally, the output image Y estimate It can be expressed as follows: (17) in, and Indicates that all the points that contain this Window The average value of the coefficients is: (18)。

Images