Multispectral demosaicing method based on adaptive spectral correlation iteration

Abstract

The invention relates to the technical field of image data processing, in particular to a multispectral demosaicing method based on adaptive spectral correlation iteration, which comprises the following steps of: firstly, generating an initial pseudo panchromatic image from a mosaic image through weighted smoothing filtering, and reconstructing an initial multispectral image by adopting a guided filtering method; then, using a Pearson's correlation coefficient to quantify spectral correlation among different wavebands, and constructing a high-correlation pseudo panchromatic image to generate a high-correlation multispectral image; and finally, iteratively updating the high-correlation pseudo panchromatic image by using an adaptive iteration strategy, and outputting a high-quality final multispectral image by adaptively selecting an optimal parameter. Therefore, the invention provides the multispectral image demosaicing method which considers the spectral correlation and dynamically updates the guide image, the calculation is simple, the spectral fidelity is high, and the method is more convenient for a user to use.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a multispectral demosaic method based on adaptive spectral correlation iteration, which can be widely applied in fields such as biomedicine, remote sensing monitoring, precision agriculture and industrial inspection. Background Technology

[0002] Compared to traditional RGB imaging, multispectral imaging can capture the reflectance information of a target across multiple discrete spectral bands, enabling more detailed analysis of inherent scene features. However, limited by sensor hardware costs and technology, most multispectral cameras employ multispectral filter arrays for data acquisition. Each pixel records only a single specific band of signal, resulting in a sparsely sampled image, known as a mosaic image. Therefore, recovering the full-resolution image from the mosaic image (de-mosaic) has become a core challenge for multispectral imaging systems.

[0003] The essence of multispectral image demosaic is to accurately estimate unsampled pixels in different spectral bands. To address this issue, several methods have been developed in the field of multispectral demosaic. Brauers et al. proposed weighted bilinear (WB) interpolation, which utilizes inter-channel correlation by considering spectral correlation (SD). Rathi et al. proposed a spectral difference method using convolutional filters to generate a complete multispectral image by calculating the sparsity differences between bands. Miao et al. proposed a binary tree-based edge-aware method that prioritizes the center positions of unsampled pixels through interpolation. Yu et al. proposed WB interpolation based on convolutional filters and optimized it using multi-scale large kernel attention. Antonucci et al. proposed a strategy combining traditional interpolation and non-convex matrix completion, utilizing low-rank structures to improve reconstruction quality. Zhang et al. proposed a method that iteratively updates the inter-channel spectral differences and then reconstructs a full-resolution multispectral image by integrating these differences. However, these methods often ignore the spectral correlation between different bands, easily leading to spectral aliasing when band differences are large. To address this issue, researchers have explored methods utilizing pseudo-panchromatic images. Rathi et al. proposed a general method to calculate the sparse differences between each spectral band and the pseudo-panchromatic image, and then generate a multispectral image through iterative updates of the pseudo-panchromatic image. Jeong et al. proposed an iterative linear regression model to estimate the pseudo-panchromatic image, which generates an iterative guided filter to balance spatial and spectral fidelity. Liao et al. proposed a strategy to adjust the guided filter window by utilizing the correlation between different channels. However, these methods use a fixed guided image and ignore the spectral correlation between different bands. Currently, researchers have studied deep learning-based demosaic methods, but these methods rely on large-scale data to learn the mapping relationship and have poor generalization ability in a few scenarios. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a multispectral demosaic method that features high spectral fidelity, simple calculation, consideration of spectral correlation, and dynamic updating of the guiding image.

[0005] This invention is achieved through the following technical solution: A multispectral demosaic method based on adaptive spectral correlation iteration is characterized by the following steps: S1, Generate mosaic image: Use a Gaussian function to simulate the transmittance function of a multispectral filter to construct a reference image, and use a multispectral filter array to downsample the reference image to generate a mosaic image; S2, Initial Multispectral Image Reconstruction: Extract the global spatial structure from the mosaic image to generate an initial pseudo-panchromatic image, and reconstruct the initial multispectral image using a guided filtering method; S3, Construction of a highly correlated pseudo-panchromatic image: The Pearson correlation coefficient is used to quantify the spectral correlation between different bands of the initial multispectral image. Highly correlated bands are selected by screening thresholds and a highly correlated pseudo-panchromatic image is constructed. The highly correlated multispectral image is reconstructed using a guided filtering method. S4, Adaptive Iteration to Determine Parameters: The highly correlated pseudo-panchromatic image is iteratively updated using an adaptive iterative strategy. The optimal number of iterations and the screening threshold are adaptively selected through a comprehensive evaluation index to output a high-quality final multispectral image.

[0006] The multispectral demosaicing method based on adaptive spectral correlation iteration described in this invention also has the following feature: the specific steps for generating the mosaic image are as follows: S11, a reference image is constructed by simulating the transmittance function of a multispectral filter using a Gaussian function, and a CIE D65 light source is used. The pixel value of the simulated multispectral camera is calculated from the spectral reflectance at a point. Finally, the pixel value obtained by integration is normalized to ensure that the pixel value of each band is within the range of [0, 255]. The calculation method is as follows: In the formula For filter Pixel values ​​of the band, It is a point Spectral reflectance of the scene The spectral power distribution of the light source, express The sensitivity function of the filter in the band. These are the normalization coefficients; For wavelength range; S12, a mosaic image is generated by downsampling the reference image using a multispectral filtering array. This multispectral filtering array has a 4×4 binary tree arrangement structure, and each spatial pixel records information of only a single band. The final generated single-band mosaic image can be represented as follows: In the formula for Binary mask for a band, when the pixel is in The value is 1 for bands and 0 for other bands. This indicates element-wise multiplication.

[0007] The multispectral demosaicing method based on adaptive spectral correlation iteration described in this invention also has the following feature: the specific steps of the initial multispectral image reconstruction are as follows: S21, a pseudo-panchromatic image is generated by fusing the mosaic image through weighted smoothing filtering and extracting the global spatial structure: The asterisk (*) indicates a two-dimensional convolution operation, using the "same" mode and the "replicate" padding strategy. This represents the corresponding weighted smoothing filter kernel, which enhances spatial continuity through weighted averaging, enabling the pseudo-panchromatic image to retain the main edges and texture structures of the scene; S22, the initial multispectral image is reconstructed using a guided filtering method, because the image to be reconstructed... Band image It satisfies a local linear relationship with the pseudopanchromatic image: In the formula, Represented in pixels A local window centered at a radius of 4, For any pixel in this window, and It is a set of locally linear coefficients within the window, only related to the center position. related; S23, Solve and By minimizing the energy function To preserve the structural information of the guide map, suppress noise and prevent overfitting, and minimize the difference from the output image, since a single pixel i may belong to multiple windows, the final output needs to be a weighted average of the linear coefficients of all windows containing that pixel. The calculation method is as follows: In the formula, It is a regularization parameter to prevent Too large an amount can lead to overfitting. express Mosaic image of the band at location Pixel values; binary mask The constraint-guided filtering is used to calculate linear coefficients using only the effective sampling points of that band. S24, the final output, which preserves the spatial structure, can be represented as: The multispectral demosaicing method based on adaptive spectral correlation iteration described in this invention also has the following feature: the specific steps for constructing the highly correlated pseudo-panchromatic image are as follows: S31, taking into full account the inherent spectral correlation between bands in multispectral images, uses the Pearson correlation coefficient to quantify the spectral correlation between different bands for multispectral images. and Band spectral vector and Its Pearson correlation coefficient is defined as: In the formula For covariance, The standard deviation is denoted as Pearson correlation coefficient, which ranges from [−1, 1]. The closer the value is to 1, the stronger the linear correlation between the two bands. S32, Set the filter threshold (Step size 0.01) and determine the set of highly correlated bands. Based on this set, construct Highly correlated pseudo-panchromatic images of the bands: In the formula, For the first In the next iteration Highly correlated pseudo-panchromatic images of the bands, The number of elements in the set. For the multispectral image after the previous iteration Band image, and its related band set It is dynamically updated as the correlation of different spectral bands changes during the iteration process to adapt to changes in spectral characteristics; S33, with As a new guiding image, a highly correlated multispectral image is reconstructed using a guided filtering method: In the formula, This represents the guided filter operator, with the guided graph as input. With mosaic image The multispectral demosaicing method based on adaptive spectral correlation iteration described in this invention also has the following feature: wherein the specific steps for determining the parameters through adaptive iteration are as follows: S41, Excessive iterations may lead to overfitting to noise or local anomalous spectra, and also increase computational costs; therefore, the maximum number of iterations is set to 16. To determine the optimal number of iterations t and the screening threshold... To balance reconstruction quality and computational cost, a comprehensive evaluation index was defined. To quantify reconstruction error: In the formula, RMSE is the root mean square error, which measures the deviation of pixel values ​​from the true values; SAM is the spectral angle mapping, which measures the angular deviation of the spectral vector; PSNR is the peak signal-to-noise ratio, which evaluates the quality of spatial detail recovery; and SSIM is the structural similarity, which reflects the degree of preservation of image structure and contrast. S42, determine the optimal parameter combination by minimizing the comprehensive evaluation index: S43 outputs high-quality final multispectral images. This completes the mosaic removal process for multispectral images.

[0008] The beneficial effects of this invention are as follows: According to the multispectral demosaicing method of this invention, a pseudo-panchromatic image is generated from a mosaic image through weighted smoothing filtering, and the initial multispectral image is recovered using a guided filtering method. Then, the spectral correlation between different bands is quantified using the Pearson correlation coefficient to construct a band-selective pseudo-panchromatic image for multispectral image estimation. Finally, an adaptive iterative strategy for dynamically updating the band-selective pseudo-panchromatic image is adopted to generate a high-quality multispectral image by adaptively selecting the optimal parameters. Therefore, this invention provides a multispectral image demosaicing method that considers spectral correlation and dynamically updates the guided image. It is computationally simple, has high spectral fidelity, and is relatively convenient for users. Attached Figure Description

[0009] Figure 1 This is a flowchart of the multispectral demosaicing method based on adaptive spectral correlation iteration of the present invention; Figure 2 This is the Pearson correlation coefficient matrix of the 16-band multispectral image of the present invention; Figure 3 A schematic diagram of the selected scene for the CAVE dataset of this invention; Figure 4 For the present invention Figure 3 A comparison of the spectral curves at Position 1 in the CAVE scene shown; Figure 5 For the present invention Figure 3 A comparison of the spectral curves at Position 2 in the CAVE scene shown; Figure 6 For the present invention Figure 3 A comparison of the spectral curves at Position 3 in the CAVE scene shown; Figure 7 A schematic diagram of the selected scenario for the TT-31 dataset of this invention; Figure 8 For the present invention Figure 7A comparison of the spectral curves at Position 1 in the TT-31 scene shown; Figure 9 For the present invention Figure 7 A comparison of the spectral curves at Position 2 in the TT-31 scene shown; Figure 10 For the present invention Figure 7 A comparison of the spectral curves at Position 3 in the TT-31 scene shown; Figure 11 A schematic diagram of the selected scene for the NTIRE dataset of this invention; Figure 12 For the present invention Figure 11 A comparison of the spectral curves at Position 1 in the NTIRE scene shown; Figure 13 For the present invention Figure 11 A comparison of the spectral curves at Position 2 in the NTIRE scene shown; Figure 14 For the present invention Figure 11 The image shows a comparison of the spectral curves at Position 3 in the NTIRE scene. Detailed Implementation

[0010] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments are described in detail with reference to the accompanying drawings.

[0011] Figures 1-14 This is a specific embodiment of the present invention, which is... A multispectral demosaicing method based on adaptive spectral correlation iteration includes the following steps: Step 1, Generate Mosaic Image: A reference image is constructed by simulating the transmittance function of a multispectral filter using a Gaussian function. The reference image is then downsampled using a multispectral filter array to generate a mosaic image. The specific steps are as follows: (1) A reference image was constructed by simulating the transmittance function of the multispectral filter using a Gaussian function, and a CIE D65 light source was used. The pixel value of the simulated multispectral camera is calculated from the spectral reflectance at a point. Finally, the pixel value obtained by integration is normalized to ensure that the pixel value of each band is within the range of [0, 255]. The calculation method is as follows: In the formula For filter Pixel values ​​of the band, It is a point Spectral reflectance of the scene The spectral power distribution of the light source, express The sensitivity function of the filter in the band. These are the normalization coefficients; For wavelength range; (2) A mosaic image is generated by downsampling the reference image using a multispectral filtering array. This multispectral filtering array has a 4×4 binary tree arrangement structure, and each spatial pixel records information of only a single band. The final generated single-band mosaic image can be represented as: In the formula for Binary mask for a band, when the pixel is in The value is 1 for bands and 0 for other bands. This indicates element-wise multiplication.

[0012] Step 2, Initial Multispectral Image Reconstruction: Extract the global spatial structure from the mosaic image to generate an initial pseudo-panchromatic image, and then reconstruct the initial multispectral image using a guided filtering method. The specific steps are as follows: (1) By fusing the mosaic image through weighted smoothing filtering, the global spatial structure is extracted to generate a pseudo-panchromatic image: The asterisk (*) indicates a two-dimensional convolution operation, using the "same" mode and the "replicate" padding strategy. This represents the corresponding weighted smoothing filter kernel, which enhances spatial continuity through weighted averaging, enabling the pseudo-panchromatic image to retain the main edges and texture structures of the scene; (2) Reconstruct the initial multispectral image using the guided filtering method, assuming that the image to be reconstructed is... Band image It satisfies a local linear relationship with the pseudopanchromatic image: In the formula, Represented in pixels A local window centered at a radius of 4, For any pixel in this window, and It is a set of locally linear coefficients within the window, only related to the center position. related; (3) Solve and By minimizing the energy function To preserve the structural information of the guide map, suppress noise and prevent overfitting, and minimize the difference from the output image, since a single pixel i may belong to multiple windows, the final output needs to be a weighted average of the linear coefficients of all windows containing that pixel. The calculation method is as follows: In the formula, It is a regularization parameter to prevent Too large an amount can lead to overfitting. express Mosaic image of the band at location Pixel values; binary mask The constraint-guided filtering is used to calculate linear coefficients using only the effective sampling points of that band. (4) The final output, which preserves the spatial structure, can be represented as: Step 3, Construction of a High-Correlation Pseudo-Palmchromatic Image: The Pearson correlation coefficient is used to quantify the spectral correlation between different bands of the initial multispectral image. High-correlation bands are selected by a threshold and a high-correlation pseudo-panchromatic image is constructed. The highly correlated multispectral image is then reconstructed using a guided filtering method. The specific steps are as follows: (1) Taking full account of the inherent spectral correlation between bands in multispectral images, the Pearson correlation coefficient is used to quantify the spectral correlation between different bands for multispectral images. and Band spectral vector and Its Pearson correlation coefficient is defined as: In the formula For covariance, The standard deviation is denoted as Pearson correlation coefficient, which ranges from [−1, 1]. The closer the value is to 1, the stronger the linear correlation between the two bands. (2) Set the filtering threshold (Step size 0.01) and determine the set of highly correlated bands. Based on this set, construct Highly correlated pseudo-panchromatic images of the bands: In the formula, For the first In the next iteration Highly correlated pseudo-panchromatic images of the bands, The number of elements in the set. For the multispectral image after the previous iteration Band image, and its related band set It is dynamically updated as the correlation of different spectral bands changes during the iteration process to adapt to changes in spectral characteristics; (3) with As a new guiding image, a highly correlated multispectral image is reconstructed using a guided filtering method: In the formula, This represents the guided filter operator, with the guided graph as input. With mosaic image .

[0013] Step 4, Adaptive Iteration to Determine Parameters: The highly correlated pseudo-panchromatic image is iteratively updated using an adaptive iteration strategy. The optimal number of iterations and screening threshold are adaptively selected through a comprehensive evaluation index, outputting a high-quality final multispectral image. The specific operation steps are as follows: (1) Too many iterations may lead to overfitting to noise or local anomalous spectra, and will also increase computational costs. The maximum number of iterations is set to 16. In order to determine the optimal number of iterations t and the screening threshold, To balance reconstruction quality and computational cost, a comprehensive evaluation index was defined. To quantify reconstruction error: In the formula, RMSE is the root mean square error, which measures the deviation of pixel values ​​from the true values; SAM is the spectral angle mapping, which measures the angular deviation of the spectral vector; PSNR is the peak signal-to-noise ratio, which evaluates the quality of spatial detail recovery; and SSIM is the structural similarity, which reflects the degree of preservation of image structure and contrast. (2) Determine the optimal parameter combination by minimizing the comprehensive evaluation index: Output high-quality final multispectral images This completes the mosaic removal process for multispectral images.

[0014] The role and effect of the embodiments This embodiment provides a multispectral demosaicing method that considers spectral correlation and dynamically updates the guiding image to improve the accuracy of multispectral image reconstruction. First, a pseudo-panchromatic image is generated from the mosaic image using weighted smoothing filtering, and the initial multispectral image is recovered using a guided filtering method. Then, the spectral correlation between different bands is quantified using the Pearson correlation coefficient to construct a band-selective pseudo-panchromatic image for multispectral image estimation. Finally, an adaptive iterative strategy for dynamically updating the band-selective pseudo-panchromatic image is employed to adaptively select the optimal parameters and generate a high-quality multispectral image.

[0015] In this invention, the Pearson correlation coefficient was selected as the quantitative index of spectral correlation. Compared with other correlation coefficients, the Pearson correlation coefficient has advantages such as high accuracy in quantifying linear correlation, low computational complexity, and high spectral fidelity in capturing spectral features. Figure 2 The Pearson correlation coefficient matrix of the 16-band multispectral image is displayed.

[0016] from Figure 2 As can be seen, the correlation between adjacent bands is significantly higher than that between non-adjacent bands.

[0017] Table 1. Comparison of the average values ​​of PSNR, SSIM, RMSE, and SAM for this invention and other demosaic methods across all datasets. The table above shows the average PSNR, SSIM, RMSE, and SAM for the proposed method and other demosaic methods across all datasets, with the best results highlighted in bold.

[0018] The results show that the present invention outperforms the comparative methods in all metrics. It performs exceptionally well in spatial detail recovery and spectral fidelity, and maintains strong generalization ability across different scenarios.

[0019] To verify the spectral reconstruction capability of the present invention Figures 3-14 A test scene was selected from the CAVE, TT-31, and NTIRE datasets, and the spectral curves of three texture-rich locations in the reconstructed spectral images obtained by different demosaic methods were compared.

[0020] from Figures 3-14 As can be seen from the data, compared with other methods, the reconstructed spectral data curve of the present invention is closer to the true ground value and has better spectral recovery performance than other methods.

[0021] In the multispectral demosaic method of this embodiment, since spectral correlation and dynamic updating of the guide image are taken into account, the calculation is simple, the spectral fidelity is high, and it is relatively convenient for users to use.

[0022] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A multispectral demosaicing method based on adaptive spectral correlation iteration, characterized in that, This method is implemented by the following steps: S1, Generate mosaic image: Use a Gaussian function to simulate the transmittance function of a multispectral filter to construct a reference image, and use a multispectral filter array to downsample the reference image to generate a mosaic image; S2, Initial Multispectral Image Reconstruction: Extract the global spatial structure from the mosaic image to generate an initial pseudo-panchromatic image, and reconstruct the initial multispectral image using a guided filtering method; S3, Construction of a highly correlated pseudo-panchromatic image: The Pearson correlation coefficient is used to quantify the spectral correlation between different bands of the initial multispectral image. Highly correlated bands are selected by screening thresholds and a highly correlated pseudo-panchromatic image is constructed. The highly correlated multispectral image is reconstructed using a guided filtering method. S4, Adaptive Iteration to Determine Parameters: The highly correlated pseudo-panchromatic image is iteratively updated using an adaptive iterative strategy. The optimal number of iterations and the screening threshold are adaptively selected through a comprehensive evaluation index to output a high-quality final multispectral image.

2. The multispectral demosaicing method based on adaptive spectral correlation iteration according to claim 1, characterized in that, The specific steps for generating the mosaic image in S1 are as follows: S11, a reference image is constructed by simulating the transmittance function of a multispectral filter using a Gaussian function, and a CIE D65 light source is used. The pixel value of the simulated multispectral camera is calculated from the spectral reflectance at a point. Finally, the pixel value obtained by integration is normalized to ensure that the pixel value of each band is within the range of [0, 255]. The calculation method is as follows: In the formula For filter Pixel values ​​of the band, It is a point Spectral reflectance of the scene The spectral power distribution of the light source, express The sensitivity function of the filter in the band. These are the normalization coefficients; For wavelength range; S12, a mosaic image is generated by downsampling the reference image using a multispectral filtering array. This multispectral filtering array has a 4×4 binary tree arrangement structure, and each spatial pixel records information of only a single band. The final generated single-band mosaic image can be represented as follows: In the formula for Binary mask for a band, when the pixel is in The value is 1 for bands and 0 for other bands. This indicates element-wise multiplication.

3. The multispectral demosaicing method based on adaptive spectral correlation iteration according to claim 1, characterized in that, The specific steps for initial multispectral image reconstruction in S2 are as follows: S21, a pseudo-panchromatic image is generated by fusing the mosaic image through weighted smoothing filtering and extracting the global spatial structure: Where '*' represents a two-dimensional convolution operation, using the "same" mode and the "replicate" padding strategy; This represents the corresponding weighted smoothing filter kernel, which enhances spatial continuity through weighted averaging, enabling the pseudo-panchromatic image to retain the main edges and texture structures of the scene; S22, the initial multispectral image is reconstructed using a guided filtering method, because the image to be reconstructed... Band image It satisfies a local linear relationship with the pseudopanchromatic image: In the formula, Represented in pixels A local window centered at a radius of 4, For any pixel in this window, and It is a set of locally linear coefficients within the window, only related to the center position. related; S23, Solve and By minimizing the energy function To preserve the structural information of the guiding image, suppress noise and prevent overfitting, and minimize the difference from the output image, since a single pixel i may belong to multiple windows, the final output needs to be a weighted average of the linear coefficients of all windows containing that pixel. The calculation method is as follows: In the formula, It is a regularization parameter to prevent Too large an area leads to overfitting; express Mosaic image of the band at location Pixel values; binary mask The constraint-guided filtering is used to calculate linear coefficients using only the effective sampling points of that band. S24, the final output, which preserves the spatial structure, can be represented as: 。 4. The multispectral demosaicing method based on adaptive spectral correlation iteration according to claim 1, characterized in that, The specific steps for constructing the highly correlated pseudo-panchromatic image in S3 are as follows: S31, taking into full account the inherent spectral correlation between bands in multispectral images, uses the Pearson correlation coefficient to quantify the spectral correlation between different bands for multispectral images. and Band spectral vector and Its Pearson correlation coefficient is defined as: In the formula For covariance, The standard deviation is denoted by Pearson correlation coefficient, which ranges from [−1, 1]. The closer the value is to 1, the stronger the linear correlation between the two bands; the closer the value is to 0, the weaker the linear correlation. S32, Set the filter threshold And determine the set of highly correlated bands. Based on this set, construct Highly correlated pseudo-panchromatic images of the bands: In the formula, For the first In the next iteration Highly correlated pseudo-panchromatic images of the bands, The number of elements in the set. For the multispectral image after the previous iteration Band image, and its related band set It is dynamically updated as the correlation of different spectral bands changes during the iteration process to adapt to changes in spectral characteristics; S33, with As a new guiding image, a highly correlated multispectral image is reconstructed using a guided filtering method: In the formula, This represents the guided filter operator, with the guided image as input. With mosaic image .

5. The multispectral demosaicing method based on adaptive spectral correlation iteration according to claim 1, characterized in that, The specific steps for determining parameters through adaptive iteration in S4 are as follows: S41, Excessive iterations may lead to overfitting to noise or local anomalous spectra, and also increase computational costs; therefore, the maximum number of iterations is set to 16. To determine the optimal number of iterations t and the screening threshold... To balance reconstruction quality and computational cost, a comprehensive evaluation index was defined. To quantify reconstruction error: In the formula, RMSE is the root mean square error, which measures the deviation of pixel values ​​from the true values; SAM is the spectral angle mapping, which measures the angular deviation of the spectral vector; PSNR is the peak signal-to-noise ratio, which evaluates the quality of spatial detail recovery; and SSIM is the structural similarity, which reflects the degree of preservation of image structure and contrast. S42, determine the optimal parameter combination by minimizing the comprehensive evaluation index: ； S43 outputs high-quality final multispectral images. This completes the mosaic removal process for multispectral images.

Images