Snapshot multispectral camera demosaicing method, apparatus, device and storage medium

Abstract

The application discloses a snapshot multispectral camera demosaicing method and device based on a binary tree coating array, according to a multispectral original image, a plurality of single spectral band mosaic images are extracted according to spectral bands, and a pseudo panchromatic estimation image is estimated according to the multispectral original image; a difference value between each spectral band sparse pseudo panchromatic estimation image and an original spectral band sparse image is calculated, and the difference value at all pixel positions is calculated; then, the difference value image after filtering and interpolation is added to the pseudo panchromatic estimation image, to obtain a demosaicing reconstruction result based on the pseudo panchromatic estimation image; and the color and spatial correlation between adjacent spectral bands is utilized to enhance the spatial high-frequency information of a binary tree low node spectral band image based on a binary tree high node spectral band image. The above method can solve the artifact problem caused by the weak spectral correlation of a wide spectral distribution filter, reduce the artifact of a reconstructed image, and improve the spatial resolution of the image after demosaicing.

Description

Technical Field

[0001] This invention relates to the field of multispectral camera technology, and in particular to a method, apparatus, device, and storage medium for demosaicing of a snapshot multispectral camera. Background Technology

[0002] Snapshot multispectral cameras, based on multispectral filter arrays, can acquire a single mosaic image in a single imaging session. They offer advantages such as compact structure and the ability to acquire images across all spectral bands at once. The method of designing filter arrays in a mosaic pattern conforms to the characteristics of human vision. This method was initially applied to color digital cameras, which used a color filter array composed of three colors (R, G, B) coupled to a detector. This ensured that each pixel of the detector could only measure the spectral information of one color, and the missing values ​​for each color could be estimated from neighboring pixels. Similar to color filter arrays, after a multispectral filter array is coupled to a detector, each pixel of the detector can only measure the spectral information of one spectral band. To obtain the complete multispectral image, it is necessary to estimate the positions of pixels lacking information in each spectral band. This reconstruction process is commonly referred to as the demosaicing process of the multispectral filter array.

[0003] Compared to color filter arrays in digital cameras, multispectral filter arrays have more spectral bands, narrower bandwidth, and wider spectral coverage. However, each spectral band lacks more pixels. Directly applying demosaic methods used for color filter arrays to multispectral filter arrays yields poor results. Therefore, it is necessary to research demosaic methods suitable for multispectral filter arrays. Another existing method is applicable to non-redundant multispectral filter arrays. Based on the spectral and spatial correlations between spectral bands, it estimates a pseudo-panchromatic intensity image and then uses it as a reference image to reconstruct images for each spectral band. Since only one band channel is available for each pixel in each n*n array of the original image, assuming that the intensity is strongly correlated between adjacent pixels, considering this spatial correlation, a smoothing filter M can be used to directly estimate the intensity on the original image, defining the intensity level of each pixel as the average level of all channels. However, this method is only applicable to non-redundant filter arrays and is not suitable for filter arrays based on binary tree designs. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, device, and storage medium for de-mosaicing of a snapshot multispectral camera. This method can solve the artifact problem caused by the weak spectral correlation of broadband distribution filters, reduce artifacts in reconstructed images, and improve the spatial resolution of the de-mosaiced image.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A method for demosaicing a snapshot multispectral camera, the method comprising:

[0007] Step 1: Based on the original multispectral image, extract multiple single-band mosaic images according to the spectral bands, and estimate the pseudo-panchromatic image based on the original multispectral image;

[0008] Step 2: Calculate the difference between the sparse pseudo-panchromatic estimated image of each spectral band and the original sparse spectral band image. Select the corresponding weighted interpolation filter kernel according to the spectral band at different nodes, and convolve it with the difference result to calculate the difference at all pixel positions. Then add the filtered and interpolated difference image to the pseudo-panchromatic estimated image to obtain the de-mosaic reconstruction result based on the pseudo-panchromatic estimated image.

[0009] Step 3: Utilizing the color and spatial correlation between neighboring spectral bands, based on the demosaic reconstruction results obtained in Step 2, enhance the spatial high-frequency information of the low-node spectral band image of the binary tree by using the high-node spectral band image of the binary tree.

[0010] A snapshot-type multispectral camera demosaic device, the device comprising:

[0011] The pseudo-panchromatic estimation image acquisition unit is used to extract multiple single-band mosaic images according to the spectral bands based on the original multispectral image, and to estimate the pseudo-panchromatic estimation image based on the original multispectral image.

[0012] The demosaic reconstruction result acquisition unit is used to calculate the difference between the sparse pseudo-panchromatic estimated image of each spectral band and the original sparse spectral band image. It selects the corresponding weighted interpolation filter kernel according to the spectral band at different nodes and convolves it with the difference result to calculate the difference at all pixel positions. Then, it adds the filtered and interpolated difference image with the pseudo-panchromatic estimated image to obtain the demosaic reconstruction result based on the pseudo-panchromatic estimated image.

[0013] The binary tree low-node spectral band image enhancement unit is used to enhance the spatial high-frequency information of the binary tree low-node spectral band image based on the high-node spectral band image of the binary tree by utilizing the color and spatial correlation between neighboring spectral bands and the demosaic reconstruction result obtained in step 2.

[0014] As can be seen from the technical solutions provided by the present invention, the above-mentioned methods and devices can solve the artifact problem caused by the weak spectral correlation of broadband distribution filters, reduce artifacts in reconstructed images, and improve the spatial resolution of images after mosaic removal. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram of the process for a snapshot multispectral camera demosaicing method based on a binary tree coating array provided in an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of a 10-band filter array based on a binary tree design for coating, as exemplified in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, and do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0019] like Figure 1 The diagram shows a flowchart of a snapshot multispectral camera demosaic method based on a binary tree coating array provided in an embodiment of the present invention. The method includes:

[0020] Step 1: Based on the original multispectral image, extract multiple single-band mosaic images according to the spectral bands, and estimate the pseudo-panchromatic image based on the original multispectral image;

[0021] In this step, a raw multispectral image I is obtained through a single imaging process using a snapshot multispectral camera. MSFA Multiple single-band mosaic images are obtained by extracting each spectral band separately;

[0022] Count the number of times each spectral segment appears within the local window of each pixel, n, and construct a filter kernel P centered on each spectral segment. Set the value of each element of the filter kernel P to 1 / n.

[0023] Then, examine the original multispectral image I. MSFA Perform filtering operations, selecting the appropriate filter kernel P based on the spectral band of each pixel. k Let k be the spectral band, and a filtering operation be performed. The calculation is completed pixel by pixel to obtain the pseudo-panchromatic estimation image I. P , represented as:

[0024] I P =I MSFA *Pk .

[0025] For example, consider a 10-band multispectral filter array based on a binary tree design, such as... Figure 2 The diagram shown is a schematic of a 10-band filter array based on a binary tree design for coating, as illustrated in this invention. The filter kernels P for bands 0 to 9 are as follows:

[0026]

[0027] Step 2: Calculate the difference between the sparse pseudo-panchromatic estimated image of each spectral band and the original sparse spectral band image. Select the corresponding weighted interpolation filter kernel according to the spectral band at different nodes, and convolve it with the difference result to calculate the difference at all pixel positions. Then add the filtered and interpolated difference image to the pseudo-panchromatic estimated image to obtain the de-mosaic reconstruction result based on the pseudo-panchromatic estimated image.

[0028] In this step, for each spectral band k, the sparse pseudo-panchromatic estimation image I under each spectral band k is calculated. P ⊙m k Compared with the original spectral sparse image I k The difference Δ k , represented as:

[0029] I k =I MSFA ⊙m k

[0030] Δ k =I k -I P ⊙m k

[0031] ⊙ represents dot product; m k A binary mask is used, the mask size is the same as the original image I. MSFA Consistent, where spectral band k is assigned a value of 1 at the corresponding position of the pixel in the original image, and a value of 0 at other positions; I P It is a pseudo-panchromatic estimated image;

[0032] For each spectral band k, the difference at all pixel positions is calculated using weighted interpolation filtering, and is expressed as:

[0033]

[0034] The image shows the difference after filtering and interpolation; H-weighted interpolation low-pass filter kernel;

[0035] For different spectral segments at different nodes in the binary tree, the weighted interpolation low-pass filter kernel H is different, with... Figure 2 For example, the weighted interpolation low-pass filter kernel H is:

[0036]

[0037] Finally, the filtered and interpolated difference image is added to the pseudo-panchromatic estimation image to obtain the demosaic reconstruction result based on the pseudo-panchromatic estimation image, as shown below:

[0038]

[0039] The image is a demosaic reconstruction result based on a pseudo-panchromatic estimation image. Since the estimation of the pseudo-panchromatic image is obtained by convolving the mean filter with the original mosaic image, this operation will lose some of the spatial high-frequency information in the original image.

[0040] Step 3: Utilizing the color and spatial correlation between neighboring spectral bands, based on the demosaic reconstruction results obtained in Step 2, enhance the spatial high-frequency information of the low-node spectral band image of the binary tree by using the high-node spectral band image of the binary tree.

[0041] In this step, such as Figure 2 As shown, in the 10-band filter array based on binary tree coating, there are 4 spectral bands, namely bands 4, 5, 6, and 9, located at node 1 / 16, which is the low node of the binary tree; and 6 spectral bands, namely bands 0, 1, 2, 3, 7, and 8, located at node 1 / 8, which is the high node of the binary tree.

[0042] Within each 4*4 window, the spectral segment of a 1 / 16 node contains only 1 pixel, while the spectral segment of a 1 / 8 node contains 2 pixels. The spatial sampling frequency of the spectral segment of a 1 / 8 node is twice that of the spectral segment of a 1 / 16 node. Therefore, the spectral segment located at the high node of the binary tree contains more spatial information.

[0043] First, for the 1 / 16 node spectral segment, find the nearest 1 / 8 node spectral segment adjacent to this segment (e.g., ...). Figure 2 In the image, spectral band 4 corresponds to spectral band 3) as the reference image. Fourier transform is used to transform the de-mosaic reconstructed images corresponding to the two spectral bands to the frequency domain, as shown below:

[0044]

[0045] This is a spectral segment image of 1 / 8 nodes of a binary tree; This is a spectral segment image of 1 / 16 nodes of a binary tree; FFT represents Fourier transform. for The Fourier transform result; for The Fourier transform result;

[0046] After multiplying the frequency domain image by a low-pass filter D (such as a rectangle or circle), and then performing an inverse Fourier transform on the result, a spatial low-frequency image of two spectral bands is obtained, represented as:

[0047]

[0048] This represents the spatial low-frequency image of the spectral segment of a binary tree with 1 / 8 nodes. Represents the spatial low-frequency image of the spectral segment of a binary tree at node 1 / 16; FFT -1 Indicates the inverse Fourier transform;

[0049] Next, calculate the difference between the spatial low-frequency images, add the difference result to the binary tree high-node reference image, and obtain the enhanced binary tree low-node spectral band image, represented as:

[0050]

[0051] This represents the enhanced binary tree spectral segment image of 1 / 16 nodes.

[0052] In practice, in addition to frequency domain processing, a low-pass convolution kernel (such as mean filtering, Gaussian filtering, etc.) can be directly convolved with the image to obtain a blurred low-frequency image.

[0053] In addition to Fourier transform, other methods for converting images to the frequency domain include wavelet transform and shear wave transform. The purpose of this part of the processing is to transform to the frequency domain and filter out high-frequency information to obtain low-frequency images.

[0054] Based on the above method, this embodiment of the invention also provides a snapshot-type multispectral camera demosaic device, the device comprising:

[0055] The pseudo-panchromatic estimation image acquisition unit is used to extract multiple single-band mosaic images according to the spectral bands based on the original multispectral image, and to estimate the pseudo-panchromatic estimation image based on the original multispectral image.

[0056] The demosaic reconstruction result acquisition unit is used to calculate the difference between the sparse pseudo-panchromatic estimated image of each spectral band and the original sparse spectral band image. It selects the corresponding weighted interpolation filter kernel according to the spectral band at different nodes and convolves it with the difference result to calculate the difference at all pixel positions. Then, it adds the filtered and interpolated difference image with the pseudo-panchromatic estimated image to obtain the demosaic reconstruction result based on the pseudo-panchromatic estimated image.

[0057] The binary tree low-node spectral band image enhancement unit is used to enhance the spatial high-frequency information of the binary tree low-node spectral band image based on the high-node spectral band image of the binary tree by utilizing the color and spatial correlation between neighboring spectral bands and the demosaic reconstruction result obtained in step 2.

[0058] The specific implementation process of each unit in the above device is described in the above method embodiments.

[0059] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method.

[0060] This invention also provides a computer storage medium storing a plurality of instructions adapted for loading and executing the method by a processor.

[0061] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0062] In summary, the method described in the embodiments of the present invention has the following advantages:

[0063] 1. For snapshot-type multispectral images of binary tree coated arrays, this invention designs a demosaic method based on pseudo-panchromatic image estimation to solve the artifact problem caused by the weak spectral correlation of wide spectral distribution filters.

[0064] 2. Based on the results of mosaic reconstruction from pseudopanchromatic images, this method utilizes the high color and spatial correlation between neighboring spectral bands and leverages the high spatial sampling rate image located in the high node spectral band of the binary tree to enhance the image detail information in the low spatial sampling rate image of the lower node spectral band, thereby reducing artifacts in the reconstructed image and improving the spatial resolution of the mosaic-reconstructed image.

[0065] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A snapshot multispectral camera demosaicing method characterized by, The method comprises: Step 1, according to the multispectral original image, a plurality of single-spectral band mosaic images are extracted according to spectral bands, and a pseudo-panchromatic estimation image is estimated according to the multispectral original image; Step 2, the difference between each spectral band sparse pseudo-panchromatic estimation image and the original spectral band sparse image is calculated, a corresponding weighted interpolation filter kernel is selected according to the spectral bands at different nodes, and the difference value at all pixel positions is calculated by convolution; then the difference image after filtering and interpolation is added to the pseudo-panchromatic estimation image to obtain a demosaicing reconstruction result based on the pseudo-panchromatic estimation image; Step 3, based on the color and spatial correlation between adjacent spectral bands, the spatial high-frequency information of the binary tree low node spectral band image is enhanced based on the binary tree high node spectral band image on the basis of the demosaicing reconstruction result obtained in step 2.

2. The snapshot multispectral camera demosaicing method of claim 1, wherein, In step 1, A multispectral raw image I is obtained by one imaging with a snapshot multispectral camera MSFA A plurality of single-spectral mosaic images are extracted separately for each spectral band The number n of each spectral band appearing in the local window of each pixel is counted to form a filter kernel P with each spectral band as the center, and each element value of the filter kernel P is set to 1 / n; The multispectral original image I is filtered again MSFA according to the spectral band of each pixel to select the corresponding filter kernel P k , k is the spectral band, and a filtering operation is performed to complete the calculation of all pixels pixel by pixel to obtain a pseudo-panchromatic estimation image I P , which is represented as: I P = I MSFA * P k .

3. The snapshot multispectral camera demosaicing method of claim 1, wherein, The process of step 2 is specifically: For each spectral band k, compute a sparse pseudo-panchromatic estimation image I P ⊙m k the difference Δ k k is expressed as:​ I k = I MSFA ⊙m k Δ k = I k - I P ⊙ m k ⊙ denotes dot product; m k is a binary mask, the mask size is consistent with the original image I MSFA , wherein the spectral band k is assigned a value of 1 at the corresponding position of the pixel in the original image, and a value of 0 at the remaining positions; I P is a pseudo full-color estimation image; For each spectral band k, the weighted interpolation filter is used to calculate the difference value at all pixel positions, which is represented as: to filter the interpolated difference image; H weighted interpolation low-pass filter kernel; Wherein, for the spectral bands at different nodes of the binary tree, the weighted interpolation low-pass filter kernel H is different; Finally, the difference image after filtering and interpolation is added to the pseudo-panchromatic estimation image to obtain a demosaicing reconstruction result based on the pseudo-panchromatic estimation image, which is represented as: The pseudo-panchromatic estimate is used to estimate the demosaiced reconstruction result.

4. The snapshot multispectral camera demosaicing method of claim 1, wherein, In step 3, in the 10-spectral band filter array based on the binary tree coating, 4 spectral bands, i.e. 4, 5, 6 and 9 spectral bands, are located at 1 / 16 nodes, i.e. binary tree low nodes; 6 spectral bands, i.e. 0, 1, 2, 3, 7 and 8 spectral bands, are located at 1 / 8 nodes, i.e. binary tree high nodes; In every 4*4 window, the 1 / 16 node spectral band contains only one pixel, while the 1 / 8 node spectral band contains two pixels, and the spatial sampling frequency of the 1 / 8 node spectral band is twice that of the 1 / 16 node spectral band, so the spectral band located at the binary tree high node contains more spatial information; First, for the 1 / 16 node spectral band, find the nearest 1 / 8 node spectral band adjacent to the spectral band as the reference image, and use Fourier transform to transform the demosaicing reconstruction images corresponding to the two spectral bands into the frequency domain space, which is represented as: is a binary tree 1 / 8 node spectral band image; is a binary tree 1 / 16 node spectral band image; FFT denotes a Fourier transform; is a Fourier transform result of is a Fourier transform result of ​​ After multiplying the low-pass filter D with the frequency domain image, the inverse Fourier transform is performed on the result to obtain the spatial low-frequency images of the two spectral bands, which is represented as: a spatial low frequency image representing the 1 / 8 node spectrum of the binary tree; a spatial low frequency image representing the 1 / 16 node spectrum of the binary tree; FFT -1 represents an inverse Fourier transform; Then the difference between the spatial low-frequency images is calculated, and the difference result is added to the binary tree high node reference image to obtain the enhanced binary tree low node spectral band image, which is represented as: represents the enhanced binary tree 1 / 16 node spectral band image.

5. A snapshot multispectral camera demosaicing apparatus characterized by, The device comprises: A pseudo-panchromatic estimation image acquisition unit is configured to extract a plurality of single-spectral band mosaic images according to spectral bands according to a multispectral original image, and estimate a pseudo-panchromatic estimation image according to the multispectral original image; A pseudo-panchromatic estimation image acquisition unit is configured to extract a plurality of single-spectral band mosaic images according to spectral bands according to a multispectral original image, and estimate a pseudo-panchromatic estimation image according to the multispectral original image; The de-mosaicking reconstruction result acquisition unit is configured to calculate a difference value between each spectral band sparse pseudo-panchromatic estimation image and an original spectral band sparse image, select a corresponding weighted interpolation filter kernel according to spectral bands at different nodes, and convolve the difference value with the filter kernel to calculate the difference value at all pixel positions; and add the filtered and interpolated difference image to the pseudo-panchromatic estimation image to obtain a de-mosaicking reconstruction result based on the pseudo-panchromatic estimation image. The binary tree low node spectral band image enhancement unit is configured to utilize color and spatial correlation between adjacent spectral bands to enhance spatial high frequency information of binary tree low node spectral band images based on binary tree high node spectral band images on the basis of the de-mosaicking reconstruction result obtained in step 2.

6. An electronic device comprising a memory and a processor, characterized in that The memory stores a computer program, and the processor is configured to execute the computer program to perform the method in any one of claims 1 to 4.

7. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are adapted to be loaded and executed by the processor to perform the method in any one of claims 1 to 4.

Images