Snapshot multispectral camera demosaicing method, device and equipment and storage medium

Abstract

The invention discloses a snapshot multispectral camera demosaicing method and device based on a binary tree coating array, and the method comprises the steps: extracting a plurality of single-spectrum mosaic images according to a multispectral original image and a spectrum, and carrying out the estimation of a pseudo panchromatic estimation image according to the multispectral original image; calculating a difference value between each spectral band sparse pseudo panchromatic estimation image and the original spectral band sparse image, and calculating difference values at all pixel positions; adding the difference image after filtering interpolation with the pseudo panchromatic estimation image to obtain a demosaicing reconstruction result based on the pseudo panchromatic estimation image; and enhancing the spatial high-frequency information of the binary tree low-node spectral band image based on the binary tree high-node spectral band image by using the color and spatial correlation between adjacent spectral bands. According to the method, the problem of artifacts caused by weak spectrum correlation of the wide spectrum distribution optical filter can be solved, the artifacts of the reconstructed image are reduced, and the spatial resolution capability of the demosaiced image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multispectral cameras, and in particular to a snapshot multispectral camera de-mosaicing method, device, equipment and storage medium. Background Art

[0002] The snapshot multispectral camera is based on a multispectral filter array. It can obtain a mosaic original image in one imaging. It has the advantages of compact structure and can obtain all spectral images at one time. The method of designing filter arrays in mosaic form can meet the characteristics of human vision. The initial application of this method is a color digital camera, which uses a color filter array composed of three colors R, G, and B to couple with the detector, so that each pixel of the detector can only measure the spectral information of one color, and the vacancy of each color can be obtained by estimating the adjacent pixel values. Similar to the color filter array, after the multispectral filter array is coupled with the detector, each pixel of the detector can only measure the spectral information of one spectral band. In order to obtain the entire multispectral image, it is necessary to estimate the pixel position where the information of each spectral band is missing. This reconstruction process is usually called the demosaicing process of the multispectral filter array.

[0003] Compared with the color filter array of a digital camera, the multispectral filter array has more spectral bands, narrower bandwidth, wider spectral coverage, and more missing pixels in each spectral band. If the de-mosaicing method applied to the color filter array is directly applied to the multispectral filter array, the effect is poor. Therefore, it is necessary to study the de-mosaicing method applicable to the multispectral filter array. Another method of the prior art is applicable to a non-redundant multispectral filter array. Based on the spectral correlation and spatial correlation between the spectral bands, a pseudo-full-color intensity image is estimated, and then it is used as a reference image to reconstruct the image of each spectral band. Since there is only one band channel available on each pixel in each n*n array of the original image, it is assumed that the intensity is strongly correlated between adjacent pixels. Considering this spatial correlation, a smoothing filter M can be used to directly estimate the intensity on the original image, and the intensity level of each pixel is defined as the average level of all channels. However, this method is only applicable to non-redundant filter arrays and is not applicable to filter arrays based on binary tree design. Summary of the invention

[0004] The purpose of the present invention is to provide a snapshot multispectral camera demosaicing method, device, equipment and storage medium, which can solve the artifact problem caused by weak spectral correlation of wide spectral distribution filters, reduce reconstructed image artifacts, and improve the spatial resolution of the demosaiced image.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A snapshot multispectral camera demosaicing method, the method comprising:

[0007] Step 1: extract multiple single-spectral mosaic images according to the spectrum segment based on the multispectral original image, and estimate the pseudo-full-color estimated image based on the multispectral original image;

[0008] Step 2: Calculate the difference between the sparse pseudo-panchromatic estimated image of each spectral band and the sparse image of the original spectral band, select the corresponding weighted interpolation filter kernel according to the spectral bands at different nodes, and convolve with the difference result to calculate the difference at all pixel positions; then add the difference image after filtering and interpolation to the pseudo-panchromatic estimated image to obtain the demosaic reconstruction result based on the pseudo-panchromatic estimated image;

[0009] Step 3: Using the color and spatial correlation between adjacent spectral segments, based on the demosaic reconstruction result obtained in step 2, the spatial high-frequency information of the binary tree low-node spectral segment image is enhanced based on the binary tree high-node spectral segment image.

[0010] A snapshot multispectral camera de-mosaicing device, the device comprising:

[0011] A pseudo-panchromatic estimated image acquisition unit is used to extract a plurality of single-spectral mosaic images according to the spectrum bands based on the multi-spectral original image, and to estimate the pseudo-panchromatic estimated image based on the multi-spectral original image;

[0012] The demosaic reconstruction result acquisition unit is used to calculate the difference between the sparse pseudo-panchromatic estimation image of each spectral band and the sparse image of the original spectral band, select the corresponding weighted interpolation filter kernel according to the spectral bands at different nodes, and convolve with the difference result to calculate the difference at all pixel positions; then add the difference image after filtering and interpolation to the pseudo-panchromatic estimation image to obtain the demosaic reconstruction result based on the pseudo-panchromatic estimation image;

[0013] The binary tree low node spectrum segment image enhancement unit is used to utilize the color and spatial correlation between adjacent spectrum segments, based on the demosaic reconstruction result obtained in step 2, and based on the binary tree high node spectrum segment image to enhance the spatial high frequency information of the binary tree low node spectrum segment image.

[0014] It can be seen from the technical solution provided by the present invention that the above method and device can solve the artifact problem caused by the weak spectral correlation of the wide spectral distribution filter, reduce the reconstructed image artifacts, and improve the spatial resolution capability of the demosaiced image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0016] Figure 1 A schematic flow chart of a method for de-mosaicing a snapshot multi-spectral camera based on a binary tree coating array provided in an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of a 10-band filter array based on a binary tree design coating according to an example of the present invention. DETAILED DESCRIPTION

[0018] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] like Figure 1 The figure is a schematic flow chart of a method for de-mosaicing a snapshot multi-spectral camera based on a binary tree coating array provided by an embodiment of the present invention, the method comprising:

[0020] Step 1: extract multiple single-spectral mosaic images according to the spectrum segment based on the multispectral original image, and estimate the pseudo-full-color estimated image based on the multispectral original image;

[0021] In this step, a multispectral original image I is obtained by imaging with a snapshot multispectral camera. MSFA , extract each spectrum segment separately to obtain multiple single spectrum segment mosaic images;

[0022] Count the number of times n that each spectrum segment appears in the local window of each pixel, and construct a filter kernel P centered on each spectrum segment. The value of each element of the filter kernel P is set to 1 / n.

[0023] Then for the multispectral original image I MSFA Perform filtering operations and select the corresponding filter kernel P according to the spectrum of each pixel k , k is the spectrum segment, and filtering operation is performed, and the calculation of all pixels is completed pixel by pixel to obtain the pseudo full color estimation image I P , expressed as:

[0024] I P =I MSFA *Pk .

[0025] For example, take a 10-band multispectral filter array based on a binary tree design as an example. Figure 2 The figure shows a schematic diagram of a 10-band filter array based on a binary tree design coating according to an example of the present invention, where the filter kernels P of the 0 to 9 bands are respectively:

[0026]

[0027] Step 2: Calculate the difference between the sparse pseudo-panchromatic estimated image of each spectral band and the sparse image of the original spectral band, select the corresponding weighted interpolation filter kernel according to the spectral bands at different nodes, and convolve with the difference result to calculate the difference at all pixel positions; then add the difference image after filtering and interpolation to the pseudo-panchromatic estimated image to obtain the demosaic reconstruction result based on the pseudo-panchromatic estimated image;

[0028] In this step, for each spectral segment k, the sparse pseudo-panchromatic estimation image I is calculated for each spectral segment k. P ⊙m k With the original spectral sparse image I k The difference Δ k , expressed as:

[0029] I k =I MSFA ⊙m k

[0030] Δ k =I k -I P ⊙m k

[0031] ⊙ represents dot product; m k is a binary mask, the mask size is the same as the original image I MSFA The spectrum segment k is assigned a value of 1 at the corresponding position of the pixel in the original image, and 0 at the other positions; P is the pseudo-panchromatic estimated image;

[0032] For each spectral segment k, a weighted interpolation filter is used to calculate the difference at all pixel positions, expressed as:

[0033]

[0034] is the difference image after filtering and interpolation; H is the weighted interpolation low-pass filter kernel;

[0035] For different node spectrum segments in the binary tree, the weighted interpolation low-pass filter kernel H is different. Figure 2 Take as an example, the weighted interpolation low-pass filter kernel H is:

[0036]

[0037] Finally, the difference image after filtering and interpolation is added to the pseudo-panchromatic estimation image to obtain the demosaic reconstruction result based on the pseudo-panchromatic estimation image, which is expressed as:

[0038]

[0039] This is the demosaic reconstruction result based on the pseudo-panchromatic estimation image. Since the estimation of the pseudo-panchromatic estimation image is obtained by convolving the mean filter with the original mosaic image, this operation will lose part of the spatial high-frequency information in the original image.

[0040] Step 3: Using the color and spatial correlation between adjacent spectral segments, based on the demosaic reconstruction result obtained in step 2, the spatial high-frequency information of the binary tree low-node spectral segment image is enhanced based on the binary tree high-node spectral segment image.

[0041] In this step, if Figure 2 As shown, in the 10-band filter array based on binary tree coating, there are 4 bands, namely, bands 4, 5, 6, and 9, located at the 1 / 16 node, i.e., the low node of the binary tree; there are 6 bands, namely, bands 0, 1, 2, 3, 7, and 8, located at the 1 / 8 node, i.e., the high node of the binary tree;

[0042] In each 4*4 window, the spectrum segment of the 1 / 16 node contains only 1 pixel, while the spectrum segment of the 1 / 8 node contains 2 pixels. The spatial sampling frequency of the spectrum segment of the 1 / 8 node is twice that of the spectrum segment of the 1 / 16 node. Therefore, the spectrum segment at the high node of the binary tree contains more spatial information.

[0043] First, for the 1 / 16 node spectrum segment, find the 1 / 8 node spectrum segment closest to the spectrum segment (such as Figure 2 In the figure, spectrum segment 4 corresponds to spectrum segment 3) as the reference image, and Fourier transform is used to transform the demosaiced reconstructed images corresponding to the two spectrum segments into the frequency domain space, which is expressed as:

[0044]

[0045] It is the spectrum image of 1 / 8 node of binary tree; It is the spectrum image of 1 / 16 node of binary tree; FFT means Fourier transform; for The Fourier transform result of for The Fourier transform result of

[0046] After multiplying the frequency domain image with a low-pass filter D (such as a rectangular or circular filter), the result is inverse Fourier transformed to obtain the spatial low-frequency image of the two spectral bands, which is expressed as:

[0047]

[0048] Represents the spatial low-frequency image of the 1 / 8 node spectrum of the binary tree; Represents the spatial low-frequency image of the spectrum of the 1 / 16 node of the binary tree; FFT -1 represents inverse Fourier transform;

[0049] Then calculate the difference between the spatial low-frequency images, add the difference result to the binary tree high node reference image, and obtain the enhanced binary tree low node spectral image, which is expressed as:

[0050]

[0051] Represents the enhanced 1 / 16 node spectrum image of the binary tree.

[0052] In a specific implementation, in addition to frequency domain processing, a low-pass convolution kernel (such as mean filtering, Gaussian filtering, etc.) can also be directly used to convolve with the image to obtain a blurred low-frequency image.

[0053] In addition to Fourier transform, other methods for converting images to the frequency domain can also be wavelet transform, shearlet transform and other methods. The purpose of this part of the processing is to transform to the frequency domain and filter out high-frequency information to obtain a low-frequency image.

[0054] Based on the above method, an embodiment of the present invention further provides a snapshot multispectral camera demosaicing device, the device comprising:

[0055] A pseudo-panchromatic estimated image acquisition unit is used to extract a plurality of single-spectral mosaic images according to the spectrum bands based on the multi-spectral original image, and to estimate the pseudo-panchromatic estimated image based on the multi-spectral original image;

[0056] The demosaic reconstruction result acquisition unit is used to calculate the difference between the sparse pseudo-panchromatic estimation image of each spectral band and the sparse image of the original spectral band, select the corresponding weighted interpolation filter kernel according to the spectral bands at different nodes, and convolve with the difference result to calculate the difference at all pixel positions; then add the difference image after filtering and interpolation to the pseudo-panchromatic estimation image to obtain the demosaic reconstruction result based on the pseudo-panchromatic estimation image;

[0057] The binary tree low node spectrum segment image enhancement unit is used to utilize the color and spatial correlation between adjacent spectrum segments, based on the demosaic reconstruction result obtained in step 2, and based on the binary tree high node spectrum segment image to enhance the spatial high frequency information of the binary tree low node spectrum segment image.

[0058] The specific implementation process of each unit in the above device is described in the above method embodiment.

[0059] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the method.

[0060] An embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method.

[0061] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field.

[0062] In summary, the method described in the embodiment of the present invention has the following advantages:

[0063] 1. For the snapshot multispectral image of the binary tree coating array, the present invention designs a demosaicing method based on pseudo-panchromatic image estimation to solve the artifact problem caused by the weak spectral correlation of the wide spectral distribution filter;

[0064] 2. Based on the demosaicing reconstruction results of pseudo-full-color images, this method uses the high spatial sampling rate images located in the high-node spectral segments of the binary tree based on the high color correlation and spatial correlation between adjacent spectral segments to enhance the image detail information in the low spatial sampling rate images in the lower-node spectral segments, so as to reduce the reconstructed image artifacts and improve the spatial resolution of the demosaiced image.

[0065] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A snapshot multispectral camera demosaicing method, characterized in that: The method comprises: Step 1: extract multiple single-spectral mosaic images according to the spectrum bands based on the multispectral original image, and estimate a pseudo-full-color estimated image based on the multispectral original image; Step 2: Calculate the difference between the sparse pseudo-panchromatic estimated image of each spectral band and the sparse image of the original spectral band, select the corresponding weighted interpolation filter kernel according to the spectral bands at different nodes, and convolve with the difference result to calculate the difference at all pixel positions; then add the difference image after filtering and interpolation to the pseudo-panchromatic estimated image to obtain the demosaic reconstruction result based on the pseudo-panchromatic estimated image; Step 3: Using the color and spatial correlation between adjacent spectral segments, based on the demosaic reconstruction result obtained in step 2, the spatial high-frequency information of the binary tree low-node spectral segment image is enhanced based on the binary tree high-node spectral segment image.

2. The method for de-mosaicing a snapshot multispectral camera according to claim 1, characterized in that: In step 1, The multispectral original image I is obtained by imaging with a snapshot multispectral camera MSFA , extract each spectrum segment separately to obtain multiple single spectrum segment mosaic images; Count the number of times n that each spectrum segment appears in the local window of each pixel, and construct a filter kernel P centered on each spectrum segment. The value of each element of the filter kernel P is set to 1 / n; Then for the multispectral original image I MSFA Perform filtering operations and select the corresponding filter kernel P according to the spectrum of each pixel k , k is the spectrum segment, and filtering operation is performed, and the calculation of all pixels is completed pixel by pixel to obtain the pseudo full color estimation image I P , expressed as: I P =I MSFA *P k 。 3. The method for de-mosaicing a snapshot multispectral camera according to claim 1, characterized in that: The process of step 2 is specifically as follows: For each spectral segment k, calculate the sparse pseudo-panchromatic estimation image I under each spectral segment k P ⊙m k With the original spectral sparse image I k The difference Δ k , expressed as: I k =I MSFA ⊙m k Δ k =I k -I P ⊙m k ⊙ represents dot product; m k is a binary mask, the mask size is the same as the original image I MSFA The spectrum segment k is assigned a value of 1 at the corresponding position of the pixel in the original image, and 0 at the other positions; P is the pseudo-panchromatic estimated image; For each spectral segment k, a weighted interpolation filter is used to calculate the difference at all pixel positions, expressed as: is the difference image after filtering and interpolation; H is the weighted interpolation low-pass filter kernel; Among them, for different node spectrum segments in the binary tree, the weighted interpolation low-pass filter kernel H is different; Finally, the difference image after filtering and interpolation is added to the pseudo-panchromatic estimation image to obtain the demosaic reconstruction result based on the pseudo-panchromatic estimation image, which is expressed as: This is the demosaic reconstruction result based on the pseudo-panchromatic estimation image.

4. The method for de-mosaicing a snapshot multispectral camera according to claim 1, characterized in that: In step 3, in the 10-band filter array based on the binary tree coating, there are 4 bands, namely, bands 4, 5, 6, and 9, located at the 1 / 16 node, i.e., the low node of the binary tree; there are 6 bands, namely, bands 0, 1, 2, 3, 7, and 8, located at the 1 / 8 node, i.e., the high node of the binary tree; In each 4*4 window, the spectrum segment of the 1 / 16 node contains only 1 pixel, while the spectrum segment of the 1 / 8 node contains 2 pixels. The spatial sampling frequency of the spectrum segment of the 1 / 8 node is twice that of the spectrum segment of the 1 / 16 node. Therefore, the spectrum segment at the high node of the binary tree contains more spatial information. First, for the 1 / 16 node spectrum segment, find the 1 / 8 node spectrum segment closest to the spectrum segment as the reference image, and use Fourier transform to transform the demosaiced reconstructed images corresponding to the two spectrum segments into the frequency domain space, which is expressed as: It is the spectrum image of 1 / 8 node of binary tree; It is the spectrum image of 1 / 16 node of binary tree; FFT means Fourier transform; for The Fourier transform result of for The Fourier transform result of After multiplying the frequency domain image with the low-pass filter D, the result is inverse Fourier transformed to obtain the spatial low-frequency image of the two spectral bands, which is expressed as: Represents the spatial low-frequency image of the 1 / 8 node spectrum of the binary tree; Represents the spatial low-frequency image of the spectrum of the 1 / 16 node of the binary tree; FFT -1 represents inverse Fourier transform; Then calculate the difference between the spatial low-frequency images, add the difference result to the binary tree high node reference image, and obtain the enhanced binary tree low node spectral image, which is expressed as: Represents the enhanced 1 / 16 node spectrum image of the binary tree.

5. A snapshot multispectral camera de-mosaicing device, characterized in that: The device comprises: A pseudo-panchromatic estimated image acquisition unit is used to extract a plurality of single-spectral mosaic images according to the spectrum bands based on the multi-spectral original image, and to estimate the pseudo-panchromatic estimated image based on the multi-spectral original image; The demosaic reconstruction result acquisition unit is used to calculate the difference between the sparse pseudo-panchromatic estimation image of each spectral band and the sparse image of the original spectral band, select the corresponding weighted interpolation filter kernel according to the spectral bands at different nodes, and convolve with the difference result to calculate the difference at all pixel positions; then add the difference image after filtering and interpolation to the pseudo-panchromatic estimation image to obtain the demosaic reconstruction result based on the pseudo-panchromatic estimation image; The binary tree low node spectrum segment image enhancement unit is used to utilize the color and spatial correlation between adjacent spectrum segments, based on the demosaic reconstruction result obtained in step 2, and based on the binary tree high node spectrum segment image to enhance the spatial high frequency information of the binary tree low node spectrum segment image.

6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 4.

7. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 4.

Images