A method for demosaicking one-to-many infrared multispectral images

Abstract

The present invention discloses a method for one-to-many infrared multi-spectral image demosaicking. First, a pseudo-panchromatic image is obtained from the mosaic image to obtain a spectral prior. The Pearson correlation coefficient is used to calculate the correlation between each band image and the pseudo-panchromatic image. If the correlation is strong, the pseudo-panchromatic image is used as the spectral prior. A residual model based on the spectral prior is established, and the original infrared multi-spectral image and the pseudo-panchromatic image are respectively sparsified in the k band, and differential processing is performed on them to obtain a sparse residual map. The progressive demosaicking method uses the adjacent pixel gradient information to transform the sparse residual map into a full-resolution residual map. By compensating the residual map, a k-band full-resolution cube based on the spectral prior is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of image technology, and particularly relates to an image demosaicing method. Background Art

[0002] In order to reduce the volume and cost, the surface of the sensor of an infrared multi-spectral camera is mostly a filter array structure. For an imaging device with such a structure, only the light intensity information of one band can be collected at a pixel point, and the rest of the channels need to be estimated and interpolated. This estimation and interpolation process is called demosaicing. Demosaicing is a key step in the imaging of an infrared multi-spectral camera and one of the important factors for evaluating the imaging device. At the same time, it also affects other computer vision tasks.

[0003] Since the filter arrays of infrared multi-spectral cameras are mostly two-dimensional grating and metasurface array structures, their true spectral characteristics can only be obtained within an infinite period. Therefore, when designing the structure, it is expected that the area covered by a single band in the array is as large as possible, so that its spectral characteristics will be more accurate. As the area covered by a single band increases, the number of pixels it covers will also increase accordingly. Therefore, there is a situation where a single band corresponds to multiple pixels, which is called a "one-to-many" infrared multi-spectral image.

[0004] "One-to-one" refers to a multi-spectral image where a single filter corresponds to a single pixel. For a "one-to-one" multi-spectral image, usually the surrounding pixel information is used to estimate the pixel with the missing pixel value. This method only simply uses the spatial information between pixels for estimation and recovery, which will not only lose the high-frequency information of the image, but also waste the information between spectra. Due to the structural characteristics of the "one-to-many" image, the pixel span between the same bands is relatively large, and the traditional demosaicing algorithm for "one-to-one" cannot be used. It is necessary to combine its "one-to-many" characteristics and spectral correlation to perform a more effective demosaicing algorithm.

[0005] In summary, it is necessary to propose a demosaicing method based on "one-to-many" infrared multi-spectral images by using their spatial information and spectral correlation according to the "one-to-many" characteristics of the images. Summary of the Invention

[0006] To overcome the deficiencies of the prior art, the present invention provides a one-to-many infrared multi-spectral image demosaicing method. First, a pseudo-panchromatic image is obtained from the mosaic image to obtain a spectral prior. The Pearson correlation coefficient is used to calculate the correlation between each band image and the pseudo-panchromatic image. If the correlation is strong, the pseudo-panchromatic image is used as the spectral prior. A residual model based on the spectral prior is established, and the original infrared multi-spectral image and the pseudo-panchromatic image are respectively sparsified in the k band, and differential processing is performed on them to obtain a sparse residual map. The progressive demosaicing method uses the gradient information of adjacent pixels to transform the sparse residual map into a full-resolution residual map. By compensating the residual map, a k-band full-resolution cube based on the spectral prior is obtained.

[0007] The technical solution adopted by the present invention to solve its technical problems includes the following steps:

[0008] Step 1: Use the original infrared multi-spectral mosaic image to obtain a pseudo-panchromatic image as the spectral prior; sparsify the original infrared multi-spectral mosaic image and the pseudo-panchromatic image in K channels respectively, and perform differential processing to obtain K-channel sparse residual maps;

[0009] Step 2: Use the sparse residual map for progressive demosaicing guided by the gradient of the pixel values of adjacent pixels in the same band, further complete the sparse residual map, and obtain K-channel full-resolution residual maps; then fuse the full-resolution residual maps with the pseudo-panchromatic image obtained in Step 1 to estimate the K-channel full-resolution spectral cube;

[0010] Step 3: Sparsify the original infrared multi-spectral mosaic image in K channels, use the Euclidean distance as the judgment basis, and use a dilated convolution kernel to restore the sparse k-band image to a full-resolution image, and estimate the K-channel full-resolution spectral cube;

[0011] Step 4: Fuse the two sets of K-channel full-resolution spectral cubes estimated in Step 2 and Step 3 to obtain the final K-channel spectral cube.

[0012] Further, the specific method of Step 1 is: use the original infrared multi-spectral mosaic image to obtain a pseudo-panchromatic image, where the pixel value corresponding to each pixel is composed of the mean value of the pixel in all channels; through the determination of the Pierce correlation coefficient, the mean value of the correlation coefficients between the pseudo-panchromatic image and each band is 0.9756, and the pseudo-panchromatic image is used as the spectral reference map for each band, and a residual model of the spectral prior and the K-channel sparse image is established;

[0013] Pseudo-panchromatic map I M is defined as the value of any pixel p in the pseudo-panchromatic map being the mean value of the pixels of the pixel in all bands:

[0014]

[0015] Estimate the pseudo-panchromatic image using a fixed convolution kernel M, which is implemented by means of dilated convolution. The specific implementation method is represented by the following formula:

[0016] I M = I MSFA * M

[0017]

[0018] where I MSFA represents the infrared multi-spectral mosaic image;

[0019] Build a residual model based on spectral prior, sparsify the original infrared multi-spectral mosaic image I sparse and the pseudo-panchromatic image respectively in k channels, and perform differential processing on them to obtain a sparse residual map

[0020]

[0021] Furthermore, for the progressive demosaicing, first complete the pixel values corresponding to the intermediate pixels to be solved in the true value part to obtain Use the true value and complete the pixel values corresponding to the intermediate pixels to be solved in to obtain Use the value of to complete the pixel values corresponding to the remaining pixels to be solved to obtain Then, use the horizontal gradient, vertical gradient, and diagonal gradient of the image as weights to guide the progressive restoration of the entire image;

[0022] For the vertical gradient, it is specifically expressed as follows:

[0023]

[0024] δx, δy = {±1, ±2}

[0025] After demosaicing, the obtained is of full resolution Since the restored result still contains the information of the pseudo-panchromatic image, the corresponding information needs to be removed:

[0026]

[0027] For the demosaicking method based on one-to-many spatial features, it is first necessary to obtain a sparse K-channel spectral cube from the infrared multispectral mosaic image, and use the sparse single-band information to restore the spatial information. Taking the Euclidean distance as the judgment basis, combined with the one-to-many characteristics, the full-resolution spectral cube of K channels is estimated by means of dilated convolution, which is specifically expressed as follows:

[0028] I Spa_k = I MSFA * H

[0029] F = [1 0 0 2 0 0 3 0 0 2 0 0 1]

[0030] H = F T × F

[0031] Furthermore, step 4 is specifically expressed as:

[0032] Fuse I Spe_k obtained by the progressive demosaicking method based on spectral prior and I Spa_k obtained by the demosaicking method based on one-to-many spatial features, and finally obtain the spectral cube I k in the k band:

[0033] I k = λ1 × I Spe_k + λ2 × I Spa_k

[0034] where λ1 and λ2 are the fusion coefficients of the two spectral cubes respectively;

[0035] The beneficial effects of the present invention are as follows:

[0036] For the progressive demosaicking model based on spectral prior of the present invention, first, a pseudo-panchromatic image is created as the spectral prior, and a residual model is created to obtain more accurate high-frequency image information, laying a good foundation for the subsequent restoration of small features in a single band. By stepwise filling the pixel values of the pixels to be solved in the same band and using the gradient information of adjacent pixels in the same band, the problem that the single-band information span is too large due to "one-to-many" and the pixel values of the pixels to be solved cannot be accurately estimated is solved. For the demosaicking model based on the "one-to-many" spatial features, taking the Euclidean distance as the judgment basis and combining the "one-to-many" characteristics, dilated convolution is used to avoid the formation of pseudo-mosaics and obtain more accurate low-frequency image information. The two models provide strong guarantees for demosaicking from both high-frequency and low-frequency levels. Description of the Drawings

[0037] Figure 1 It is the infrared multispectral mosaic image of the embodiment of the present invention.

[0038] Figure 2This is the overall flowchart for removing mosaics from the infrared multi-spectral images of the present invention.

[0039] Figure 3 This is the pseudo-panchromatic map model of the present invention.

[0040] Figure 4 This is the schematic diagram of the progressive restoration method of the present invention, (a) progressive interpolation method; (b) gradient guidance method;

[0041] Figure 5 This is the schematic diagram of the "one-to-many" spatial feature de-mosaicking model of the present invention. Detailed implementation manners

[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0043] The object of the present invention is to provide a de-mosaicking method applicable to "one-to-many" infrared multi-spectral images, which restores a single mosaic image containing information of K bands into a full-resolution spectral map corresponding to K bands.

[0044] The present invention adopts the following technical solutions: a de-mosaicking method for "one-to-many" infrared multi-spectral images, establishing a spatial-spectral progressive de-mosaicking model and a de-mosaicking model based on the "one-to-many" characteristics.

[0045] First, obtain the spectral prior from the mosaic image, design a residual model with k sparse-channel images based on the spectral prior, and perform progressive de-mosaicking based on the algorithm obtained from this residual model to obtain the high-frequency information of the image. At the same time, design a de-mosaicking model based on spatial information for the "one-to-many" characteristics to obtain the low-frequency information of the image.

[0046] According to the spatial-spectral progressive de-mosaicking model and the de-mosaicking model for the "one-to-many" characteristics, obtain two sets of full-resolution spectral cubes, and fuse the data of the two sets of spectral cubes to restore the full-resolution spectral image information corresponding to K channels.

[0047] A de-mosaicking method applicable to one-to-many infrared multi-spectral images includes the following steps:

[0048] Step 1: Use the infrared multi-spectral mosaic image to obtain a pseudo-panchromatic image as the spectral prior, which is the key reference information for subsequent de-mosaicking tasks. Perform sparsification on the original infrared multi-spectral image and the pseudo-panchromatic image for K channels respectively, and perform differential processing to obtain K-channel sparse residual maps;

[0049] Step 2: Perform progressive demosaicking using the K-channel sparse residual maps obtained in Step 1. By utilizing the gradient information of adjacent pixels in the same band, the sparse residual maps are further completed to obtain full-resolution residual maps for K channels; then, the full-resolution residual maps are fused with the pseudo-panchromatic map obtained in Step 1 to estimate the full-resolution spectral cubes for K channels.

[0050] Step 3: Sparse the original infrared multi-spectral image for K channels, and use the dilated convolution kernel based on the "one-to-many" characteristic to estimate the full-resolution spectral cubes for K channels.

[0051] Step 4: Fuse the two sets of full-resolution spectral cubes for K channels estimated in Step 2 and Step 3 to obtain the final spectral cubes for K channels.

[0052] The specific method of Step 1 is as follows: Use the original infrared multi-spectral mosaic image to obtain a pseudo-panchromatic image, where the pixel value corresponding to each pixel is composed of the mean value of the pixel in all channels. Through the determination of the Pearson correlation coefficient, the average correlation coefficient between the full pseudo-color map and each band is 0.9756, and there is a strong spectral correlation with each band. The pseudo-panchromatic map can be used as the spectral reference map for each band, and a residual model of the spectral prior and the K-channel sparse image is established.

[0053] The specific method of Step 2 is as follows: Due to the "one-to-many" single-band sparse image, the pixel interval span in the same band is too large. Taking the k-th band as an example, the progressive demosaicking method uses the gradient information of adjacent pixels in the same band to change its sparse residual map into a full-resolution residual map to obtain the high-frequency information of the image.

[0054] The specific method of Step 3 is as follows: Only use the spatial information of the k-th band image, take the Euclidean distance as the judgment basis, combine the "one-to-many" characteristic, and use the dilated convolution method to restore the sparse k-th band map to a full-resolution map to obtain the low-frequency information of the image. Specific embodiment:

[0056] The present invention provides a demosaicking method applicable to "one-to-many" infrared multi-spectral images. The "one-to-many" infrared multi-spectral mosaic image collected by an infrared multi-spectral focal plane camera is as Figure 1 shown. Each period of the filter contains 9 bands, corresponding to 3×3 pixels in a single band, and the 9-band pixels are arranged periodically. The overall demosaicking process is as Figure 2As shown in the figure, first, a spectral prior is obtained by using a mosaic image, a residual model between the spectral prior and the sparse images of K channels is established, and spatio-spectral joint progressive demosaicking is performed based on the results obtained from the residual model. During the demosaicking process, not only spectral information is used, but also residual information and spatial information are utilized to better obtain the high-frequency features of the image. A demosaicking model based on spatial information is designed for the "one-to-many" characteristic to obtain the low-frequency information of the image. By fusing the image data obtained from the above model, a full-resolution spectral cube corresponding to K channels can be obtained.

[0057] The method is specifically implemented according to the following steps:

[0058] 1. First, by using the original infrared multi-spectral mosaic image, a corresponding spectral prior map, i.e., a pseudo-panchromatic image, is obtained.

[0059] 2. The original infrared multi-spectral image and the pseudo-panchromatic image established in step 1 are respectively sparsified in K channels, and a residual model is established to obtain the detailed features of the image.

[0060] 3. Progressive demosaicking guided by the pixel value gradient of adjacent pixels in the same band is performed using the residual map obtained in step 2 to further complete the sparse residual map and obtain a full-resolution residual map of K channels.

[0061] 4. The full-resolution spectral cube of K channels is estimated by fusing the full-resolution residual map obtained in step 3 and the pseudo-panchromatic map obtained in step 1.

[0062] 5. The original infrared multi-spectral image is sparsified in K channels. Based on the Euclidean distance as the judgment basis and combined with the "one-to-many" characteristic, the sparse k-band map is restored to a full-resolution map by using the method of dilated convolution, and the full-resolution spectral cube of K channels is estimated.

[0063] 6. The two groups of full-resolution spectral cubes of K channels obtained in step 4 and step 5 are fused to obtain the final full-resolution spectral cube.

[0064] In the technical solution of the present invention, the demosaicking of the infrared multi-spectral image mainly includes a progressive demosaicking method based on spatio-spectral joint and a demosaicking method based on the "one-to-many" spatial feature.

[0065] For the progressive demosaicking method based on spatio-spectral joint, first, a spectral prior needs to be obtained, that is, a pseudo-panchromatic image is obtained. The definition of the pseudo-panchromatic image I M is that the value of any pixel p in the pseudo-panchromatic image is the average value of the pixels of that pixel in all bands.

[0066]

[0067] Since each pixel in the infrared multi-spectral mosaic image has a single channel value, it is difficult to obtain the mean value of any pixel p in all bands. However, according to the spatial correlation, it can be inferred that any pixel has a strong correlation with its adjacent pixels. Therefore, a fixed convolution kernel is used to estimate the full pseudo-color map. Due to the "one-to-many" characteristic of the infrared multi-spectral mosaic image, the method of dilated convolution is selected for implementation, as Figure 3 shown.

[0068] The specific implementation method can be expressed by the following formula:

[0069] I M = I MSFA * M

[0070]

[0071] Analyze the Pearson correlation coefficient between each single band k and the pseudo-panchromatic map. If there is a strong correlation, it can be used as the spectral prior for the k band. After calculation and analysis, the correlation coefficients between each single band and the pseudo-panchromatic map are all above 0.97, so they can be used as spectral priors.

[0072] Build a residual model based on the spectral prior, sparsify the original infrared multi-spectral image and the pseudo-panchromatic image in the k channel respectively, and perform differential processing on them to obtain a sparse residual map

[0073]

[0074] Progressive demosaicking. First, complete the pixel values corresponding to the intermediate pixels to be solved in the true value part to obtain Use the true value and to complete the pixel values corresponding to the intermediate pixels to be solved to obtain Use to complete the pixel values corresponding to the remaining pixels to be solved to obtain Then, use the horizontal gradient, vertical gradient, and diagonal gradient of the image as weights to guide the progressive restoration of the entire image. The progressive restoration steps are as shown in Figure 4 (a), and the weight guidance is as shown in Figure 4 (b).

[0075] Taking the vertical gradient as an example, it is specifically expressed as follows:

[0076]

[0077] δx, δy = {±1, ±2}

[0078] The obtained after demosaicking is of full resolution Since the reply result still contains information about the pseudo - full - color image, the corresponding information needs to be removed.

[0079]

[0080] For the demosaicking method based on the "one - to - many" spatial feature, first, a sparse K - channel spectral cube needs to be obtained from the infrared multi - spectral mosaic image, and the spatial information is restored using the sparse single - band information. Taking the Euclidean distance as the judgment basis and combining the "one - to - many" characteristic, the full - resolution spectral cubes of K channels are estimated by means of dilated convolution as Figure 5 shown. The specific representation is as follows:

[0081] I Spa_k = I MSFA * H

[0082] F = [1 0 0 2 0 0 3 0 0 2 0 0 1]

[0083] H = F T × F

[0084] The I Spe_k obtained by the progressive demosaicking method based on spectral prior and the I Spa_k obtained by the demosaicking method based on the "one - to - many" spatial feature are fused, and finally the spectral cube I k of the k - band is obtained.

[0085] I k = λ1 × I Spe_k + λ2 × I Spa_k .

Claims

1. A method for demosaicking one-to-many infrared multispectral images, characterized in that, The following steps are involved: Step 1: Use the original infrared multispectral mosaic image to obtain a pseudo-panchromatic image as a spectral prior; perform K-channel sparse processing on the original infrared multispectral mosaic image and the pseudo-panchromatic image respectively, and perform difference processing to obtain K-channel sparse residual images; Step 2: Use the sparse residual map to perform progressive demosaicing guided by the gradient of the pixel values of adjacent pixels in the same band, further complete the sparse residual map, and obtain a full-resolution residual map of K channels; then use the full-resolution residual map to fuse with the pseudo-panchromatic image obtained in step 1 to estimate the full-resolution spectral cube of K channels; Step 3: Sparse the K channels of the original infrared multispectral mosaic image, and use the Euclidean distance as the basis for judgment. Use the dilated convolution kernel to restore the sparse k-band image to the full-resolution image, and estimate the full-resolution spectral cube of K channels; Step 4: Fuse the two sets of K-channel full-resolution spectral cubes estimated in steps 2 and 3 to obtain the final K-channel spectral cube.

2. A method for demosaicking a pair of multi - infrared multi - spectral images according to claim 1, characterized in that, The specific method of step 1 is as follows: using the original infrared multispectral mosaic image to obtain a pseudo-panchromatic image, wherein the pixel value corresponding to each pixel is composed of the average value of the pixel over all channels; judging by the Pierce correlation coefficient, the average correlation coefficient between the pseudo-panchromatic image and each band is 0.9756, using the pseudo-panchromatic image as a spectral reference for each band, and establishing a residual model between the spectral prior and the K-channel sparse image; Pseudo-panchromatic image I M is defined as the value of any pixel p in the pseudo-panchromatic image being the average value of the pixel in all bands: A fixed convolution kernel M is used to estimate the pseudo full color image, which is implemented by dilated convolution. The specific implementation method is expressed by the following formula: I M = I MSFA * M where I MSFA represents an infrared multi-spectral mosaic image; Build a residual model based on spectral prior for the original infrared multi-spectral mosaic image I sparse And perform k-channel sparsification on the pseudo-panchromatic image respectively, and perform differential processing on it to obtain a sparse residual map 3. A method for demosaicking a pair of multi-infrared and multi-spectral images according to claim 1, characterized in that, The progressive demosaicing first completes the pixel values corresponding to the intermediate pixels to be solved in the true value part of to obtain Using the true value and the pixel values corresponding to the intermediate pixels to be solved in are completed to obtain Using the values to complete the pixel values corresponding to the remaining pixels to be solved to obtain Then, the horizontal gradient, vertical gradient, and diagonal gradient of the image are respectively used as weight guidance to progressively restore the entire image; For the vertical gradient, it is specifically expressed as follows: δx,δy={±1,±2} Obtained after demosaicing It is fully resolved Since the response result still contains pseudo-full-color image information, the corresponding information needs to be removed: The demosaicing method based on one-to-many spatial features first needs to obtain a sparse K-channel spectral cube through the infrared multispectral mosaic image, and use the sparse single-band information to restore the spatial information; based on the Euclidean distance and combined with the one-to-many characteristics, the full-resolution spectral cube of K channels is estimated by using the dilated convolution method, which is specifically expressed as follows: I Spa_k = I MSFA * H F＝[1 0 0 2 0 0 3 0 0 2 0 0 1] H=F T ×F。 4. A method for demosaicking a pair of multi - infrared multi - spectral images according to claim 1, characterized in that, The step 4 is specifically expressed as follows: The I obtained by the progressive demosaicking method based on spectral prior Spe_k is fused with the I obtained by the demosaicking method based on one-to-many spatial features Spa_k to finally obtain the spectral cube I in the k band k : I k =λ1×I Spe_k +λ2×I Spa_k Among them, λ1 and λ2 are the fusion coefficients of the two spectral cubes respectively.

Images