Color computational ghost imaging method based on plug-and-play generalized alternating projection algorithm

Abstract

A color computational ghost imaging method based on a plug-and-play generalized alternating projection algorithm comprises the following steps: (1) measuring the response coefficients of a single pixel detector to three colors: red, green and blue; (2) randomly generating a binary matrix, and using a Bayer matrix to synthesize a color illumination pattern, and projecting it onto a scene to be measured through a projector; (3) digitizing the collected light intensity; (4) dividing a forward physical model and an image prior into two independent modules; (5) pre-training a denoising neural network and a demosaicing neural network; (6) using the trained denoising neural network in the image prior module, and embedding the demosaicing neural network into the iterative process of image reconstruction, correcting the mosaic image of the reconstructed scene, and outputting the reconstruction result of the color scene after the iteration is completed. The present invention can restore a high-fidelity color image at a low sampling rate, and has the advantages of being easy to implement, having good robustness and strong generalization ability.

Description

Technical Field

[0001] The invention relates to a method for color computational ghost imaging, belonging to the technical field of computational optical imaging. Background Art

[0002] As an information carrier with convenient storage, transmission and high visualization, images have become an indispensable part of human life. Currently, one of the main means of acquiring images is imaging. Ghost imaging is a typical computational imaging technology, which realizes the imaging of the scene by encoding and sampling the scene and corresponding reconstruction calculation. Since the Computational Ghost Imaging (CGI) scheme has the advantages of low cost and can work in invisible wavelengths and low light conditions, it has received widespread attention and development since it was proposed.

[0003] However, the current CGI scheme still has some shortcomings. First, if a random matrix is ​​used to encode and sample the scene, CGI based on the association reconstruction method often requires several times the number of samples of the original image pixels to obtain a high signal-to-noise ratio image, which greatly limits the development of CGI applications. Although the later compressed sensing-based CGI uses the sparse prior information of natural scenes, allowing the CGI reconstruction system to break through the limitations of the Nyquist sampling theorem and achieve image reconstruction under undersampling, the computational overhead and recovery time required for reconstruction cannot be ignored.

[0004] In recent years, deep learning-based methods have been proposed to further improve the information acquisition capability, imaging speed, imaging function and imaging quality of CGI systems in a data-driven way. However, it should be pointed out that deep learning-based methods are highly dependent on data sets, and the quality of data sets determines the accuracy of CGI reconstruction. However, in practice, the data collection process is complicated and requires a lot of time and effort. Secondly, the generalization ability of the network after training is also very limited.

[0005] At the same time, most of the above methods focus on the reconstruction of grayscale images and lack color information that is sensitive to human vision. For color CGI, one of the most direct methods is to obtain the measurement values ​​of the three color channels of red, green, and blue respectively, and then superimpose them after three monochrome reconstructions to obtain a color image. However, this method is very inefficient and difficult to implement. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the prior art and to provide a color computational ghost imaging (CGI) method based on a Plug-And-Play Generalized Alternating Projection (PnP-GAP) algorithm, which can achieve high-fidelity color image reconstruction at a low sampling rate.

[0007] To achieve the above object, the color computational ghost imaging method based on the plug-and-play generalized alternating projection algorithm of the present invention comprises the following steps:

[0008] (1) Measure the response coefficient μ of a single pixel detector to red, green and blue colors j ;

[0009] (2) Randomly generate a binary matrix D i , while using the Bayer matrix S j Synthetic Color Lighting Mode P i,j , and projected onto the scene to be measured by a projector j superior;

[0010] (3) The illumination pattern modulated by the scene to be tested is collected by a single-pixel detector, and the collected light intensity Y is then i digitize (transfer to a computer);

[0011] (4) According to the PnP-GAP algorithm, an auxiliary parameter v is introduced into the color CGI reconstruction model, and the forward physical model and image prior are divided into two independent modules;

[0012] (5) pre-training a denoising neural network and a demosaicing neural network;

[0013] (6) The trained denoising neural network is used in the image prior module, and the demosaicing neural network is embedded in the iterative process of image reconstruction. Before the demosaicing operation is performed, the measured response coefficient is used to correct the mosaic image of the reconstructed scene. After the iteration is completed, the reconstruction result of the color scene is output.

[0014] The color lighting mode P in step (2) i,j The synthesis method is:

[0015] P i,j (x,y)=D i (x,y)·S j (x,y),

[0016] In the formula, i is the measurement index, i = 1, 2, ..., N, N is the total number of measurements, j = R, G, B, representing the three color channels of red, green and blue, (x, y) represents the pixel index coordinates; D i is a randomly generated binary matrix, S j is the Bayer matrix.

[0017] The light intensity Y collected in step (3) i It can be expressed as:

[0018]

[0019] In the formula: μ j is the detector’s response coefficient to different colors, P i,j For color lighting mode, O j is the scene to be measured, D i is a randomly generated binary matrix, S j is the Bayer matrix, O m A mosaic image representing the scene under test.

[0020] The color CGI formula based on the PnP-GAP algorithm in step (4) is:

[0021]

[0022] In the formula, f m represents vectorizing the mosaic image of the scene to be measured into a column vector, λ is the weight coefficient between the CGI imaging model and the regularization term, H is rearranging the two-dimensional illumination pattern into a one-dimensional row vector, and reorganizing it row by row according to the modulation order to obtain a two-dimensional measurement matrix;

[0023] Subsequently, this minimization problem can be solved by the following two sub-problems:

[0024] Update f m :

[0025] f m (k+1) =v (k) +H T (HH T ) -1 (Y-Hv (k) ),

[0026] Where the superscript k represents the number of iterations;

[0027] Update v:

[0028]

[0029] Where Φ Dm and Φ DσThey represent denoising and demosaicing algorithms respectively.

[0030] The present invention combines a randomly distributed binary matrix and a color Bayer matrix into a modulation pattern, and simultaneously encodes the spatial distribution and color information of the object into a one-dimensional light intensity sequence, which is measured by a single-pixel detector. A pre-trained deep denoising network is used in the PnP-GAP algorithm to obtain better results. In addition, a joint reconstruction and demosaicing method is developed, that is, the demosaicing algorithm is embedded in the iterative loop, and color correction is performed by measuring the response of the single-pixel detector to different colors, so that the color information of the target scene can be restored more realistically. The present invention can restore high-fidelity color images at a low sampling rate, and has the advantages of simple and easy implementation of the device, good robustness and strong generalization ability.

[0031] The present invention is characterized in that:

[0032] (1) The illumination pattern is formed based on the Bayer array and combined with the de-mosaic algorithm. There is no need for multiple monochromatic measurements. The experimental structure is simple, the cost is low, and it is easy to implement.

[0033] (2) Applying the plug-and-play generalized alternating projection algorithm to color computational ghost imaging technology can achieve high-fidelity color image reconstruction at low sampling rate.

[0034] (3) The use of deep learning-based denoising and demosaicing algorithms avoids the inapplicability of artificially selected priors. At the same time, embedding the demosaicing algorithm into the reconstruction process can reduce the loss of color during the iteration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Fig. 1 is a schematic diagram of a color computational ghost imaging experimental device in the present invention, wherein (a) is a black and white image; (b) is a color image.

[0036] Figure 2 shows the results obtained using various reconstruction algorithms at different sampling rates, where (a) is a black and white image and (b) is a color image.

[0037] Figure 3 The results of quantitative evaluation of the proposed method using different denoising algorithms are shown in Figure 2. (a) is the PNSR curve; (b) is the SSIM curve.

[0038] Figure 4 The results of quantitative evaluation of the proposed method using different de-mosaicing algorithms are shown in Figure 1. (a) is the PNSR curve; (b) is the SSIM curve. DETAILED DESCRIPTION

[0039] The present invention proposes a new algorithm for reconstructing color CGI at a low sampling rate. The projection mode adopted is formed by a random binary matrix and a color Bayer matrix, which can simultaneously measure the spatial information and color information of the scene to be tested. Subsequently, the imaging quality is further improved by integrating the deep denoising prior into the PnP-GAP framework and combining the demosaicing prior with the reconstruction process. At the same time, the response coefficient of the single-pixel detector to different colors is measured and the reconstructed mosaic image is corrected, so that the color information of the reconstructed image is more in line with the actual scene to be tested.

[0040] The present invention simplifies the experimental device, reduces the cost, and can achieve high-quality reconstruction, which will promote the application of color computational ghost imaging in practice. Figure 3 Both black and white and color images are given.

[0041] FIG1 shows the experimental setup of color computational ghost imaging (CGI) in the present invention, where (a) is a black and white image and (b) is a color image. The color illumination pattern used is a randomly generated binary (-1, 1) matrix D i and the color Bayer matrix S j The color illumination pattern is then projected onto the target scene by a projector (Panasonic 3Liquid Crystal Display). After being modulated by the object, the total light intensity Y is collected by a single pixel detector (Thorlabs PDA-100A). i . The colored letters "CGI" printed on white paper serve as the imaging scene. The data acquisition card (DAQ, NI USB-6341) digitizes the collected light intensity and transmits it to the computer. Due to the presence of negative values ​​in the modulation pattern and in order to reduce the influence of noise in the experiment, the present invention adopts a differential measurement method. Before starting the experiment, the relative response coefficient of the color CGI imaging system to the three color channels is first measured. The measurement method is to project red, green, and blue patterns of the same size and amplitude onto white paper respectively, and record the intensity of the reflected light received by the single-pixel detector. The relative response of the experimental system to different colors is μ R :μ G :μ B =1:1.572:1.158. In color CGI reconstruction, the relative response of the detector to the spectrum is used for color correction before the demosaicing process. Then the light intensity Y collected by color CGI is i It can be expressed as:

[0042]

[0043] In the formula, μ j is the detector's response coefficient to different colors, O j is the reflectivity distribution of the scene to be measured, O m represents the mosaic image of the scene to be measured, i is the measurement index, i=1,2,…,N, N is the total number of measurements, j=R,G,B, represents the three color channels, (x,y) represents the pixel index coordinates. For further simplicity, each random binary matrix can be rearranged into a row vector d i , and stacked into matrix H in projection order, which can be expressed as:

[0044] H=[d1 T ,d2 T ,…,d N T ] T , (2)

[0045] Where T represents the transpose of the matrix. The mosaic image of the scene to be tested is vectorized and rearranged into a column vector f m After that, formula (1) can be written as:

[0046] Y=Hf m , (3)

[0047] In order to more accurately limit the range of solutions, regularization terms are usually used. Therefore, the inversion problem of solving mosaic images can be modeled as:

[0048]

[0049] Wherein, “arg min” represents the process of solving the minimization problem. g(·) represents the regularization term, λ is the weight coefficient between the CGI imaging model and the regularization term. ||·||2 is the L2 norm operator.

[0050] Then, the Plug-And-Play Generalized Alternating Projection (PnP-GAP) algorithm is used to divide the forward physical model and image prior into two modules by introducing an auxiliary parameter. The image reconstruction and denoising processes are independent of each other, and the change of the prior model only involves one module. Therefore, the most advanced denoising prior model can be used to match the beneficial forward model. At the same time, the present invention embeds the demosaicing prior into the iterative process of image reconstruction, which can restore color information while reconstructing spatial information to prevent color loss.

[0051] According to the PnP-GAP architecture, formula (4) can be converted to:

[0052]

[0053] This minimization problem can be solved by the following subproblems:

[0054] Update f m :

[0055] f m (k+1) =v (k) +H T (HH T ) -1 (Y-Hv (k) ), (6)

[0056] The superscript k represents the number of iterations.

[0057] Update v:

[0058]

[0059] Where Φ Dm and Φ Dσ Denote the denoising and demosaicing algorithms respectively. In order to speed up the convergence of the algorithm, v is initialized before the iteration starts. (0) =H T Y. Because of the flexibility of the PnP-GAP algorithm, various denoising and demosaicing algorithms can be used to obtain different results. Compared with traditional denoising and demosaicing methods, methods based on deep neural networks can learn complete prior information of images from a large number of data sets during training. Therefore, the present invention pre-trains a Gaussian denoising network (FFD-Net) and a deep demosaicing network (DDN) as denoising and demosaicing priors in PnP-GAP. The neural network and reconstruction algorithm run in the environment of Python 3.7, using the Pytorch framework version 1.8. The network model is trained on a computer with an i7-9700K@3.60GHzCPU and NVIDIARTX3060GPU.

[0060] The present invention first compares the results obtained by different reconstruction algorithms, including the performance of the association algorithm, the compressed sensing algorithm based on TVAL3, and the PnP-GAP algorithm using only the denoising prior, that is, the demosaicing process and the iterative process are separated. In addition, the present invention also compares the reconstruction results without using the response coefficients of different colors for color correction. After obtaining the demosaiced image of the scene to be tested by different algorithms, the same DDN is used to restore the color information of the scene. The reconstruction results of different algorithms when the sampling rate (SR) is from 0.0625 to 1 are shown in Figure 2, where (a) is a black and white image; (b) is a color image. From the results obtained, it can be seen that the proposed method has good color information recovery performance under different SRs, and as the SR increases, the reconstructed image contains richer details and is closer to the real image. However, the traditional association algorithm is difficult to reconstruct clear results even when the SR is 1. Although the compressed sensing algorithm and the PnP-GAP algorithm using only the denoising prior can restore image details and reconstruct clear results, the color information is seriously damaged. This is because in the process of reconstructing mosaic images, due to multiple iterations and image prior processing, the color Bayer matrix information is lost, so it is difficult for the demosaicing algorithm to restore color information. In addition, without color correction, although the spatial distribution of the object can be restored, the color is very different from the color of the original object. Because the single-pixel detector responds more strongly to green, the reconstructed result is greener than the original object. In the proposed method, since the demosaicing algorithm is embedded in each iteration and color correction is performed, the color Bayer matrix is ​​well preserved, and color correction is performed using response coefficients of different colors, so that the color information of the reconstructed image is closer to the original image. In addition, although the differential measurement method is used in the experiment, the system instrument itself will bring noise that is difficult to eliminate. The traditional iterative optimization algorithm is more obviously affected by noise, the quality of the reconstruction result decreases faster, and more noise information appears in the result. The present invention includes two priors, denoising and mosaic, so that the reconstruction result has less noise and has better robustness to noise.

[0061] Due to the flexibility of the PnP-GAP algorithm, different denoising algorithms can be replaced during the iterative cycle to produce different results. Therefore, four different denoising algorithms, including bilateral filtering, wavelet filtering, total variation (TV) and FFD-Net, are used as denoising priors to reconstruct the image.

[0062] In order to quantitatively compare the reconstruction results of different algorithms, the present invention introduces two metric functions, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), which are defined below.

[0063] (1) Mean Square Error (MSE)

[0064]

[0065] Where M represents the total number of pixels in the scene to be tested.

[0066] (2) Structural Similarity (SSIM)

[0067]

[0068] Where η represents the mean, σ 2 represents the variance, represents the covariance between the reconstructed image and the original image, c1 and c2 are regularization parameters.

[0069] (2) Peak signal-to-noise ratio (PSNR)

[0070]

[0071] Where n represents the bit depth of the image.

[0072] The image evaluation results are as follows Figure 3 As shown, (a) is the PNSR curve graph; (b) is the SSIM curve graph.

[0073] Similarly, color images are reconstructed by embedding different demosaicing algorithms into PnP-GAP to compare their reconstruction performance. Four different demosaicing algorithms, Bilinear interpolation (Bilinear), Malvar 2004, Menon 2007, and DDN are used to restore the color information of the image. The comparison results are shown in Figure 4 As shown, (a) is the PNSR curve graph; (b) is the SSIM curve graph.

[0074] It can be seen that the reconstruction results of FFD-Net and DNN based on deep learning methods are significantly better than other algorithms. This is because the traditional model-based denoising and demosaicing algorithms use hand-crafted priors, resulting in a single type of noise and the inevitable appearance of some visual artifacts, which reduces the image quality. FFD-Net and DNN directly learn the prior information of the image from a large number of training images, thereby achieving better performance in denoising and demosaicing tasks and obtaining better reconstruction results.

Claims

1. A color computational ghost imaging method based on a plug-and-play generalized alternating projection algorithm, characterized in that: The following steps are involved: (1) Measure the response coefficient μ of a single pixel detector to red, green and blue colors j ; (2) Randomly generate a binary matrix D i , while using the Bayer matrix S j Synthetic Color Lighting Mode P i,j , and projected onto the scene to be measured by a projector j Above; where i is the measurement index, i = 1, 2, ..., N, N is the total number of measurements, j = R, G, B, representing the three color channels of red, green and blue; (3) The illumination pattern modulated by the scene to be tested is collected by a single-pixel detector, and the collected light intensity Y i Go digital; (4) According to the PnP-GAP algorithm, an auxiliary parameter v is introduced into the color CGI reconstruction model, and the forward physical model and image prior are divided into two independent modules; (5) pre-training a denoising neural network and a demosaicing neural network; (6) The trained denoising neural network is used in the image prior module, and the demosaicing neural network is embedded in the iterative process of image reconstruction. Before the demosaicing operation is performed, the measured response coefficient is used to correct the mosaic image of the reconstructed scene. After the iteration is completed, the reconstruction result of the color scene is output.

2. The color computational ghost imaging method based on the plug-and-play generalized alternating projection algorithm according to claim 1 is characterized in that: The color lighting mode P in step (2) i,j The synthesis method is: P i,j (x,y)=D i (x,y)·S j (x,y), In the formula, i is the measurement index, i = 1, 2, ..., N, N is the total number of measurements, j = R, G, B, representing the three color channels of red, green and blue, (x, y) represents the pixel index coordinates; D i is a randomly generated binary matrix, S j is the Bayer matrix.

3. The color computational ghost imaging method based on the plug-and-play generalized alternating projection algorithm according to claim 1, characterized in that: The light intensity Y collected in step (3) i It is expressed as: In the formula: μ j is the detector’s response coefficient to different colors, P i,j For color lighting mode, O j is the scene to be measured, D i is a randomly generated binary matrix, S j is the Bayer matrix, O m A mosaic image representing the scene under test.

4. The color computational ghost imaging method based on the plug-and-play generalized alternating projection algorithm according to claim 1 is characterized in that: The color CGI formula based on the PnP-GAP algorithm in step (4) is: In the formula, f m represents vectorizing the mosaic image of the scene to be measured into a column vector, λ is the weight coefficient between the CGI imaging model and the regularization term, H is rearranging the two-dimensional illumination pattern into a one-dimensional row vector, and reorganizing it row by row according to the modulation order to obtain a two-dimensional measurement matrix; Then, this minimization problem is solved by the following two sub-problems: Update f m : f m (k+1) =v (k) +H T (HH T ) -1 (Y-Hv (k) ), Where the superscript k represents the number of iterations; Update v: Where Φ Dm and Φ Dσ They represent denoising and demosaicing algorithms respectively.

Images