Noise engulfing ghost imaging with depth information

Abstract

The application discloses noise-eating ghost imaging with depth information, first uses the characteristics that light reaches different distances of objects at different times, acquires two-dimensional images of scenes at each depth, and reconstructs the depth information of the object. The noise-eating algorithm makes the noise of the reconstructed image reduced. In this paper, two layers of slices with high depth resolution are used for experiments. In the normal lighting condition, laser and random speckle can be used, and a darkroom is not needed. Compared with radar imaging, the application has lower cost, simpler equipment and is more convenient to debug. In the field of remote sensing detection, the application can not be affected by extreme environment and has faster imaging speed. Compared with the original image, the noise-eating algorithm has an NRSS value increase of 45%, a Brisque increase of 25% and a Niqe increase of 33% in the image quality evaluation. Compared with the untreated image, the application greatly reduces the influence of noise on the depth information of the image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing detection, and particularly relates to noise swallowing ghost imaging with depth information. BACKGROUND

[0002] Ghost imaging is a new imaging method, which uses a detector without spatial resolution for continuous sampling, and obtains target scene information through sampling data, and is one of the research hotspots in the field of quantum and image processing in recent years. Compared with traditional imaging methods, ghost imaging realizes high-resolution imaging, has strong anti-interference ability, is suitable for imaging in complex environments, can realize full-band imaging of light sources, and has a longer action distance than traditional imaging. Therefore, ghost imaging is becoming one of the practical candidate technologies in the fields of medical imaging, laser radar, optical encryption, remote sensing satellite, and three-dimensional computational ghost imaging is also rapidly developing in recent years. In 2014, Peng Jinye proposed a high-speed single-pixel camera data acquisition system, in 2019, Sun Mingjie proposed a topic of single-pixel imaging and its application in three-dimensional reconstruction, and in 2019, Han Shengsheng's group in Shanghai proposed three-dimensional imaging using pre-set intensity correlation laser, which quickly opened the curtain of three-dimensional ghost imaging. In October 2021, XIANYELI proposed a three-primary-color light photometric stereo method based on three-primary-color light, which irradiates the object from three angles to realize millimeter-level three-dimensional reconstruction.

[0003] In summary, in the fields of laser radar and remote sensing satellite, in order to facilitate the display of object-related information, the image obtained by ghost imaging needs to carry depth information, but the precision of the device for obtaining a three-dimensional reconstruction map is too high, and the calculation amount is too large, which leads to the inability to display in real time and high cost. Therefore, it can be seen that the demand for two-dimensional ghost imaging carrying depth information is becoming more and more obvious. SUMMARY

[0004] The purpose of the present application is to solve the problem that ghost imaging cannot reconstruct depth information, and to provide noise swallowing ghost imaging with depth information.

[0005] The noise swallowing ghost imaging with depth information comprises:

[0006] 1) correlating the speckle matrix received by the CCD with the light intensity value to restore the image of the target;

[0007] 2) ghost imaging with depth information

[0008] The return waveform obtained by the time-resolved bucket detector is divided into time slice signals of different stages, and then the signals in each time slice are integrated, and a matrix is generated by correlating the speckle with the light intensity values measured under different slices; a two-dimensional image of the scene at each depth is obtained, which will further form a cubic of two-dimensional images, and these two-dimensional images are superimposed together to obtain a two-dimensional image carrying depth information;

[0009] 3) Noise swallowing algorithm

[0010] (1) Three-dimensional weighted median filtering

[0011] The weighted mean filtering formula of the 3x3 window is:

[0012]

[0013] (6)

[0014] wherein represents the weight, for example: a b = b x b x... b (a) is a weight matrix, is the value of the point in the weight matrix.

[0015]

[0016] wherein p in is the height number weight, N is the height, F i is the height of the point to be measured, and F n is the target point height.

[0017] (2) Bilateral filtering

[0018] Perform a Gaussian filter and then perform a Gaussian filter.

[0019] Step 1) The correlation operation is: in N times sampling, the kth speckle matrix generated by the spatial light modulator is , and the light intensity value measured by the bucket detector corresponding to it is B(k), and the transmission coefficient of the object to be measured is The speckle matrix received by the CCD is correlated with the light intensity value to restore the image of the target;

[0020] The reconstruction formula is:

[0021]

[0022] In the formula, T(x, y) is the reconstructed ghost image, and <▪> represents the operation of the average value inside, k = 1, 2,..., N, and N is the total sampling number;

[0023] The signal integration in each time slice is performed by correlating the speckle with the light intensity values measured under different slices, which generates a matrix as follows:

[0024]

[0025] The calculation formula of the image of each slice is as follows:

[0026]

[0027]

[0028] In the formula is the matrix summation symbol, that is, the sum of all elements in the matrix,

[0029]

[0030] is the speckle obtained by the i-th measurement of the CCD, is the light intensity value corresponding to the a-th time slice of the i-th measurement collected by the bucket detector, and M is the total sampling number of the bucket detector;

[0031] The calculation formula of the two-dimensional ghost imaging with depth information is as follows:

[0032]

[0033] 3) Noise swallowing algorithm

[0034] 4. The noise swallowing ghost imaging with depth information, characterized in that step 3), (2) the bilateral filtering is:

[0035] The specific formula is as follows:

[0036]

[0037] wherein represents the bilateral filtering result of point p, and S represents the range of the filtering window; is the spatial neighborhood standard deviation, is the pixel brightness standard deviation; , are the height proximity function and the surrounding weight function respectively, which are Gaussian functions; W p is a standard quantity, representing the weighted sum of the product of the gray weight and the spatial weight, and is defined as:

[0038]

[0039]

[0040] .

[0041] The present application provides noise-eating ghost imaging with depth information, the present inventors propose a method of two-dimensional ghost imaging with depth information, and propose a related high-quality optimized noise-eating filtering algorithm. By calculating the light at the same position, the light reaches the object at different distances at different times, to obtain a two-dimensional ghost imaging image with depth information, and by using a related image processing method to improve the imaging quality.

[0042] The present inventors propose a noise-eating ghost imaging with depth information according to the characteristics of the bucket detector. First, the feature that light reaching objects at different distances takes different times is used to obtain a two-dimensional image of the scene at each depth, to reconstruct the depth information of the object. The proposed noise-eating algorithm reduces the noise of the reconstructed image. The present inventors use two high-depth-resolution slices to do experiments, which perfectly verify the proposed reconstruction method. This method can be realized under normal lighting conditions using laser and random speckle, without the need for a darkroom. Compared with radar imaging, it has lower cost, simpler equipment, and is more convenient to debug. In the field of remote sensing detection, it can be unaffected by extreme environments and has faster imaging speed. Compared with the original image, the noise-eating algorithm has an NRSS value improvement of 45%, a Brisque improvement of 25%, and a Niqe improvement of 33% in image quality evaluation. The subjective evaluation effect is better, and compared with the untreated image, it greatly reduces the influence of noise on image depth information, and has a very far-reaching influence. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 Figure 1 is a schematic diagram of the principle of ghost imaging;

[0044] Figure 2 is a flowchart of noise-eating ghost imaging with depth information;

[0045] Figure 3 is a schematic diagram of a bucket detector with time resolution;

[0046] Figure 4 is a schematic diagram of an experimental scene;

[0047] Figure 5 is an experimental recovery diagram;

[0048] Figure 6 is a comparison diagram of experimental results. DETAILED DESCRIPTION

[0049] Example 1: Noise-eating ghost imaging with depth information

[0050] 1. Ghost imaging

[0051] The experimental device of ghost imaging is as follows Figure 1The laser light is emitted from the laser emitter, the light field is modulated by the ground glass, the modulated speckle is divided into two parts by the beam splitter, one part is received by the computer and stored through the image sensor (CCD), and the other part is irradiated onto the object, and the light reflected by the object is collected by the bucket detector (B). The above experimental process is repeatedly repeated, the data is summarized and processed, and the image of the object is recovered through correlation operation. In N times of sampling, the kth speckle matrix generated by the spatial light modulator is I(x, y, k), the light intensity value measured by the bucket detector is B(k), and the transmission coefficient of the object to be measured is T(x, y). The speckle matrix received by the CCD and the light intensity value are correlated to restore the image of the target. The reconstruction formula of the ghost imaging is:

[0052]

[0053] In the formula, T(x, y) is the reconstructed ghost image, <▪> represents the operation of the average value in the number, k=1, 2,..., N, and N is the total sampling number.

[0054] 2. Ghost imaging with depth information

[0055] In two-dimensional ghost imaging, one speckle corresponds to one measured light intensity. However, when we use a detector (PMT) with super strong time resolution to measure, one speckle will correspond to a series of light intensity values at different depth positions in the scene. Figure 3 The two-dimensional signal returns a bucket detection value, and signals at different depths will return multiple wave peaks at different depths.

[0056] Therefore, the inventors propose a ghost imaging with depth information based on the light time-of-flight method. First, the return waveform collected by the bucket detector (PMT) is divided into time (distance) slice signals at different stages, then the signals in each time slice are integrated, and the speckle and the measured light intensity value at different slices are correlated The generated matrix is as follows:

[0057]

[0058] The calculation formula of the image of each slice is as follows:

[0059]

[0060] In the formula, is the matrix summation symbol, that is, the sum of all elements in the matrix,

[0061]

[0062] is the speckle obtained by the i th measurement of the CCD, The light intensity value corresponding to the ith measurement of the a time slice collected by the bucket detector, M is the total sampling number of the bucket detector .

[0063] The above formula can obtain the two-dimensional image of the scene at each depth, which will further form a cube of two-dimensional images. By superimposing these two-dimensional images, a two-dimensional image carrying depth information is obtained. Therefore, we can obtain the calculation formula of two-dimensional ghost imaging with depth information:

[0064]

[0065] Because the image carries depth information, the article adopts three methods of gray scale, HSV pseudo-color and stereoscopic display to represent. The reason for adopting HSV color domain is that this color representation method can have perfect correspondence with depth information. In the process of displaying some terrain maps, HSV pseudo-color display is often adopted. The stereoscopic display method can more intuitively see the depth information carried by the image.

[0066] 3. Noise swallowing algorithm

[0067] For the ghost imaging image with depth information, if it is obviously different from the surrounding pixel points, it can be considered as noise, and it can be considered that the point is infected by noise.

[0068] The noise of ghost imaging with depth information mainly has four sources: one is the distance ambiguity existing in the light flight time method; two is the dust existing in the experimental process; three is the slight scattering after passing through the lens mixed with the main light path; four is the weak light generated by a small amount of diffuse reflection after being irradiated onto the object and received by the bucket detector. The first and fourth reasons are the main reasons for producing noise.

[0069] Based on the above analysis, the inventor first adopts three-dimensional weighted median filtering, which can reduce the influence of noise on the image. At the same time, some small burrs are preserved, so that the depth information is not clear. Only considering the influence of the template on the center height of the template, the edge will be more obviously blurred. Therefore, the inventor proposes a noise swallowing algorithm to ensure the removal of noise and the outline of the object edge. It is hoped that a Gaussian filtering is performed again. However, Gaussian filtering will more obviously blur the edge because it only considers the influence of the position on the center pixel. In order to eliminate noise while well preserving the edge, the noise swallowing algorithm proposed by the inventor is a very effective method.

[0070] 1) Three-dimensional weighted median filtering

[0071] In order to remove background noise interference of ghost imaging, the inventors proposed a three-dimensional weighted median filtering method based on depth information.

[0072] Median filtering is a commonly used filtering method to remove noise and protect edges. It is a nonlinear smoothing filter. With a certain point as the center, select a window around it, sort the values ​​of the pixels, and take the median value to replace the value of the pixel. The formula for median filtering is:

[0073]

[0074] G(x,y) is the input ghost image, S is the set of coordinates of the points in the neighborhood of point (x,y), which includes point (x,y). M is the total number of coordinate points in S.

[0075] Obviously, different points in the neighborhood have different effects on the current pixel, and the classic median filter method does not take this into account. The weighted median filter solves this problem very well. The weighted mean filter formula for a 3×3 window is:

[0076]

[0077]

[0078] in Indicates weighting, for example: a b=b×b×...b(a) W xy is the weight matrix, w xy is the value of the point in the weight matrix.

[0079]

[0080] where p in is the height weight, N is the height, F i is the height of the point to be measured, F n is the target point height.

[0081] 2) Bilateral filtering

[0082] Simply put, bilateral filtering is a kind of local weighted average. Since bilateral filtering has one more Gaussian variance than Gaussian filtering, pixels farther away near the edge will not have much influence on the pixel values ​​on the edge, thus ensuring that the edge pixels will not change significantly. This is equivalent to ensuring denoising while also reducing the impact of neighborhood averaging on edges. The specific formula is as follows:

[0083]

[0084] in The bilateral filtering result of the point p, S represents the range of the filtering window. The spatial neighborhood standard deviation, The pixel brightness standard deviation. 、 The height proximity function and the surrounding weight function are respectively Gaussian functions. A standard quantity, which represents the weighted sum of the product of the gray weight and the spatial weight, is defined as:

[0085]

[0086]

[0087] From the above formula, it can be seen that the bilateral filtering has two important key parameters: δ s and δ γ , δ s controls the spatial proximity, and its size determines the number of pixels contained in the filtering window, that is, when δ s becomes larger, the window contains more pixels, and the pixel points far away will also affect the center pixel point, and the smoothing degree is improved. δ s is used to control the height proximity, when δ γ becomes larger, the point with a larger height difference can also affect the pixel value of the center point, but the pixel with a height value greater than δ γ will not participate in the operation, so that the high-frequency edge information of the image can be preserved, and when δ s and δ γ are very small, the image will hardly be smoothed. It can be seen that the parameter selection of δ s and δ γ directly affects the output result of the bilateral filtering, that is, the smoothing degree of the image.

[0088] Experimental results and analysis

[0089] In order to detect the optimization effect of the method, the inventors use the MATLAB platform to carry out simulation and real experiment, and the inventors use the MATLAB platform to carry out experiment, and use NRS to evaluate. Since there is no real reference picture, the evaluation index needs to use the non-parametric evaluation index, the phagocytosis algorithm uses NRSS, Brisque and Niqe to evaluate the pros and cons of the restored image, and the smaller the index is, the better.

[0090] 1. Process analysis

[0091] 1) Use the generated random speckle to irradiate the unknown target , and use the bucket detector to record the total light intensity waveform ;

[0092] 2) The waveform is decomposed into several segments and each segment is integrated to obtain , forming a bucket detection value matrix with i rows and a columns;

[0093] 3) Perform second-order correlation between the bucket detection values ​​of each column and the random speckle pattern to obtain a slice image;

[0094] 4) Synthesize the slices into an image with depth information;

[0095] 5) The engulfment algorithm is used to process the image to obtain better quality images.

[0096] 2. Experimental depth map restoration results

[0097] To investigate the practical application of the proposed method, the speckle pattern used in the experiment was random speckle generated by frosted glass, sampled 20,000 times, and the inventors used a T and a star slice for the experiment, where the T is 30cm higher than the star. The target object resolution is 200×200pix, and the depth resolution of the inventors is 30cm. The experimental scene is as follows: Figure 4 shown.

[0098] The restored image is Figure 5 As shown in the figure, (a) is the HSV pseudo-color image that can reflect the corresponding depth information, (b) is the 45-degree depth recovery image, and (c) is the frontal depth recovery image.

[0099] 3. Evaluation Method

[0100] Because there is no original image in the experiment, the inventor uses NRSS , Brisque, and Niqe are three parameter-free evaluations to verify the algorithm proposed above.

[0101] The inventors selected the NRSS evaluation method to evaluate the experimental results. NRSS is a method that evaluates image quality based on image clarity, without a reference image, based on subjective perception. The smaller the value, the better the imaging quality. As shown in Formula 11, Represents structural similarity.

[0102]

[0103] 4. Results after noise swallowing algorithm processing

[0104] To investigate the application effect of the proposed enhancement algorithm in practice, this section uses the noise swallowing algorithm to optimize the reconstructed image. Since there is no real reference picture, the evaluation index needs to use the non-parametric evaluation index. The inventors use the NRSS, Brisque and Niqe parameters to evaluate the pros and cons of the restored image. The indexes of the three parameters are all the smaller the better. The experimental results are shown in Figure 6 Fig. 1, where (a) is the gray image of the original image, (b) is the gray image of the image after using the algorithm, (c) is the 45-degree image of the original image, (d) is the 45-degree image after using the algorithm, (e) is the front image of the original image, and (f) is the 45-degree image after using the algorithm.

[0105]

[0106] 5、Conclusion

[0107] Figure 6 The flowchart of the method, the inventors make the two-dimensional ghost imaging reconstructed image carry depth information under the framework of ghost imaging, and propose a noise swallowing algorithm, and introduce the working principle of the algorithm. Finally, after the simulation and experiment of the star and T-shaped target object, the data show that the method proposed by the inventors can make the reconstructed image carry depth information compared with the traditional two-dimensional image reconstruction method, and the depth resolution is 30 cm. The scene is used for experiment. It is feasible in theory and experiment. The subjective and objective evaluation of the proposed noise swallowing algorithm is better. Compared with the original experimental recovery graph, the method proposed by the inventors has a large increase in the NRSS, Brisque and Niqe evaluation methods. The NRSS value is increased by 45%, the Brisque is increased by 25%, and the Niqe is increased by 33%. For detecting the depth information of the object, the method is more cost-effective than radar.

Claims

1. Noise-swallowed ghost imaging with depth information, which includes: 1) Correlate the speckle matrix received by the CCD with the light intensity value to restore the image of the target; 2) Ghost imaging with depth information The return waveform collected by the time-resolution bucket detector is divided into slice signals of different time stages. The signals within each time slice are then integrated. A matrix is ​​generated by correlating the speckle pattern with the light intensity values ​​measured in different slices. A two-dimensional image of the scene at each depth is obtained, which is then formed into a two-dimensional image cube. These two-dimensional images are superimposed together to form a two-dimensional image carrying depth information. 3) Noise swallowing algorithm (1) Three-dimensional weighted median filtering The weighted mean filter formula for a 3×3 window is: ； （6）； in represents weighted, a b=b×b×...b(a) W xy is the weight matrix, w xy is the value of the point in the weight matrix; ； where p in is the height weight, N is the height, F i is the height of the point to be measured, F n is the target point height; (2) Bilateral filtering Perform a Gaussian filter and then perform a Gaussian filter again.

2. The noise-swallowed ghost imaging with depth information according to claim 1, characterized in that: Step 1) The correlation operation is as follows: assuming that the kth speckle matrix generated by the spatial light modulator in N samplings is I(x, y, k), the corresponding light intensity value measured by the bucket detector is B(k), and the transmission coefficient of the object to be measured is T(x, y). The speckle matrix received by the CCD is correlated with the light intensity value to restore the image of the target; The reconstruction formula is: (1)； ； Where T(x,y) is the reconstructed ghost image, <▪> represents the operation of the average value of the values ​​therein, and k=1,2...,N, where N is the total number of samples.

3. The noise-swallowed ghost imaging with depth information according to claim 1 or 2, characterized in that: The signal integration within each time slice is performed by correlating the speckle pattern with the light intensity values ​​measured in different slices. The resulting matrix is ​​shown below: ； The calculation formula for each slice image is as follows: ； ； In the formula Sum is the matrix summation symbol, that is, the sum of all elements in the matrix. ， ， is the speckle obtained by the CCD measurement for the i-th time, is the light intensity value corresponding to the a-th time slice of the i-th measurement collected by the bucket detector, and M is the total number of sampling times of the bucket detector; The calculation formula for two-dimensional ghost imaging with depth information is: (4)； 3) Noise swallowing algorithm.

4. The noise-swallowed ghost imaging with depth information according to claim 3, characterized in that: Step 3, the bilateral filtering described in (2) is: The specific formula is as follows: ； in represents the bilateral filtering result of point p, S represents the range of the filtering window; δ s is the standard deviation of the spatial neighborhood, δ γ is the standard deviation of pixel brightness; 、 are respectively the height proximity function and the surrounding weight function, which are in the form of Gaussian function; W p is a standard quantity, which represents the weighted sum of the product of grayscale weight and spatial weight, and is defined as: ； ； 。

Images