A passive three-dimensional imaging method based on an optical interference computational imaging method

Abstract

The present invention discloses a passive three-dimensional imaging method based on an optical interference computational imaging method. This imaging method uses an optical interference computational imaging system with non-concentric baselines at the center to collect the mutual correlation intensity of object light in the spatial frequency domain. Then, the reference working distance is stepped and adjusted for phase compensation to reconstruct the image. Finally, with the aid of an image optimization evaluation algorithm, a clear image of the object space and the three-dimensional coordinate information of the target of interest are obtained. This method uses a single-position optical interference computational imaging system to passively collect object light data in one exposure, and combines an image optimization evaluation algorithm to obtain a clear image of the object space and three-dimensional coordinate data, having the advantages of a wide applicable environment and high efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of optoelectronic imaging, and provides a passive three-dimensional imaging method based on an optical interference computational imaging method, which will play an important role in fields such as scientific exploration, national defense, and space exploration. Background Art

[0002] Three-dimensional imaging technology aims to obtain the stereoscopic information of a target. After decades of development, it is widely used in fields such as biomedicine, autonomous driving, and terrain exploration, and has important research value. At present, vision-based three-dimensional imaging technologies are mainly divided into two categories: active and passive. Active methods mainly include laser scanning method, structured light method, time-of-flight method, etc. These methods introduce an active light source to illuminate the target, and infer the three-dimensional information of the target through the change of light intensity or phase, and can detect weak targets or even targets without a light source. Passive methods mainly include monocular focus degree analysis method, binocular feature point matching method, multi-view image fusion method, etc. They reconstruct the three-dimensional model of the target by analyzing multiple exposure or multi-camera photographs. In short, various three-dimensional imaging methods have been successively proposed, and three-dimensional imaging has become a research hotspot in academic research and industrial applications.

[0003] In recent years, scientists have combined the interference imaging principle and photon integration technology to propose a photon integrated interference imaging system (US 8913859B1). Different from traditional spatial imaging, it collects light through a paired aperture array located on the equivalent pupil plane, and uses a waveguide array located behind each aperture to obtain a large field of view. The light of each sub-field then enters an orthogonal detector to generate photocurrent after being transmitted and processed by a grating beam splitter and a phase shifter in the optical path. Each pair of lens pairs forms an interference baseline, and the corresponding photocurrent can be calculated as a cross-correlation intensity signal of a specific spatial frequency. After obtaining an appropriate number of spatial frequency cross-correlation intensity samples through a certain number of interference baselines, the two-dimensional reconstruction image of the object space can be obtained through inverse two-dimensional Fourier transform. The photon integrated interference imaging system can be designed into various lens array structural forms such as radial (US 8913859B1), hexagonal, and checkerboard (CN202010965700.X), and the frequency information acquisition capabilities of various structural forms for an ideal two-dimensional target scene at a single distance have been studied, while ignoring the influence brought by the target depth of field distance and the system interference baseline configuration. The current related research has not discussed the imaging ability for three-dimensional space.

[0004] The inventors focused on the influence of the aggregation of inventors on the imaging quality of a photon integrated interference imaging system with respect to the target depth of field distance and the system interference baseline configuration. A reference working distance was introduced and combined with the baseline configuration to correct the system-acquired signals. When the midpoints of the baselines in the optical interference computational imaging system do not completely coincide, the influence mechanism of adjusting the reference working distance on the clarity of the reconstructed image was studied. It was found that the reference working distance that makes the target image clearest is exactly its unique actual working distance. Furthermore, a passive three-dimensional imaging method based on the optical interference computational imaging method was proposed, providing a new solution idea for three-dimensional imaging.

[0005] The present invention uses a single-position optical interference computational imaging system to passively collect object light data through a single exposure, and combines an image optimization evaluation algorithm to obtain clear images and three-dimensional coordinate data of the object space, with advantages such as a wide applicable environment and high efficiency. Summary of the Invention

[0006] The working principle of the optical interference computational imaging system is that each interference baseline located on the equivalent pupil plane collects the cross-correlation intensity signal of the object light, and then the image is reconstructed by inverse two-dimensional Fourier transform.

[0007] According to the linear property of the Fourier transform, the reconstructed image can be regarded as the superposition of the inverse images after the two-dimensional inverse Fourier transform of the signals collected by each interference baseline.

[0008] According to the Van Cittert–Zernike theorem, on the plane of the lens array of the equivalent pupil plane of the optical interference computational imaging system, for a pair of lenses forming an arbitrary interference baseline at coordinates (x1, y1) and (x2, y2), the cross-correlation intensity J of the collected light rays is:

[0009]

[0010] where λ is the wavelength, z is the target distance, I(α, β) is the light intensity of the target, Δx = x2 - x1, Δy = y2 - y1 is the distance between the lens pair, that is, the baseline B. And the phase factor is

[0011]

[0012] The spatial frequency domain collected by this aperture pair is:

[0013]

[0014] Then, the mutual intensity J can be expressed as:

[0015]

[0016] where is the midpoint of the aperture pair, is the two-dimensional Fourier transform of I(α, β), that is, the cross-correlation intensity at the object space frequency domain (u, v) corresponding to the aperture pairs (x1, y1) and (x2, y2). It can be seen that the signal collected by the aperture pairs, that is, the mutual intensity J, is related to the target space frequency domain (u, v), the target distance z, and the baseline center position (x m , y m ).

[0017] In the actual working scenario, the actual working distance z of the target is usually unknown, so a reference working distance z c is introduced. To discuss the different effects on the actual working distance z and the reference working distance z c , a correction term J c is applied to the collected signal, which consists of the aperture pair coordinates and the set reference working distance z c :

[0018]

[0019] Combining formulas (4) and (5), the corrected signal J·J c is obtained as follows:

[0020]

[0021] According to the displacement property of the two-dimensional Fourier transform, it is the two-dimensional Fourier transform after the object space is translated. Considering the periodicity of the two-dimensional inverse Fourier transform, the image translation amount is

[0022]

[0023] where is the period of the inverse image, and a and b are arbitrary integers. It can be seen that the inverse image obtained by the two-dimensional inverse Fourier transform of the interference baseline signal will be translated along with s0. To measure the influence of the translation of the inverse image on the reconstructed image, the ratio of the translation amount s0 to the size of the reconstructed image, that is, the image deviation amount, can be used for evaluation. The size of the reconstructed image is the field of view size, which is calculated as is the shortest baseline in two orthogonal directions. Therefore, the image deviation amount of the inverse image can be normalized to the field of view size as follows:

[0024]

[0025] s decreases with the increase of the field of view size and increases with the increase of the midpoint deviation amount, that is, the distance by which the interference baseline center (x m , y m ) deviates from the optical axis center. In addition, it is also related to z cis related to the value of. For different interference baselines, s will have different values, and the corresponding inverse images will have different deviations. Moreover, the discreteness of s will affect the clarity of the reconstructed image.

[0026] Substantially, after the object space is decomposed in the frequency domain, it can be regarded as the superposition of the original images corresponding to a series of spatial frequencies. The imaging system samples specific spatial frequencies using different interference baselines and composes the reconstructed image. If the distance z of the target is very far, the field of view size will be much larger than the midpoint deviation, that is, L x >> x m 、L y >> y m all s will approach According to periodicity, it is equivalent to (0, 0). All inverse images are in the correct positions, and the reconstructed image will be clear. However, when the target distance is not too far, the influence of the midpoint deviation will become apparent. If the coordinates of the midpoints of each interference baseline are the same but not zero, then all s can obtain the same value, and all inverse images have the same deviation. The reconstructed image is clear but has a translational deviation. For an imaging system with non-coincident interference baseline centers, by adjusting the value of z c the value of, the value and discreteness of the image deviation amount s can be changed. When z c = z, all s can take the same value (a, b), which is equivalent to (0, 0) according to periodicity. At this time, the reconstructed image is clear; in addition, when z c is near z but z c ≠ z, all s are discrete, and the image deviations of each inverse image are discrete, just like the colors not being aligned when printing a newspaper, and a blurred reconstructed image will be obtained. Therefore, within the range near the actual working distance of the target, only when z c = z is the reconstructed image clear, and as z c moves away from z, the reconstructed image will become more and more blurred.

[0027] Based on the above working principle, the present invention discloses a passive three-dimensional imaging method based on an optical interference computational imaging method. This imaging method uses an optical interference computational imaging system in which the apertures are relatively discrete with respect to the midpoint of the array baseline, for example, the baseline centers formed by each aperture pair do not coincide, or at least several groups do not coincide. The cross-correlation intensity of the object light spatial frequency domain is interferometrically recorded. Then, with the help of an image optimization evaluation algorithm, the clarity of the inverse reconstructed image under the step-by-step adjustment of the reference working distance is analyzed to obtain the clearest reconstructed image and the corresponding reference working distance. Finally, the relative position and size of the target are calculated based on the best reference working distance and the reconstructed image, and the three-dimensional imaging of the object space is completed. This method is a single-position single-exposure passive three-dimensional imaging method. The key steps of three-dimensional imaging are as follows:

[0028] Step 1: Use an optical interference computational imaging system in which the apertures are relatively discrete with respect to the midpoint of the array baseline, that is, the centers of the baselines formed by each pair of apertures do not coincide, or at least several groups do not coincide, and record the cross-correlation intensity of the object light spatial frequency domain by interference.

[0029] Step 2: Stepwise adjust the reference working distance within a certain range to compensate for the phase difference of the cross-correlation intensity of each spatial frequency domain corresponding to the baselines of different aperture pairs, and then inversely reconstruct the object-space image through the Fourier transform algorithm.

[0030] Step 3: Adopt an image optimization evaluation algorithm to evaluate the sharpness of each reconstructed object-space image, obtain a reconstructed image with a clear object-space scene target or a locally clear target, and the corresponding reference working distance.

[0031] Step 4: Based on the clear or local image and the corresponding reference working distance, calculate the relative position and size information of the target of interest in the image, and then reconstruct the three-dimensional image of the object-space scene to complete the passive three-dimensional imaging and image reconstruction of the object-space scene. Description of the Drawings

[0032] Figure 1 : Flowchart of the implementation scheme of the passive three-dimensional imaging method.

[0033] Figure 2 : Composition and working principle diagram of the checkerboard imager. In the figure, A is the aperture pair array for collecting object light, B is the two-dimensional PIC optical waveguide array for beam splitting, C is the three-dimensional optical waveguide array for transmitting light beams and matching the optical path difference, D is the two-dimensional PIC optical waveguide array for aperture pair matching and coherence, E is the readout circuit and data processing system, where B.1 is the cross-section of the waveguide array, B.2 is the beam splitting grating, E.1 is the phase shifter, and E.2 is the balanced quadrature coupler.

[0034] Figure 3 : Sharpness of the inverse image at different reference working distances. (a) is the resolution target pattern as the simulation input image; the reconstructed images shown in (b)-(f) are for reference working distances z c Take the reconstructed images at infinity, 1500m, 1000m, 1475m, and 1550m.

[0035] Figure 4 : Relationship between the sharpness of the reconstructed object-space scene image at a single working distance and the value of the reference working distance. The dashed line is the evaluation by the Laplacian gradient function, and the solid line is the evaluation by the structural similarity.

[0036] Figure 5: 3D imaging simulation scene diagram. (a) Spatial relative position scene diagram of the imager, UAV, and the ground. In the figure, F is the imager and G is the UAV; (b) Shape of the UAV; (c) Ground image containing the projection of the UAV; (d) Image of the "car" area.

[0037] Figure 6 : Relationship between the clarity of the reconstructed image evaluated by the Laplace gradient function and the reference distance value under different object-side scene conditions at different working distances. The solid line is the clarity curve of the UAV area image, and the dashed line is the clarity curve of the car area image.

[0038] Figure 7 : Simulation results of targets at different distances. (a) Reconstructed image when the reference working distance is 8.034 km; (b) Reconstructed image when the reference working distance is 9.997 km; (c) UAV area when the reference working distance is 8.034 km; (d) "Car" area when the reference working distance is 8.034 km; (e) UAV area when the reference working distance is 9.997 km; (f) "Car" area when the reference working distance is 9.997 km. Detailed implementation mode

[0039] Example 1: 3D imaging results of a single-distance target

[0040] Taking the "chessboard" imager with a (2N + 1)×(2N + 1) matrix arrangement of aperture pairs for the lens array based on the principle of optical interference computational imaging as an example for 3D imaging effect analysis, the composition and working principle diagram of the chessboard imager are as Figure 2 shown. A is the aperture pair array for collecting object light, B is the two-dimensional PIC optical waveguide array for beam splitting, C is the three-dimensional optical waveguide array for transmitting light beams and matching the optical path difference, D is the two-dimensional PIC optical waveguide array for aperture pair pairing coherence, E is the readout circuit and data processing system, where B.1 is the cross-section of the waveguide array, B.2 is the beam splitting grating, E.1 is the phase retarder, and E.2 is the balanced quadrature coupler.

[0041] The parameters of the imager and the target are shown in Table 1. The midpoint deviation of the interference baseline of the "chessboard" imager is dispersed at four centers: (0.051 m, 0.051 m), (0.051 m, -0.050 m), (-0.050 m, 0.051 m), (-0.050 m, -0.050 m). Select the resolution target pattern as shown in Figure 3 (a) as the simulation input image.

[0042] The simulation process follows that the optical information on the object surface is coupled into the optical waveguide array through the lens array and spectroscopically analyzed by the grating. After passing through the phase retarder, the light rays obtain photocurrent through the orthogonal interferometer. The corrected acquisition signal is calculated using the photocurrent and the reference working distance, and the reconstructed image is obtained after inverse Fourier transform.

[0043] During the inversion reconstruction process, the reference working distance z is sequentially set c to vary from 500 m to 2500 m, and the corresponding reconstructed images are obtained, where z c The reconstructed images at infinity, 1500 m, 1000 m, 1475 m, and 1550 m are as shown in Figure 3 (b) to Figure 3 (f). Let z c be infinity. When the reconstructed image with a deviation from the actual working distance z = 1500 m is as shown in Figure 3 (b), since the midpoint of the interference baseline has four different values, that is, the deviations of the inversion reconstructed images are four different values, resulting in the blurred and distorted image after overlapping the inversion reconstructed images. Let z c = 1500 m. The reconstructed image obtained is as shown in Figure 3 (c). That is, when the actual working distance is used as the reference working distance, after the deviations of the inverse spectral images are effectively corrected, the inversion reconstructed images are overlapped to obtain a clearer image. As can be seen from Figure 3 (d)-3(f), when the reference working distance deviates from the actual working distance, the deviations of the corresponding inversion reconstructed images are four different values, resulting in different degrees of blurred and distorted images after overlapping the inversion reconstructed images.

[0044] The normalized results of evaluating each reconstructed image using the image sharpness evaluation function based on the gradient of the Laplace function are as shown by the dashed line in Figure 4 The normalized results of evaluating each reconstructed image using the negative value of the structural similarity between the reconstructed image and the image after removing the highest frequency filtering as the evaluation function are as shown in Figure 4As shown by the solid line in []. It can be seen that both evaluation functions exhibit good unimodality. The evaluation functions based on gradient and structural similarity reach their maximum values at 1496.48m and 1499.22m respectively, which are very close to the actual working distance of the target. The performance of the evaluation function based on gradient is somewhat insufficient but can be used for the analysis of local patterns. The evaluation function based on structural similarity has smaller oscillations when the reference working distance is far from the actual working distance, and has a smaller full-width at half-maximum near the actual working distance, but is only applicable to the evaluation of the overall image. Taking 1499.22m as the estimated distance of the target, combining the working wavelength and the minimum baseline, the target size is obtained as 0.4498m × 0.4498m, which is close to the actual size of the target. Thus, for the working scenario of single-distance target imaging based on this imaging method, the peak-finding algorithm can be used to find the position where the evaluation function takes the extreme value, and then the best reference working distance is used as the estimated distance of the target, so as to obtain the size and clear image of the target.

[0045] Table 1: Parameters of checkerboard imager and target

[0046]

[0047]

[0048] Example 2: 3D imaging simulation of targets at different distances

[0049] In this example, the "checkerboard" aperture arrangement is still selected as in Example 1, and the imager parameters shown in Table 2 are selected. The midpoint deviation of the interference baseline is distributed at the centers of four parts: (0.0765m, 0.0765m), (0.0765m, -0.075m), (-0.075m, 0.0765m), (-0.075m, -0.075m). Considering the scenario where the target distances in the field of view are not unique and there are occlusions, as Figure 5 (a) shows, it is assumed that the imager F installed on the reconnaissance aircraft flies at a height of z1 = 10 km above the ground to image a section of highway. Let the coordinates of the imager F be (0m, 0m, 0m), and the side length of the ground imaging area be L p = FOV z1 = 24m. At this time, a drone G flying at a height of 2 km flies over the highway. Its size is 1.79m × 1.43m, and the center point coordinates are (-2.19m, -4.00m, 8000m). Its shape is as Figure 5 (b) shows, that is, the distance between the drone G and the imager F is z2 = 8 km. The image of the highway area is as Figure 5 (c) shows, and the projection of the drone on the ground is included in the figure. The area of the car used for comparison is as Figure 5As shown in (d), where the point coordinates are (3.96m, 9.83m, 10000m) and the length of the white car is 2.83m.

[0050] Table 2: Checkerboard imager and target parameters

[0051]

[0052]

[0053] Following the same simulation process, successively let the reference working distance z c vary from 6 km to 12 km, obtain the corresponding reconstructed images, use the Laplacian gradient as the evaluation function to evaluate the reconstructed images of the "drone area" and the "car area", and the results are as Figure 6 shown, reaching the maximum values at 8.034 km and 9.997 km respectively. The reference working distance z c When taking 8.034 km and 9.997 km respectively, the reconstructed images are as Figure 7 (a) and Figure 7 (b) shown. Figure 7 (c) to 7(f) respectively show the reconstructed images of the "drone area" and the "car area", where Figure 7 (c) is the drone area image when the reference distance is 8.034 km, Figure 7 (d) is the "car" area image when the reference distance is 8.034 km. It can be seen that when the reference working distance is 8.034 km, the reconstructed image of the ground part is relatively blurred and stripes appear; Figure 7 (e) is the drone area image when the reference distance is 9.997 km, Figure 7 (f) is the "car" area image when the reference distance is 9.997 km. It can be seen that when the reference working distance is 9.997 km, the ground part is relatively clear, and only the clarity of the drone area image is affected. From this, it can be seen that when the reference working distance is close to the actual working distance of the drone, the drone is relatively clear and the car is relatively blurred. When the reference working distance is close to the actual working distance of the ground, the car is relatively clear and the drone is relatively blurred.

[0054] Taking 8.034 km as the distance of the drone, the size of the reconstructed image is calculated as 19.28 m × 19.28 m at this time. According to the relative position of the drone in the reconstructed image, its size is calculated as 1.80 m × 1.44 m, and the coordinates of its center point are (-2.20 m, -4.02 m, 8034 m). Taking 9.997 km as the distance of the ground, the size of the reconstructed image is calculated as 23.99 m × 23.99 m at this time. According to the relative position of the vehicle area in the reconstructed image, the coordinates of its center point are calculated as (3.96 m, 9.82 m, 9997 m), and the length of the vehicle is calculated as 2.83 m.

[0055] The simulation experiment results show that adjusting the value of the reference working distance z c can change the clarity of different target images in the reconstructed image, and the value of the reference working distance z c at which the reconstructed image of the target of interest is the clearest is its actual working distance. By using some image segmentation methods and autofocus algorithms to find the value of the reference working distance at which each distance target table is the clearest, the distances and sizes of each target can be estimated.

Claims

1. A passive three-dimensional imaging method based on an optical interference computational imaging method, characterized in that: The first step: The apertures are relatively discrete with respect to the midpoint of the array baseline, that is, the centers of the baselines formed by each pair of apertures do not coincide, or at least several groups do not coincide. For an optical interference computational imaging system, the cross-correlation intensity of the object light spatial frequency domain is interferometrically recorded. The second step: Stepwise adjust the reference working distance within a certain range to compensate for the phase difference of the cross-correlation intensity of each spatial frequency domain corresponding to the baselines of different aperture pairs, and then inversely reconstruct the object-space image through the Fourier transform algorithm. The third step: Adopt an image optimization evaluation algorithm to evaluate the sharpness of each reconstructed object-space image, obtain a reconstructed image with a clear or locally clear object-space scene target, and the corresponding reference working distance. The fourth step: Based on the clear or partial image and the corresponding reference working distance, calculate the relative position and size information of the target of interest in the image, and then reconstruct the three-dimensional image of the object-space scene, completing the passive three-dimensional imaging and image reconstruction of the object-space scene.

Images