Array single photon detection and two-dimensional camera co-path fusion device and pixel matching method

Abstract

Array single photon detection and two-dimensional camera common path fusion device and pixel matching method belong to the field of optical precision measurement. The array single photon detection and two-dimensional camera common path fusion device comprises a signal generator, a laser, an emitting optical system, a receiving optical system, a dichroic mirror, a two-dimensional camera, an array single photon detector, an array single photon counter and a data acquisition and processing fusion module. The method of the present application is as follows: the array single photon detection waveband pulsed laser and the two-dimensional camera sensing waveband ambient light returned by the target object are separated by using a common receiving optical path and a dichroic mirror to determine the pixel size and the number of the array single photon detector and the two-dimensional camera; the transmission light offset d is calculated according to the thickness of the dichroic mirror; the transmission light offset d is used as a correction factor to realize the pixel matching of the array single photon detection three-dimensional imaging and the two-dimensional camera image; the array single photon detection three-dimensional image and the two-dimensional camera image are superimposed, and the high resolution of the two-dimensional image is used to enrich the three-dimensional point cloud imaging information.

Description

Technical Field

[0001] This invention belongs to the field of optical precision measurement and relates to an array single-photon detection and two-dimensional camera co-path fusion device and a pixel matching method. Background Technology

[0002] Long-range, high-precision target localization and recognition is of great significance in fields such as obstacle location and early warning, remote sensing, and geological disaster early warning. Single-photon detection imaging technology belongs to the next generation of quantum imaging technology, possessing picosecond-level accuracy and photon-level sensitivity. Combined with time-correlated single-photon counting technology, it can achieve long-range 3D imaging. However, due to the time accumulation required, single-point scanning single-photon imaging is very slow. Array single-photon detection imaging can achieve long-range single-photon imaging of relatively fast dynamic targets, but the number of pixels in existing array single-photon cameras is relatively small, generally 64×64 or 128×128, resulting in a sparse point cloud, making it difficult to distinguish the details of target objects at long distances. Two-dimensional cameras are relatively mature, offering grayscale and color imaging, with pixel counts easily reaching millions or even higher, capable of obtaining rich details and texture information of target objects, but lacking distance and depth information. Image fusion of array single-photon detection imaging and two-dimensional cameras can realize multi-dimensional information of the target, making target recognition and classification easier. Existing multi-sensor image fusion methods mostly involve fusing independent sensor systems through field-of-view registration. This can easily lead to field-of-view mismatch when zooming at different distances, and also requires high backend computing power. Summary of the Invention

[0003] To achieve multi-sensor fusion matching for target localization and recognition at different focal lengths over long distances, the present invention aims to provide a co-path fusion device and pixel matching method for array single-photon detection and two-dimensional camera. A signal generator is used to synchronize the laser and array single-photon counter. A common-receiver optical system and a dichroic mirror are used to achieve co-field matching reception of the laser in the array single-photon detector band and the light in the two-dimensional camera sensing band. A pixel correction matching algorithm is used to achieve image fusion of the array single-photon detection three-dimensional imaging and the two-dimensional image, thereby obtaining richer multi-dimensional information about the imaging target.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] The array single-photon detector and two-dimensional camera co-path fusion device disclosed in this invention includes a signal generator, a laser, a transmitting optical system, a receiving optical system, a dichroic mirror, a two-dimensional camera, an array single-photon detector, an array single-photon counter, and a data acquisition, processing, and fusion module. The signal generator provides synchronization pulse signals to the laser and the array single-photon counter. The pulsed laser emitted by the laser is in the sensing band of the array single-photon detector. The pulsed laser illuminates the target object through the transmitting optical system. The pulsed laser and the ambient light in the sensing band of the 2D camera are not in the same band. The reflected light from the target object includes both the pulsed laser and the ambient light in the sensing band of the 2D camera. The reflected light is received by the receiving optical system and then separated from the pulsed laser in the sensing band of the array single-photon detector and the ambient light in the sensing band of the 2D camera by a dichroic mirror. The pulsed laser is detected by the array single-photon detector and converted into an electrical signal, which is sent to the array single-photon counter. After histogram accumulation and statistics, the pulsed laser flight time and target distance value corresponding to each pixel of the array single-photon detector are extracted and calculated. The target distance value is sent to the data acquisition, processing and fusion module to generate a 3D point cloud image. The ambient light is sensed and acquired by the 2D camera to generate a 2D image, which is sent to the data acquisition, processing and fusion module. The data acquisition, processing and fusion module performs pixel correction matching on the 3D point cloud image and the 2D image to achieve image fusion of the array single-photon detector's 3D imaging and the 2D image, obtaining richer multidimensional information about the imaging target.

[0006] The data acquisition, processing, and fusion module performs pixel correction matching on the pixels of the array of single-photon detectors and the pixels of the two-dimensional camera. The specific implementation method is as follows:

[0007] The array of single-photon detectors has m pixels. 水平 ×n 竖直 The pixel pitch is d1, and the number of pixels in a 2D camera is a. 水平 ×b 竖直 Given a pixel pitch of d², a dichroic mirror thickness of L, and a transmitted light shift of d, and considering that the dichroic mirror is placed at a 45° angle relative to the incident light, resulting in an incident angle of 45°, and the dichroic mirror's refractive index n, the transmitted light shift d is calculated using geometric optics as follows:

[0008]

[0009] If the direction of the transmitted light translation corresponding to the dichroic mirror is either horizontal or vertical, then the number of pixels x1 that the transmitted light translation d corresponds to in the array single-photon detector is:

[0010]

[0011] The number of pixels shifted by the 2D camera corresponding to the transmitted light translation d is x2.

[0012]

[0013] Pixel matching is achieved by moving either the single-photon array detector pixel position or the 2D camera pixel position. If moving the single-photon array detector pixel, the pixel is shifted by x1 pixels along the transmission light translation direction to achieve pixel matching. If moving the 2D camera pixel, the pixel is shifted by x2 pixels along the transmission light translation direction to achieve pixel matching, thus realizing pixel matching and fusion of the 3D imaging from the single-photon array detector and the 2D camera image.

[0014] In an array single-photon counter, the method for calculating flight time and target distance is as follows: A signal generator provides synchronization pulse signals to both the laser and the array single-photon counter. The array single-photon counter records the start time t0. The laser emits a pulsed laser beam, which illuminates the target object through the transmitting optical system. The reflected pulsed laser beam is received by the receiving optical system, passes through a dichroic mirror, and is received by the array single-photon detector, converted into an electrical signal, and input to the array single-photon counter, which records the end time t1. Because the signal after reflection is relatively weak, multiple pulse cycles are required for accumulation. Histogram statistics are used to reconstruct the pulse shape and position corresponding to the target object. The time point t corresponding to the target object is extracted using a pulse positioning method. 目标 Obtain the target object's flight time Δt = t 目标 -t0, the target distance value D is obtained based on the photon flight time, where the refractive index of the laser wavelength in air is n. g .

[0015]

[0016] The array single-photon detector is one of the following: Geiger mode avalanche diode array single-photon detector, superconducting nanowire array single-photon detector, or photomultiplier tube array single-photon detector.

[0017] A method for co-path pixel matching of array single-photon detection and two-dimensional camera is implemented based on the aforementioned array single-photon detection and two-dimensional camera co-path fusion device. The method includes the following steps:

[0018] The first step involves using a common receiving optical path and a dichroic mirror to separate the pulsed laser light from the array single-photon detector band and the ambient light from the 2D camera's sensing band. The second step involves determining the pixel size and number of the array single-photon detector and the 2D camera. The third step involves calculating the transmitted light offset d based on the thickness of the dichroic mirror. The fourth step involves using the transmitted light offset d as a correction factor to achieve pixel matching between the array single-photon detector's 3D imaging and the 2D camera image. The fifth step involves superimposing the array single-photon detector's 3D image and the 2D camera image, utilizing the high resolution of the 2D image to enrich the 3D point cloud imaging information.

[0019] Beneficial effects:

[0020] 1. The array single-photon detector and two-dimensional camera co-path fusion device and pixel matching method disclosed in this invention provide synchronization pulse signals to the laser and the array single-photon counter respectively. The pulsed laser emitted by the laser is in the sensing band of the array single-photon detector. The pulsed laser illuminates the target object through the emitting optical system. The pulsed laser and the ambient light in the sensing band of the two-dimensional camera are not in the same band. The reflected light from the target object includes both the pulsed laser and the ambient light in the sensing band of the two-dimensional camera. The reflected light is received by the receiving optical system and then separated from the pulsed laser in the sensing band of the array single-photon detector and the ambient light in the sensing band of the two-dimensional camera by a dichroic mirror. The pulsed laser is detected by the array single-photon detector and converted into an electrical signal, which is sent to the array single-photon counter. After histogram accumulation and statistics, the time of flight of the pulsed laser is extracted and sent to the data acquisition, processing and fusion module to generate a three-dimensional point cloud image. The ambient light is sensed and acquired by the two-dimensional camera to generate a two-dimensional image, which is sent to the data acquisition, processing and fusion module. The data acquisition, processing and fusion module performs pixel correction matching on the three-dimensional point cloud image and the two-dimensional image to achieve image fusion of the array single-photon detector three-dimensional imaging and the two-dimensional image.

[0021] 2. Traditional multi-sensor fusion suffers from complexity, difficulty in field-of-view registration, and limited imaging range due to the different receiving optical systems and fields of view. The array single-photon detection and 2D camera co-path fusion device and pixel matching method disclosed in this invention fuse array single-photon detection imaging and 2D camera images using a common receiving optical system, a dichroic mirror, and a pixel correction matching method. This solves the problems of field-of-view mismatch, registration difficulties, and short imaging distance in existing multi-sensor fusion methods, acquiring richer multi-dimensional information about the imaging target. Furthermore, array single-photon detection imaging, compared to traditional 3D imaging radar, has photon-level sensitivity and picosecond-level accuracy, and does not require scanning imaging. After fusion with a 2D camera, it can achieve high frame rate, rapid imaging of moving and static targets at ultra-long distances of tens of kilometers using single-photon 3D imaging and 2D camera matching fusion imaging. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a co-path fusion device based on array single-photon detection and a two-dimensional camera;

[0023] Figure 2 A schematic diagram of array single-photon detection and two-dimensional camera pixel matching;

[0024] Among them, 1—signal generator, 2—laser, 3—emitting optical system, 4—target object, 5—receiving optical system, 6—dichroic mirror, 7—two-dimensional camera, 8—array single-photon detector, 9—array single-photon counter, and 10—data acquisition, processing, and fusion module. Detailed Implementation

[0025] To better illustrate the purpose and advantages of the present invention, the invention will be further described below in conjunction with the accompanying drawings and examples.

[0026] Example 1:

[0027] like Figure 1 As shown, the array single-photon detector and two-dimensional camera co-path fusion device disclosed in this embodiment includes a signal generator 1, a laser 2, an emitting optical system 3, a receiving optical system 5, a dichroic mirror 6, a two-dimensional camera 7, an array single-photon detector 8, an array single-photon counter 9, and a data acquisition, processing, and fusion module 10. The signal generator 1 provides synchronization pulse signals to the laser 2 and the array single-photon counter 9 respectively. The pulsed laser emitted by the laser 2 has a wavelength of 1550nm. The array single-photon detector is made of indium gallium arsenide material, and its sensing band is near-infrared 900nm~1600nm. The pulsed laser emitted by the laser 2 is within the sensing band of the array single-photon detector 8. The pulsed laser illuminates the target object 4 through the emitting optical system 3. The pulsed laser and the ambient light in the sensing band of the two-dimensional camera 7 are not in the same band. The sensing band of the two-dimensional camera is 400nm~900nm, mainly in the visible light band. The reflected light from the target object 4 includes both the pulsed laser and the ambient light in the sensing band of the two-dimensional camera 7. The reflected light is received by the receiving optical system 5 and then passed through the dichroic mirror 6 to amplify the array single-photon detector. The pulsed laser in the sensing band of the single-photon detector 8 and the ambient light in the sensing band of the two-dimensional camera 7 are separated. The dichroic mirror 6 is a reflection band in the 400nm-900nm band and a transmission band in the 900nm-1600nm band. Its thickness L is 1mm and its refractive index is 1.5. Therefore, the 1550nm band sensed by the array single-photon detector 8 is transmitted, and the visible light band sensed by the two-dimensional camera 7 is reflected. The pulsed laser is detected by the array single-photon detector 8 and converted into an electrical signal, which is sent to the array single-photon counter 9. After histogram accumulation and statistics, the pulsed laser flight time and target distance value corresponding to each pixel of the array single-photon detector 8 are extracted and calculated. The target distance value is sent to the data acquisition, processing and fusion module 10 to generate a three-dimensional point cloud image. The ambient light is sensed by the two-dimensional camera 7 and acquired to generate a two-dimensional image, which is sent to the data acquisition, processing and fusion module 10. The data acquisition, processing and fusion module 10 performs pixel correction matching on the three-dimensional point cloud image and the two-dimensional image to obtain the image fusion of the array single-photon detector three-dimensional imaging and the two-dimensional image, and obtain richer multi-dimensional information of the imaging target.

[0028] The data acquisition, processing, and fusion module 10 performs pixel correction matching on the 8 pixels of the array single-photon detector and the 7 pixels of the 2D camera. The specific implementation method is as follows:

[0029] like Figure 2 As shown, the array single-photon detector 8 has m pixels.水平 ×n 竖直 =64×64, pixel pitch d1=50μm, the number of pixels a in 2D camera 7 水平 ×b 竖直 =1000×1000, pixel pitch is d2=10μm, dichroic mirror 6 reflects light in the 400nm~900nm wavelength range and transmits light in the 900nm~1600nm wavelength range, thickness is L=1mm, the translation amount d of the transmitted light is calculated as follows: Since dichroic mirror 6 is placed at a 45° angle relative to the incident light, the incident angle is 45°. The refractive index of dichroic mirror 6 is n=1.5. Therefore, through geometric optics calculations, the translation amount d of the transmitted light is...

[0030]

[0031] If the direction of the transmitted light translation corresponding to the dichroic mirror 6 is either horizontal or vertical, then the number of pixels x1 that the array single-photon detector 8 is translated by the transmitted light translation d is...

[0032]

[0033] The number of pixels shifted by the 2D camera 7 corresponding to the transmitted light translation d is x2.

[0034]

[0035] Pixel matching is achieved by moving either the 8-pixel position of the array single-photon detector or the 7-pixel position of the 2D camera. If the 8-pixel array single-photon detector is moved, pixel matching is achieved by correcting the translation direction along the transmitted light by 6.6 pixels of the array single-photon detector. If the 7-pixel 2D camera is moved, pixel matching is achieved by correcting the translation direction along the transmitted light by 33 pixels of the 2D camera. This enables pixel matching and fusion of the array single-photon detector 3D imaging and the 2D camera 7 image.

[0036] In the array single-photon counter 9, the method for calculating the flight time and target distance is as follows: Signal generator 1 provides synchronization pulse signals to laser 2 and array single-photon counter 9 respectively. Array single-photon counter 9 records the start time t0. Laser 2 emits pulsed laser light, which is transmitted through the emitting optical system 3 and illuminates the target object 4. The reflected pulsed laser light is received by the receiving optical system 5, passes through the dichroic mirror 6, and is received by the array single-photon detector 8, converted into an electrical signal, and input to array single-photon counter 9, which records the end time t1. Since the signal after the pulsed laser light is reflected is relatively weak, it requires the accumulation of multiple pulse cycles. After histogram statistics, the pulse shape and position corresponding to the target object 4 are recovered. Through the pulse positioning method, the time point t corresponding to the target object 4 is extracted. 目标 , obtain the flight time of the target object 4 Δt=t 目标-t0, the target distance value D is obtained based on the photon flight time, where the refractive index of the laser wavelength in air is n. g .

[0037]

[0038] The array single-photon detector 8 is one of the following: Geiger mode avalanche diode array single-photon detector 8, superconducting nanowire array single-photon detector 8, or photomultiplier tube array single-photon detector 8.

[0039] The array single-photon detector and 2D camera common-path pixel matching method is implemented based on the array single-photon detector and 2D camera common-path fusion device. The specific implementation steps of the array single-photon detector and 2D camera common-path pixel matching method are as follows:

[0040] The first step involves using a common receiving optical path and a dichroic mirror 6 to separate the pulsed laser light in the array single-photon detector band returned by the target object 4 from the ambient light in the sensing band of the 2D camera 7. The second step involves determining the pixel size and number of the array single-photon detector 8 and the 2D camera 7. The third step involves calculating the transmitted light offset d based on the thickness of the dichroic mirror 6. The fourth step involves using the transmitted light offset d as a correction factor to achieve pixel matching between the array single-photon detector 3D image and the 2D camera 7 image. The fifth step involves superimposing the array single-photon detector 3D image and the 2D camera 7 image, utilizing the high resolution of the 2D image to enrich the 3D point cloud imaging information.

[0041] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A device for fusion of array single-photon detection and two-dimensional camera co-path, characterized in that: The system includes a signal generator, a laser, a transmitting optical system, a receiving optical system, a dichroic mirror, a 2D camera, an array of single-photon detectors, an array of single-photon counters, and a data acquisition, processing, and fusion module. The signal generator provides synchronization pulse signals to both the laser and the array of single-photon counters. The pulsed laser emitted by the laser is within the sensing band of the array of single-photon detectors. The pulsed laser illuminates the target object through the transmitting optical system. The pulsed laser and the ambient light within the sensing band of the 2D camera are not in the same band. The reflected light from the target object includes both the pulsed laser and the ambient light within the sensing band of the 2D camera. The reflected light is received by the receiving optical system and then passes through the dichroic mirror to combine the pulsed laser light within the sensing band of the array of single-photon detectors with the ambient light within the sensing band of the 2D camera. The ambient light in the two-dimensional camera's sensing band is separated. The pulsed laser is detected by the array single-photon detector and converted into an electrical signal, which is sent to the array single-photon counter. After histogram accumulation and statistics, the flight time of the pulsed laser and the target distance value corresponding to each pixel of the array single-photon detector are extracted and calculated. The target distance value is sent to the data acquisition, processing and fusion module to generate a three-dimensional point cloud image. The ambient light is sensed and acquired by the two-dimensional camera to generate a two-dimensional image, which is sent to the data acquisition, processing and fusion module. The data acquisition, processing and fusion module performs pixel correction matching on the three-dimensional point cloud image and the two-dimensional image to obtain the image fusion of the array single-photon detector's three-dimensional imaging and the two-dimensional image, thus obtaining richer multi-dimensional information of the imaging target.

2. The array single-photon detection and two-dimensional camera co-path fusion device as described in claim 1, characterized in that: The data acquisition, processing, and fusion module performs pixel correction matching on the pixels of the array single-photon detector and the pixels of the two-dimensional camera. The specific implementation method is as follows: The array of single-photon detectors has m pixels. 水平 ×n 竖直 The pixel pitch is d1, and the number of pixels in the 2D camera is a. 水平 ×b 竖直 Given a pixel pitch of d², a dichroic mirror thickness of L, and a transmitted light shift of d, and considering that the dichroic mirror is placed at a 45° angle relative to the incident light, resulting in an incident angle of 45°, and the dichroic mirror's refractive index n, the transmitted light shift d is calculated using geometric optics as follows: If the direction of the transmitted light translation corresponding to the dichroic mirror is either horizontal or vertical, then the number of pixels x1 that the transmitted light translation d corresponds to in the array single-photon detector is: The number of pixels shifted by the 2D camera corresponding to the transmitted light translation d is x2. Pixel matching is achieved by moving the pixel position of the array single-photon detector or the pixel position of the two-dimensional camera. If the array single-photon detector pixel is moved, the pixel matching is achieved by correcting the translation of the array single-photon detector pixel by x1 pixels along the direction of transmission light translation. If the two-dimensional camera pixel is moved, the pixel matching is achieved by correcting the translation of the two-dimensional camera pixel by x2 pixels along the direction of transmission light translation. This realizes pixel matching and fusion of array single-photon detector three-dimensional imaging and two-dimensional camera image.

3. The array single-photon detection and two-dimensional camera co-path fusion device as described in claim 1, characterized in that: In an array single-photon counter, the method for calculating flight time and target distance is as follows: A signal generator provides synchronization pulse signals to both the laser and the array single-photon counter. The array single-photon counter records the start time t0. The laser emits a pulsed laser beam, which illuminates the target object through the transmitting optical system. The reflected pulsed laser beam is received by the receiving optical system, passes through a dichroic mirror, and is received by the array single-photon detector, converted into an electrical signal, and input to the array single-photon counter, which records the end time t1. Because the signal after reflection is relatively weak, multiple pulse cycles are required for accumulation. Histogram statistics are used to reconstruct the pulse shape and position corresponding to the target object. The time point t corresponding to the target object is extracted using a pulse positioning method. 目标 Obtain the target object's flight time Δt = t 目标 -t0, the target distance value D is obtained based on the photon flight time, where the refractive index of the laser wavelength in air is n. g ; 。 4. The array single-photon detection and two-dimensional camera co-path fusion device as described in claim 1, characterized in that: The array single-photon detector is one of the following: Geiger mode avalanche diode array single-photon detector, superconducting nanowire array single-photon detector, or photomultiplier tube array single-photon detector.

5. A method for matching pixels in a common optical path of array single-photon detection and two-dimensional camera, implemented based on the array single-photon detection and two-dimensional camera common-path fusion device as described in claim 1, 2, 3 or 4, characterized in that: Includes the following steps, The first step is to use a common receiving optical path and a dichroic mirror to separate the array single-photon detection band pulsed laser returned by the target object from the ambient light in the two-dimensional camera sensing band. The second step is to determine the pixel size and number of the array single-photon detector and the two-dimensional camera; the third step is to calculate the transmitted light offset d based on the thickness of the dichroic mirror; the fourth step is to use the transmitted light offset d as a correction factor to achieve pixel matching between the array single-photon detector's three-dimensional imaging and the two-dimensional camera image; the fifth step is to superimpose the array single-photon detector's three-dimensional image and the two-dimensional camera image, and use the high resolution of the two-dimensional image to enrich the three-dimensional point cloud imaging information.

Images