Non-line-of-sight imaging device and non-line-of-sight imaging method

By dividing molecular areas on the intermediary wall and using galvanomic interleaving scanning technology, combining non-sight reconstruction algorithm and motion compensation module, the problem of slow imaging speed in complex scenarios of existing optical non-sight imaging technology is solved, and the problem of slow imaging speed in complex scenarios is realized. Fast real-time imaging and high-resolution imaging of complex scenes.

CN118655546BActive Publication Date: 2024-11-05UNIV OF SCI & TECH OF CHINA +1
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411127335.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-05
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

The existing optical non-field of vision imaging technology has slow imaging speed in complex scenarios, which cannot meet the real-time needs of applications such as autonomous driving and fire protection and disaster relief, and is poorly robust to environmental disturbances.

Method used

A non-sight imaging device is designed to realize fast interleaved scanning by dividing M×N sub-regions on the intermediary wall, using the transmitting galvanometer and the receiving galvanometer to change the angle in a time-to-sequential manner, and combining a time-digital converter and a processor. Obtain the flight time of photons, use non-sight reconstruction algorithm and motion compensation module to improve imaging efficiency and accuracy.

Benefits of technology

It realizes fast real-time imaging of complex scenes, improves imaging resolution and robustness, and can acquire more photons in a short time, meeting the needs of real-time non-domain imaging.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118655546B_ABST
    Figure CN118655546B_ABST
Patent Text Reader

Abstract

The present invention provides a non-line-of-sight imaging device and a non-line-of-sight imaging method. The non-line-of-sight imaging device comprises a laser emitting a pulsed laser, a beam splitter splitting the pulsed laser into a synchronous light and an imaging light, a first photodetector converting the synchronous light into a synchronous electrical signal, a transmitting galvanometer sequentially changing the emission angle, sequentially emitting the imaging light to a plurality of sub-areas on an intermediate wall, so that the imaging light is diffusely reflected into the non-line-of-sight space through each sub-area, a receiving galvanometer sequentially receiving a received light signal diffusely reflected by an object and then by each sub-area; a second photodetector sequentially photoelectrically converting the received light signal into a received electrical signal; a time-to-digital converter sequentially recording a first receiving moment of the synchronous electrical signal and a second receiving moment of the received electrical signal; a processor respectively acquiring the flight time of the photons of the imaging light according to the first receiving moment and the second receiving moment, and acquiring a non-line-of-sight image of the non-line-of-sight space according to the flight time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of optical non-line-of-sight imaging, and in particular to a non-line-of-sight imaging device and a non-line-of-sight imaging method. Background Art

[0002] Traditional imaging equipment can often only image scenes within the field of view, while non-line-of-sight imaging is designed to see hidden scenes behind obstructions. Non-line-of-sight imaging radar makes it possible to detect targets within blind spots, and can track and image objects outside the field of view at the kilometer level. This technology's ability to detect hidden scenes can be applied in many fields. For example, in machine vision and autonomous navigation, non-line-of-sight imaging radar can allow automatic navigation equipment to predict in advance whether there are objects passing around corners in complex environments, thereby avoiding collisions; in disaster relief, firefighters can use non-line-of-sight imaging radar to detect and locate the position of trapped people, which makes rescue safer and more efficient;

[0003] In the related art, a variety of methods have been proposed to achieve optical non-line-of-sight imaging. For example, there is a technical solution that uses laser emission wavefront modulation, which uses a phase modulator to control the laser illuminating the intermediate wall, so that the laser still maintains a certain propagation direction after diffuse reflection on the wall, and realizes the detection of all directions of the hidden space. However, this solution is sensitive to environmental disturbances and can only be implemented in the laboratory. In actual applications, it is easily affected by atmospheric disturbances and vibrations, and has poor robustness.

[0004] In addition, there is a non-line-of-sight imaging method based on the spatial coherence of speckles. This method emits coherent lasers to illuminate the hidden scene, and the reflected light of the hidden object will produce a speckle pattern on the intermediate wall. The speckle pattern with a memory effect can be captured by a camera, and the spatial coherence of the speckle pattern can be used to restore the information of the obscured scene. However, the main disadvantage of the non-line-of-sight imaging method based on speckle patterns is that the field of view is relatively small, because the memory effect is only effective within a few degrees.

[0005] Among many optical imaging methods, the non-line-of-sight imaging scheme based on photon time resolution is the current mainstream research direction and the most likely scheme for practical application. The time-resolved non-line-of-sight imaging system emits pulsed lasers to the hidden scene and uses time-correlated single-photon detection technology to measure the flight time of photons returning from the hidden scene with picosecond time accuracy. Based on the scanning of the intermediate wall and the flight time information of multiple diffusely reflected photons, three-dimensional imaging with centimeter-level resolution can be achieved. At present, the non-line-of-sight system based on single-photon detection can achieve real-time three-dimensional reconstruction of room-sized hidden scenes.

[0006] However, the current non-line-of-sight imaging performance still cannot meet the practical requirements. For applications such as autonomous driving and firefighting and disaster relief, real-time non-line-of-sight imaging is generally required. However, for non-line-of-sight imaging of complex scenes the size of a room, the current system often takes tens of seconds to form an image. This is because the non-line-of-sight imaging system needs to collect enough photons to reconstruct the algorithm. However, in actual applications, the signal photons that the system can receive are very weak, thus limiting the imaging speed. Summary of the invention

[0007] In view of this, the present invention provides a non-line-of-sight imaging device and a non-line-of-sight imaging method, which can perform rapid real-time imaging of the non-line-of-sight space of a complex scene.

[0008] As one aspect of an embodiment of the present invention, a non-line-of-sight imaging device is provided, wherein the non-line-of-sight space where the object is located is separated from the collection space where the non-line-of-sight imaging device is located by an obstacle, and an intermediate wall is arranged outside the non-line-of-sight space and the collection space, which is spaced from the obstacle, and the non-line-of-sight imaging device includes a laser, a beam splitter, a first photodetector, a transmitting galvanometer, a receiving galvanometer, a second photodetector, a time-to-digital converter, and a processor. The laser is suitable for emitting pulsed laser, the beam splitter splits the pulsed laser into synchronization light and imaging light, and the first photodetector is suitable for converting the synchronization light into a synchronization electrical signal. The transmitting galvanometer is suitable for sequentially changing the emission angle, sequentially emitting the imaging light to multiple sub-areas on the intermediate wall, so that the imaging light is diffusely reflected into the non-line-of-sight space through each of the sub-areas, wherein the surface of the intermediate wall facing the non-line-of-sight space and the collection space is divided into M×N sub-areas, wherein M and N are both positive integers. The receiving galvanometer is adapted to sequentially change the receiving angle, and sequentially receive a plurality of received light signals diffusely reflected by each of the above sub-areas after diffuse reflection by the above object. The second photodetector is adapted to sequentially photoelectrically convert the plurality of received light signals into a plurality of received electrical signals. The time-to-digital converter is adapted to sequentially record the first receiving moment of the above synchronous electrical signal, and the second receiving moment of receiving the above received electrical signal. The processor is adapted to obtain a plurality of flight times of the photons of the above imaging light diffusely reflected from the M×N above sub-areas according to the above first receiving moment and the above second receiving moment, respectively, and obtain a plurality of single-frame images according to the plurality of flight times, thereby obtaining a non-field of view image of the above non-field of view space.

[0009] According to an embodiment of the present invention, each of the above-mentioned sub-regions includes a scanning point matrix, the above-mentioned scanning point matrix includes K scanning points, the above-mentioned k-1th scanning point and the kth scanning point are located at adjacent positions in the above-mentioned scanning point matrix, k is a positive integer less than or equal to K, and K is a positive integer; wherein the above-mentioned transmitting galvanometer and the above-mentioned receiving galvanometer scan the kth scanning points in the 1st to M×Nth sub-regions of the M×N above-mentioned sub-regions in sequence according to the regional scanning order to obtain K single-frame images, wherein the above-mentioned regional scanning order is set based on the position of each of the above-mentioned sub-regions in the M×N above-mentioned sub-regions.

[0010] According to an embodiment of the present invention, the processor includes a reconstruction module, a boundary acquisition module, a motion estimation module, a compensation module and a restoration module. The reconstruction module is configured to reconstruct the plurality of flight times using a non-field of view reconstruction algorithm to obtain the plurality of single-frame images. The boundary acquisition module is configured to cut out the motion images including the images representing the moving objects in the plurality of single-frame images to obtain segmented images, and segment each of the segmented images using a projection segmentation method to obtain the three-dimensional boundary of the motion image. The motion estimation module is configured to obtain the motion state of each of the moving objects using a motion detection algorithm for each of the segmented images according to the three-dimensional boundary. The compensation module is configured to perform motion compensation on the plurality of segmented images according to the motion state, align and superimpose the motion images in the plurality of segmented images, and obtain a compensated image after motion compensation for the motion image. The restoration module is configured to restore the motion image to each of the single-frame images according to the compensation image to obtain the non-field of view image.

[0011] According to an embodiment of the present invention, the boundary acquisition module includes a motion determination unit, a first projection unit, a first segmentation unit, a second projection unit, a second segmentation unit, a third projection unit, and a boundary determination unit. The motion determination unit is configured to determine a still image and a moving image representing a still object according to a plurality of single-frame images, and eliminate the still image as a background to obtain a segmented image. The first projection unit is configured to perform a one-dimensional projection on the first frame segmented image along a first direction to obtain a first projection image. The first segmentation unit is configured to use a position where the projection value is 0 as a segmentation point, and to segment the first projection image into a plurality of slices along the segmentation point. The second projection unit is configured to perform a one-dimensional projection on each object slice along a second direction to obtain a second projection image, wherein the second direction is perpendicular to the first direction, and the object slice is a slice containing an element of a moving object in the plurality of slices. The second segmentation unit is configured to use a position where the projection value is 0 as a segmentation point, and to segment each of the object slices into a plurality of cylindrical strips. The third projection unit is configured to perform one-dimensional projection on each object strip along the third direction to obtain a third projection image, wherein the position with a projection value of 0 in the third projection image is a boundary strip in the object strip in the hollow area, wherein the third direction is perpendicular to the first direction and the second direction, the object strip is a cylindrical strip containing elements of the moving object in the plurality of cylindrical strips, and the boundary strip is an object strip containing the edge of the moving object in the plurality of object strips. The boundary determination unit is configured to perform one-dimensional projection on the boundary strips along the first direction, the second direction and the third direction respectively, thereby determining the boundary pixel positions of the moving image in the first direction, the second direction and the third direction, thereby determining the three-dimensional boundary.

[0012] According to an embodiment of the present invention, the motion estimation module includes a segmentation unit and a motion state calculation unit. The segmentation unit is configured to segment the plurality of segmented images using a three-dimensional coordinate frame according to the three-dimensional boundary to obtain sub-object images. The motion state calculation unit is configured to sequentially calculate the motion state of each of the sub-object images by using a cross-correlation function for the segmented image of the current frame and the segmented image of the next frame.

[0013] According to an embodiment of the present invention, the motion state calculation unit includes a maximum projection unit, a first calculation unit, and a second calculation unit. The maximum projection unit is configured to perform maximum projection on the sub-object image to obtain a front view and a top view. The first calculation unit is configured to calculate the first translation of the moving object in the second direction and the third direction and the first rotation around the first axis from the front view by a cross-correlation function, wherein the first axis is parallel to the first direction. The second calculation unit is configured to calculate the second translation of the moving object in the first direction and the second direction and the second rotation around the third axis from the top view by a cross-correlation function, wherein the third axis is parallel to the third direction. Wherein, the motion state includes at least one of the first translation, the first rotation, the second translation, and the second rotation.

[0014] According to an embodiment of the present invention, the compensation module includes a target determination unit, a transformation unit and a superposition unit. The target determination unit is configured to determine a target area containing the motion image from each of the segmented images according to the three-dimensional boundary. The transformation unit is configured to use the first frame segmented image as a reference image, and perform inverse transformation on the target area in the segmented images of other frames except the first frame according to the motion state, so that the position and angle of the motion image in the segmented images of other frames are consistent with the position and angle of the motion image in the reference image, and obtain multiple single-frame target images. The superposition unit is configured to superimpose multiple single-frame target images to obtain the compensated image.

[0015] According to an embodiment of the present invention, the restoration module includes a motion restoration unit and an image restoration unit. The motion restoration unit is configured to restore the compensated image to the motion state corresponding to each of the segmented images in turn according to the compensated image and the motion state to obtain a restored image. The image restoration unit is configured to put the restored image back into the corresponding single-frame image to obtain the non-viewing area image.

[0016] According to an embodiment of the present invention, the processor further includes a recording module, and the recording module is configured to record a sequence number of the first receiving moment or the second receiving moment to correspond to sequence numbers of the plurality of sub-areas.

[0017] As another aspect of an embodiment of the present invention, a non-line-of-sight imaging method using any of the above non-line-of-sight imaging devices is provided, comprising:

[0018] sequentially acquiring multiple flight times of photons of imaging light in M×N sub-regions;

[0019] Reconstructing the images using a non-viewing area reconstruction algorithm according to the multiple flight times to obtain multiple single-frame images;

[0020] Cutting out the moving images including the images representing the moving objects in the plurality of single-frame images to obtain segmented images, and segmenting each of the segmented images by a projection segmentation method to obtain the three-dimensional boundary of the moving image;

[0021] According to the three-dimensional boundary, a motion detection algorithm is used on each of the segmented images to obtain the motion state of each of the moving objects;

[0022] According to the motion state, motion compensation is performed on the plurality of segmented images, so that the motion images in the plurality of segmented images are aligned and superimposed to obtain a compensated image after motion compensation is performed on the motion images;

[0023] The motion image is restored to each of the single-frame images according to the compensation image to obtain a non-viewing area image.

[0024] According to the non-field of view imaging device of the embodiment of the present invention, the emission of imaging light adopts a transmitting galvanometer, and the receiving light signal after diffuse reflection by the object and diffuse reflection by each sub-area adopts a receiving galvanometer, and the deflection of the transmitting galvanometer and the receiving galvanometer is controlled in a timely manner to control the direction of the emitted imaging light and the received light signal, and the M×N sub-areas are scanned in an interleaved and rapid manner on the intermediate wall, thereby improving the efficiency of image acquisition, and the separation of the light transmission and receiving paths is used to avoid the influence of local echo signals, thereby improving the imaging accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0026] Figure 1 A system diagram of a non-line-of-sight imaging device according to an embodiment of the present invention is shown;

[0027] Figure 2 A schematic diagram showing sub-areas and scanning points according to an embodiment of the present invention is shown;

[0028] Figure 3 A block diagram of a processor according to an embodiment of the present invention is shown;

[0029] Figure 4 An image of a non-viewing space according to an embodiment of the present invention is shown;

[0030] Figure 5 shows measurement data according to an embodiment of the present invention;

[0031] Figure 6 The imaging result of the non-viewing area space according to one embodiment of the present invention is shown;

[0032] Figure 7A flow chart of a non-line-of-sight imaging method according to an embodiment of the present invention is shown.

[0033] Description of reference numerals:

[0034] 1- Object;

[0035] 2- Non-line-of-sight imaging device;

[0036] 21-Laser;

[0037] 22-beam splitter;

[0038] 23- a first photodetector;

[0039] 24-transmitting galvanometer;

[0040] 25-receiving galvanometer;

[0041] 26- a second photodetector;

[0042] 27-Time to Digital Converter;

[0043] 28-processor;

[0044] 281-Reconstruction module;

[0045] 282-boundary acquisition module; 2821-motion determination unit; 2822-first projection unit; 2823-first segmentation unit; 2824-second projection unit; 2825-second segmentation unit; 2826-third projection unit; 2827-boundary determination unit;

[0046] 283-motion estimation module; 2831-segmentation unit; 2832-motion state calculation unit;

[0047] 284-compensation module;

[0048] 285-Recovery module;

[0049] 286-recording module;

[0050] 3- Non-visual space;

[0051] 4- Collection space;

[0052] 5- Obstacles;

[0053] 6-Intermediary wall. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0055] Figure 1A system diagram of a non-line-of-sight imaging device according to an embodiment of the present invention is shown. Figure 2 A schematic diagram of sub-areas and scanning points according to an embodiment of the present invention is shown.

[0056] As one aspect of an embodiment of the present invention, a non-viewing area imaging device is provided, such as Figure 1 As shown, ( , p) represents the coordinates of the sub-area on the intermediate wall. The non-viewing space 3 where the object 1 is located is separated from the collection space 4 where the non-viewing imaging device 2 is located by an obstacle 5. An intermediate wall 6 is arranged outside the non-viewing space 3 and the collection space 4 and is spaced apart from the obstacle 5. The non-viewing imaging device 2 includes a laser 21, a beam splitter 22, a first photodetector 23, a transmitting galvanometer 24, a receiving galvanometer 25, a second photodetector 26, a time-to-digital converter 27 and a processor 28. The laser 21 is suitable for emitting pulsed laser, the beam splitter 22 splits the pulsed laser into synchronization light and imaging light, and the first photodetector 23 is suitable for converting the synchronization light into a synchronization electrical signal. The emitting galvanometer 24 is suitable for sequentially changing the emission angle, and sequentially emitting the imaging light through the above-mentioned interval to multiple sub-areas on the intermediate wall, so that the imaging light is diffusely reflected into the non-viewing space through each sub-area, wherein, as Figure 1 and Figure 2 As shown, the surface of the intermediate wall facing the non-viewing space and the collection space is divided into M×N sub-areas arranged in an array, where M and N are both positive integers. The receiving galvanometer 25 is suitable for sequentially changing the receiving angle, and sequentially receiving a plurality of received light signals that pass through the above-mentioned intervals after diffuse reflection by the object and diffuse reflection by each sub-area. The second photodetector 26 is suitable for sequentially photoelectrically converting the plurality of received light signals into a plurality of received electrical signals. The time-to-digital converter 27 is suitable for sequentially recording the first receiving moment of the synchronous electrical signal, and recording the second receiving moment of the received electrical signal. The processor 28 is suitable for acquiring a plurality of flight times of photons of the imaging light diffusely reflected from the M×N sub-areas according to the first receiving moment and the second receiving moment, respectively, and acquiring a plurality of single-frame images according to the plurality of flight times, thereby obtaining a non-viewing image of the non-viewing space.

[0057] According to the non-field of view imaging device of the embodiment of the present invention, the emission of imaging light adopts a transmitting galvanometer, and the receiving light signal after diffuse reflection by the object and diffuse reflection by each sub-area adopts a receiving galvanometer, and the deflection of the transmitting galvanometer and the receiving galvanometer is controlled in a timely manner to control the direction of the emitted imaging light and the received light signal, and the M×N sub-areas are scanned in an interleaved and rapid manner on the intermediate wall, thereby improving the efficiency of image acquisition, and the separation of the light transmission and receiving paths is used to avoid the influence of local echo signals, thereby improving the imaging accuracy.

[0058] In the process of implementing the present invention, it is found that in order to realize real-time non-line-of-sight imaging, the non-line-of-sight imaging device needs to perform uninterrupted grid scanning to collect data. Due to the limited scanning speed of the receiving galvanometer and the transmitting galvanometer, the galvanometer cannot realize dense scanning in one frame of imaging time, and can only perform sparse sampling within the scanning range, which affects the resolution and quality of the imaging.

[0059] According to an embodiment of the present invention, Figure 2 As shown, each sub-region includes a scanning point matrix, the scanning point matrix includes K scanning points, the k-1th scanning point and the kth scanning point are located at adjacent positions in the scanning point matrix, k is a positive integer less than or equal to K, and K is a positive integer. The transmitting galvanometer and the receiving galvanometer sequentially scan the kth scanning points in the 1st to the M×N sub-regions of the M×N sub-regions in accordance with the regional scanning order to obtain K single-frame images, wherein the regional scanning order is set based on the position of each sub-region in the M×N sub-regions.

[0060] According to an embodiment of the present invention, Figure 2 As shown, each sub-area is divided into A×B with a total of K scanning points. The imaging light is emitted to the scanning points corresponding to each sub-area in sequence according to the serial number of the acquired single-frame image, so that the scanning points of two adjacent single-frame images are staggered to obtain A×B with a total of K single-frame images, where A and B are both positive integers.

[0061] According to an embodiment of the present invention, the scanning area on the intermediate wall is discretized into regular M×N sub-areas, and each sub-area is divided into A×B K scanning points, where A and B are both positive integers. After completing a sparse sampling of the entire scanning area in one frame, the scanning points of the next frame are adjacent to the scanning points of the previous frame. This staggered scanning method is conducive to improving the resolution of imaging. After collecting multiple frames of data, the entire grid is completely scanned in sequence along an S-shaped route, and then the scanning point data of different frames can be combined to achieve equivalent full sampling, so as to achieve high-resolution reconstruction based on A×B single-frame images through a motion compensation algorithm, so that even when the number of photons and scanning points are limited, high-resolution reconstruction can be achieved by combining the results of different frames.

[0062] According to an embodiment of the present invention, the region scanning order may be an S-shaped (snake-shaped) scanning order, so that continuous and uninterrupted scanning of the M×N sub-regions can be achieved. The path for scanning the A×B scanning points may be an S-shaped (snake-shaped) path, so as to achieve continuous and uninterrupted scanning of the A×B scanning points. The scanning method of the S-shaped path ensures that, when a single frame image is collected, all sub-regions are visited once.

[0063] According to an embodiment of the present invention, the S-shaped path may be from left to right, or from right to left, or from top to bottom, or from bottom to top, and a suitable path may be selected according to specific needs.

[0064] In an illustrative embodiment, Figure 2 As shown, the surface of the intermediate wall facing the non-viewing space and the collection space is divided into 16×16 sub-areas, totaling 256 arranged in an array. Each sub-area is divided into 8×8 scanning points, totaling 64 arranged in an array. The transmitting galvanometer can be driven along the first S-shaped path to the scanning point numbered #1 in the 256 sub-areas, and the imaging light can be emitted in sequence from 1 to 256, and the receiving galvanometer can be driven to receive the received light signal from the scanning point numbered #1 in the 256 sub-areas in sequence. In this way, at the first receiving moment and the second receiving moment corresponding to each sub-area, the flight time of the photon corresponding to each sub-area is counted, and the first single-frame image is obtained according to the flight time of the 256 photons.

[0065] When acquiring the second single-frame image, the transmitting galvanometer can be driven to scan in the reverse direction of the first S-shaped path, that is, to sequentially emit imaging light to the scanning point numbered #2 in the sub-region from 256 to 1, and drive the receiving galvanometer to sequentially receive the received light signals from the scanning point numbered #2 in the 256 sub-regions. In this way, the first receiving moment and the second receiving moment corresponding to each sub-region are used to count the flight time of the photons corresponding to each sub-region, and the second single-frame image is acquired based on the flight time of the 256 photons. Similarly, the 64th single-frame image corresponding to the #64 scanning point is acquired.

[0066] In such an embodiment, each single-frame image needs to count the flight time of single photons in 16×16 sub-regions, and a total of 8×8 single-frame images are collected.

[0067] According to an embodiment of the present invention, the regional scanning order of the k-1th scanning point is the reverse order of the regional scanning order of the kth scanning point.

[0068] In an illustrative embodiment, the number of sub-areas arranged in an array into which the surface of the intermediate wall facing the non-viewing space and the acquisition space is divided may include any one of 8×8, 16×16, 8×16, 17×9, 10×10, 24×24, 30×20, and 36×72, etc. The number of scanning points arranged in an array into which each sub-area can be divided may include any one of 4×4, 8×8, 8×9, 13×15, 10×10, 24×24, 30×30, and 36×24, etc.

[0069] In an exemplary embodiment, the serpentine scanning path may be as follows: Figure 2The sub-areas shown are numbered from 1 to 256. Figure 2 In the perspective shown, start from the upper left corner and scan the first row to the right. After reaching the rightmost sub-area of ​​the first row, move down one grid and change direction to scan to the left. Continue scanning to the left until reaching the leftmost sub-area of ​​the row. Move down one grid and change direction to scan to the right. Repeat the above steps until all rows are scanned.

[0070] In an illustrative embodiment, the laser may be a picosecond pulse laser.

[0071] In an exemplary embodiment, the second photodetector may be a single photon avalanche diode (SPAD) with higher sensitivity.

[0072] In an illustrative embodiment, the first photodetector may be a phototube. It is understandable that the first photodetector may also be a single photon avalanche diode.

[0073] According to an embodiment of the present invention, the receiving optical system captures the diffusely reflected light of the object in the non-viewing space. The light diffusely reflected by the object in the non-viewing space is returned to the entire intermediate wall. Therefore, the receiving capability of the optical system can be described by the optical Lagrangian invariant obtained by formula (1).

[0074] (1);

[0075] Where I is the received light intensity, S is the focused receiving spot area, d is the aperture diameter, is the distance from the receiving galvanometer to the receiving point, D is the diameter of the photosensitive area of ​​the second photodetector, and α is the aperture (NA) of the optical fiber core.

[0076] From formula (1), it can be seen that the larger the photosensitive area of ​​the detector, the larger the wall area covered, and the more photons captured. Therefore, for non-field of view imaging, the detector with a larger photosensitive surface can receive a larger area on the intermediate wall, thereby collecting more photons. In the design of the receiving optical path, the second photodetector can be selected as a detector with a large photosensitive surface diameter (for example, greater than 50μm). Therefore, combined with the receiving galvanometer, a higher photon collection efficiency can be achieved while maintaining a higher imaging resolution, thereby collecting more photons in a short time.

[0077] According to an embodiment of the present invention, the second detector may collect photons in a fiber-optic coupling manner to make the optical path more flexible and the debugging more convenient.

[0078] According to an embodiment of the present invention, the receiving light path can use an optical lens with a large aperture (for example, larger than 1 inch). By adjusting the light path, the receiving light spot (field of view) can be focused on the wall, so that the receiving light spot (field of view) on the wall is small enough, thereby achieving high-resolution non-field of view imaging.

[0079] According to an embodiment of the present invention, the non-line-of-sight imaging device may further include a bandpass filter, which is disposed between the receiving galvanometer and the receiving lens and is suitable for isolating background noise.

[0080] According to an embodiment of the present invention, the beam splitter splits a small portion of the pulsed laser emitted by the transmitting laser into synchronization light and splits most of the pulsed laser into imaging light. The synchronization light is further attenuated, coupled into the optical fiber, and converted into a synchronization electrical signal by the first photodetector.

[0081] According to an embodiment of the present invention, the processor may include a computer.

[0082] According to an embodiment of the present invention, the non-viewing area image may include a non-viewing area video or a non-viewing area picture.

[0083] According to an embodiment of the present invention, the non-line-of-sight imaging device can realize the emission, reception, and detection of imaging light, and measure the flight time of photons.

[0084] Figure 3 A block diagram of a processor according to an embodiment of the present invention is shown.

[0085] According to an embodiment of the present invention, Figure 3 As shown, the processor 28 includes a reconstruction module 281, a boundary acquisition module 282, a motion estimation module 283, a compensation module 284 and a restoration module 285. The reconstruction module 281 is configured to reconstruct using a non-field of view reconstruction algorithm according to multiple flight times to obtain multiple single-frame images. The boundary acquisition module 282 is configured to cut out the motion images containing images representing moving objects in the multiple single-frame images to obtain segmented images, and to segment each segmented image by a projection segmentation method to obtain the three-dimensional boundary of the motion image. The motion estimation module 283 is configured to use a motion detection algorithm for each segmented image to obtain the motion state of each moving object according to the three-dimensional boundary. The compensation module 284 is configured to perform motion compensation on the multiple segmented images according to the motion state, so that the motion images in the multiple segmented images are aligned and superimposed to obtain the compensated image. The restoration module 285 is configured to restore the motion image to each single-frame image according to the compensated image to obtain the non-field of view image.

[0086] According to an embodiment of the present invention, the reconstruction module uses a non-field of view reconstruction algorithm to reconstruct according to each group of flight times to obtain a single-frame image, wherein each k-th scanning point in the M×N sub-areas is scanned once in turn, and the obtained M×N flight times are a group of flight times.

[0087] According to an embodiment of the present invention, by estimating the motion state of a moving object, the motion images in a plurality of segmented images are aligned and superimposed according to the motion state to obtain a high-resolution motion compensated image, and then the restored motion image is restored to each single-frame image according to the compensated image according to the motion state, thereby reconstructing a high-resolution imaging video in the process of acquiring a dynamic low-resolution image, achieving high-quality, real-time non-field of view imaging of complex scenes, and achieving centimeter-level imaging resolution and an imaging range of room size (for example, an area of ​​1.75m×1.75m).

[0088] According to an embodiment of the present invention, Figure 3 As shown, the boundary acquisition module 282 includes a motion determination unit 2821, a first projection unit 2822, a first segmentation unit 2823, a second projection unit 2824, a second segmentation unit 2825, a third projection unit 2826 and a boundary determination unit 2827. The motion determination unit 2821 is configured to determine a still image and a moving image representing a still object according to a plurality of single-frame images, and eliminate the still image as a background to obtain a segmented image. The first projection unit 2822 is configured to perform a one-dimensional projection on the first frame segmented image along a first direction to obtain a first projection image. The first segmentation unit 2823 is configured to use a position where the projection value is 0 as a segmentation point, and segment the first projection image into a plurality of slices along the segmentation point. The second projection unit 2824 is configured to perform a one-dimensional projection on each object slice along a second direction to obtain a second projection image, wherein the second direction is perpendicular to the first direction, and the object slice is a slice containing elements of a moving object among the plurality of slices. The second segmentation unit 2825 is configured to use the position where the projection value is 0 as the segmentation point to segment each object slice into multiple cylindrical strips. The third projection unit 2826 is configured to perform one-dimensional projection on each object strip along the third direction to obtain a third projection image, wherein the position where the projection value is 0 in the third projection image is a boundary strip in the object strip in the hollow area, wherein the third direction is perpendicular to the first direction and the second direction, the object strip is a cylindrical strip containing elements of the moving object in the multiple cylindrical strips, and the boundary strip is an object strip containing the edge of the moving object in the multiple object strips. The boundary determination unit 2827 is configured to perform one-dimensional projection on the boundary strips along the first direction, the second direction and the third direction respectively, so as to determine the boundary pixel positions of the moving image in the first direction, the second direction and the third direction, thereby determining the three-dimensional boundary.

[0089] According to an embodiment of the present invention, Figure 3 As shown, the motion determination unit 2821 includes a distinguishing subunit and an eliminating subunit. The distinguishing subunit is configured to distinguish between a moving image and a still image by comparing single-frame images of different sub-regions to determine whether the position of the image representing the object in the single-frame image changes over time. The eliminating subunit is configured to eliminate the still image as a background to obtain a segmented image.

[0090] According to an embodiment of the present invention, Figure 3 As shown, the motion estimation module 283 includes a segmentation unit 2831 and a motion state calculation unit 2832. The segmentation unit 2831 is configured to segment the multiple segmented images using a three-dimensional coordinate frame (3D frame) according to the three-dimensional boundary to obtain sub-object images. The motion state calculation unit 2832 is configured to sequentially calculate the motion state of each sub-object image by a cross-correlation function for the segmented image of the current frame and the segmented image of the next frame.

[0091] In such embodiments, the object is assumed to be substantially rigid and not deformable.

[0092] According to an embodiment of the present invention, Figure 3 As shown, the motion state calculation unit 2831 includes a maximum projection unit, a first calculation unit, and a second calculation unit. The maximum projection unit is configured to perform maximum projection on the sub-object image to obtain a front view and a top view. The first calculation unit is configured to calculate the first translation of the moving object in the second direction and the third direction and the first rotation around the first axis from the front view by a cross-correlation function, wherein the first axis is parallel to the first direction. The second calculation unit is configured to calculate the second translation of the moving object in the first direction and the second direction and the second rotation around the third axis from the top view by a cross-correlation function, wherein the third axis is parallel to the third direction. Wherein, the motion state includes at least one of the first translation, the first rotation, the second translation, and the second rotation.

[0093] According to an embodiment of the present invention, the cross-correlation function C can be expressed as formula (2):

[0094] (2);

[0095] Where C represents the cross-correlation function, represents a conjugate complex number, represents element-by-element multiplication, and F represents a two-dimensional discrete Fourier transform matrix. and Indicates the front view or top view of the sub-object image in the current frame and the next frame.

[0096] According to an embodiment of the present invention, considering the limited moving speed of the moving object, the cross-correlation function only performs motion estimation in the area near the position of the moving object in the previous frame, which can avoid confusion of similar objects and reduce the amount of calculation.

[0097] In an illustrative embodiment, a maximum projection is performed on a three-dimensional image of an object to obtain a front view and a top view thereof. From the front view, the translation of the object in the x / y dimension and the rotation around the z axis can be calculated by a cross-correlation function; from the top view, the translation of the object in the x / z dimension and the rotation around the y axis can be calculated by a cross-correlation function.

[0098] According to an embodiment of the present invention, Figure 3 As shown, the compensation module 284 includes a target determination unit, a transformation unit, and a superposition unit. The target determination unit is configured to determine a target area containing a motion image from each segmented image according to a three-dimensional boundary. The transformation unit is configured to use the first frame segmented image as a reference image, and perform inverse transformation on the target areas in the other frame segmented images except the first frame according to the motion state, so that the position and angle of the motion image in the other frame segmented images are consistent with the position and angle of the motion image in the reference image, and obtain multiple single-frame target images. The superposition unit is configured to superimpose multiple single-frame target images to obtain a compensated image with high resolution.

[0099] According to an embodiment of the present invention, Figure 3 As shown, the restoration module 285 includes a motion restoration unit and an image restoration unit. The motion restoration unit is configured to restore the compensated image to the motion state corresponding to the motion image in each segmented image in turn according to the compensated image and the motion state, thereby obtaining a plurality of restored images. The image restoration unit is configured to put the plurality of restored images back into the corresponding plurality of single-frame images, respectively, to obtain a non-viewing area image, thereby achieving high-quality non-viewing area imaging.

[0100] According to an embodiment of the present invention, Figure 3 As shown, the processor 28 further includes a recording module 286, which is configured to record the sequence number of the first receiving moment or the second receiving moment to correspond to the sequence numbers of the multiple sub-areas for data statistics.

[0101] In an illustrative embodiment, the second photodetector is a single photon avalanche diode (SPAD), the photosensitive area of ​​the SPAD has a diameter of 100 μm, and operates in a free-running mode. A multimode optical fiber with a core diameter of 105 μm and NA = 0.22 is used to guide the photons into the SPAD. At the same time, a receiving lens with a focal length of 20 mm and a diameter of 1 inch is used to focus the light into the optical fiber, thereby ensuring a suitable field of view of 2 cm per pixel (at a 4-meter intermediate wall). A bandpass filter is used to eliminate background noise components, so that the non-line-of-sight imaging device is robust to ambient light and noise, and the SPAD background count rate is about 1-3 MHz per second in a bright environment. The total efficiency of the non-line-of-sight imaging device is (22%), including the transmission efficiency of the lens (80%), the transmission efficiency of the reflector (70%), and the transmission efficiency of the SPAD detector (40%). The temporal resolution of the non-line-of-sight imaging device is about 95 ps in total. The influencing factors include the laser emission pulse width of 50 ps, ​​the SPAD detection time jitter of 37 ps, the first photodetector of 50 ps, ​​the time-correlated single photon counter (TCSPC) of 8 ps, and the multimode optical fiber of 0.4 m in length of 50 ps.

[0102] To prevent direct stray light from saturating the SPAD, the field of view of the second photodetector can be kept 20 cm to the left of the laser point during scanning, thereby reducing the direct reflection of the measurement. The transmitting and receiving galvanometers work in a point-to-point scanning mode and scan the grid in a serpentine sequence. The transition time from the previous scanning point to the next scanning point is about 250 μs, and the scanning rate of the non-field of view imaging device can be set to 1000 points / second (1 millisecond per point).

[0103] The non-line-of-sight imaging device completes scanning of the kth scanning point in the M×N sub-areas once every 256 ms and generates a single-frame image. Therefore, the imaging speed of the non-line-of-sight imaging device is 4 frames per second.

[0104] Figure 4 shows an image of a non-viewing space according to an embodiment of the present invention, Figure 5 shows measurement data according to an embodiment of the present invention, Figure 6 The imaging result of the non-viewing area space according to one embodiment of the present invention is shown.

[0105] exist Figure 4 In the non-visual space shown, a turntable with numbers rotates at a certain speed. Figure 5 In the figure, y represents the height of the scanning point from the ground, t represents the flight time of the photon, and the grayscale represents the photon count. Figure 6 The 1st, 22nd, 43rd and 64th non-viewing images are shown, where the image in the circular frame in each non-viewing image is the restored image.

[0106] from Figure 6 It can be seen that by restoring the compensated image to the motion state corresponding to the motion image in each segmented image, a restored image is obtained, and then the restored image is put back into the corresponding single-frame image to obtain a high-quality non-viewing area image.

[0107] Figure 7 A flow chart of a non-line-of-sight imaging method according to an embodiment of the present invention is shown.

[0108] As another aspect of an embodiment of the present invention, a non-line-of-sight imaging method using any of the above-mentioned non-line-of-sight imaging devices is provided, such as Figure 7 As shown, the non-line-of-sight imaging method includes operations S710 to S760.

[0109] In operation S710 , a plurality of flight times of photons of imaging light of M×N sub-regions are sequentially acquired.

[0110] In operation S720, a non-viewing area reconstruction algorithm is used to reconstruct the plurality of flight times to obtain a plurality of single-frame images.

[0111] In operation S730, a moving image including an image representing a moving object among the plurality of single-frame images is cut out to obtain segmented images, and each segmented image is segmented by a projection segmentation method to obtain a three-dimensional boundary of the moving image.

[0112] In operation S740, a motion detection algorithm is applied to each segmented image according to the three-dimensional boundary to obtain a motion state of each moving object.

[0113] In operation S750, motion compensation is performed on the plurality of segmented images according to the motion state, and the motion images in the plurality of segmented images are aligned and superimposed to obtain a compensated image after the motion compensation is performed on the motion image.

[0114] In operation S760, the motion image is restored to each single frame image according to the compensation image to obtain a non-viewing area image.

[0115] The non-line-of-sight imaging method provided by the present invention performs motion compensation on multiple segmented images, aligns and superimposes the motion images in the multiple segmented images, obtains a compensated image after motion compensation of the motion image, restores the motion image to each single frame image according to the compensated image, and images the non-line-of-sight space, which has the advantages of fast calculation speed and meeting real-time processing requirements. The non-line-of-sight imaging method can be integrated into the software to display the non-line-of-sight image in real time while collecting data.

[0116] According to an embodiment of the present invention, each sub-region includes a scanning point matrix, the scanning point matrix includes K scanning points, the k-1th scanning point and the kth scanning point are located at adjacent positions in the scanning point matrix, k is a positive integer less than or equal to K, and K is a positive integer. The transmitting galvanometer and the receiving galvanometer sequentially scan the kth scanning points in the 1st to the M×N sub-regions of the M×N sub-regions in accordance with the regional scanning order to obtain K single-frame images, wherein the regional scanning order is set based on the position of each sub-region in the M×N sub-regions.

[0117] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A non-line-of-sight imaging device, It is characterized in that The non-viewing space where the object is located is separated from the collection space where the non-viewing imaging device is located by an obstacle, and an intermediate wall spaced apart from the obstacle is arranged outside the non-viewing space and the collection space, and the non-viewing imaging device includes: A laser, suitable for emitting pulsed laser light; A beam splitter, which splits the pulse laser into synchronization light and imaging light; a first photodetector, adapted to convert the synchronization light into a synchronization electrical signal; An emitting galvanometer, adapted to sequentially change the emitting angle, sequentially emitting the imaging light to a plurality of sub-regions on the intermediate wall, so that the imaging light is diffusely reflected into the non-viewing space through each of the sub-regions, wherein the surface of the intermediate wall facing the non-viewing space and the collection space is divided into M×N sub-regions, wherein M and N are both positive integers; A receiving galvanometer, adapted to sequentially change a receiving angle, and sequentially receive a plurality of received light signals diffusely reflected by the object and each of the sub-areas; A second photoelectric detector is adapted to sequentially convert the plurality of received optical signals into a plurality of received electrical signals; A time-to-digital converter, adapted to sequentially record a first receiving time of the synchronous electrical signal and a second receiving time of receiving the received electrical signal; A processor is adapted to obtain a plurality of flight times of photons of the imaging light diffusely reflected from the M×N sub-areas according to the first receiving time and the second receiving time, respectively, and obtain a plurality of single-frame images according to the plurality of flight times, so as to obtain a non-viewing area image of the non-viewing area space; Wherein, the processor comprises: A reconstruction module is configured to reconstruct the plurality of flight times using a non-viewing area reconstruction algorithm to obtain a plurality of single-frame images; A boundary acquisition module is configured to perform matting on the moving images including the images representing the moving objects in the plurality of single-frame images to obtain segmented images, and segment each of the segmented images by a projection segmentation method to obtain a three-dimensional boundary of the moving image; A motion estimation module, configured to use a motion detection algorithm on each of the segmented images according to the three-dimensional boundary to obtain a motion state of each of the moving objects; a compensation module configured to perform motion compensation on the plurality of segmented images according to the motion state, so as to align and superimpose the motion images in the plurality of segmented images to obtain a compensated image after the motion compensation is performed on the motion image; The restoration module is configured to restore the motion image to each of the single-frame images according to the compensation image to obtain the non-viewing area image.

2. The non-line-of-sight imaging device according to claim 1, It is characterized in that Each of the sub-areas includes a scanning point matrix, the scanning point matrix includes K scanning points, the k-1th scanning point and the kth scanning point are located at adjacent positions in the scanning point matrix, k is a positive integer less than or equal to K, and K is a positive integer; The transmitting galvanometer and the receiving galvanometer scan the kth scanning points in the 1st to the M×Nth sub-regions in sequence according to the area scanning order to obtain K single-frame images, wherein the area scanning order is set based on the position of each of the M×N sub-regions.

3. The non-line-of-sight imaging device according to claim 1, It is characterized in that The boundary acquisition module comprises: a motion determination unit configured to determine a still image including still objects and the motion image according to the plurality of single-frame images, and eliminate the still image as a background to obtain a segmented image; A first projection unit is configured to perform one-dimensional projection on the first frame segmented image along a first direction to obtain a first projection image; A first segmentation unit is configured to use a position where the projection value is 0 as a segmentation point, and segment the first projection image into a plurality of slices along the segmentation point; A second projection unit is configured to perform one-dimensional projection on each object slice along a second direction to obtain a second projection image, wherein the second direction is perpendicular to the first direction, and the object slice is a slice containing an element of a moving object in the plurality of slices; A second segmentation unit is configured to segment each of the object slices into a plurality of cylindrical strips using a position where the projection value is 0 as a segmentation point; A third projection unit is configured to perform one-dimensional projection on each object strip along a third direction to obtain a third projection image, wherein a position with a projection value of 0 in the third projection image is a boundary strip in the object strip in the hollow area, wherein the third direction is perpendicular to the first direction and the second direction, the object strip is a columnar strip containing elements of a moving object in the plurality of columnar strips, and the boundary strip is an object strip containing edges of a moving object in the plurality of object strips; The boundary determination unit is configured to perform one-dimensional projection on the boundary strip along the first direction, the second direction and the third direction respectively to determine the boundary pixel positions of the motion image in the first direction, the second direction and the third direction, thereby determining the three-dimensional boundary.

4. The non-line-of-sight imaging device according to claim 1, It is characterized in that The motion estimation module comprises: a segmentation unit configured to segment the plurality of segmented images respectively using a three-dimensional coordinate frame according to the three-dimensional boundary to obtain sub-object images; The motion state calculation unit is configured to calculate the motion state of each sub-object image through a cross-correlation function for the segmented image of the current frame and the segmented image of the next frame in sequence.

5. The non-line-of-sight imaging device according to claim 4, It is characterized in that The motion state calculation unit comprises: a maximum projection unit, configured to perform maximum projection on the sub-object image to obtain a front view and a top view; A first calculation unit is configured to calculate, from the front view, a first translation of the moving object in a second direction and a third direction and a first rotation about a first axis by a cross-correlation function, wherein the first axis is parallel to the first direction; a second calculation unit configured to calculate, from the top view, a second translation of the moving object in the first direction and the second direction and a second rotation about a third axis by a cross-correlation function, wherein the third axis is parallel to the third direction; The motion state includes at least one of the first translation, the first rotation, the second translation and the second rotation.

6. The non-line-of-sight imaging device according to claim 1, It is characterized in that The compensation module comprises: a target determination unit configured to determine a target region including the motion image from each of the segmented images according to the three-dimensional boundary; a transforming unit configured to use the first frame segmented image as a reference image, and to perform inverse transform on the target area in the other frame segmented images except the first frame according to the motion state, so that the position and angle of the motion image in the other frame segmented images are consistent with the position and angle of the motion image in the reference image, thereby obtaining a plurality of single-frame target images; The superposition unit is configured to superimpose a plurality of the single-frame target images to obtain the compensated image.

7. The non-line-of-sight imaging device according to claim 1, It is characterized in that The recovery module comprises: a motion restoration unit configured to restore the compensated image to a motion state corresponding to each of the segmented images in sequence according to the compensated image and the motion state, so as to obtain a restored image; The image restoration unit is configured to put the restored image back into the corresponding single-frame image to obtain the non-viewing area image.

8. The non-line-of-sight imaging device according to any one of claims 1 to 7, It is characterized in that The processor further comprises: The recording module is configured to record the sequence number of the first receiving moment or the second receiving moment to correspond to the sequence numbers of the plurality of sub-areas.

9. A non-line-of-sight imaging method using the non-line-of-sight imaging device according to any one of claims 1 to 8, It is characterized in that include: sequentially acquiring multiple flight times of photons of imaging light in M×N sub-regions; Reconstructing the images using a non-viewing area reconstruction algorithm according to the plurality of flight times to obtain a plurality of single-frame images; Cutting out the moving images including the images representing the moving objects in the plurality of single-frame images to obtain segmented images, and segmenting each of the segmented images by a projection segmentation method to obtain the three-dimensional boundary of the moving image; Using a motion detection algorithm on each of the segmented images according to the three-dimensional boundary to obtain the motion state of each of the moving objects; According to the motion state, motion compensation is performed on the plurality of segmented images, so that the motion images in the plurality of segmented images are aligned and superimposed to obtain a compensated image after motion compensation is performed on the motion image; The motion image is restored to each of the single-frame images according to the compensation image to obtain a non-viewing area image.

Citation Information

Patent Citations

  • Scanning photon counting non-visual-field three-dimensional imaging device and method

    CN106526612A

  • Non-vision-field imaging device and method

    CN111880194A

  • Non-vision-field dynamic imaging method and device based on sub-aperture gating type camera

    CN112040116A

  • Three-dimensional information acquisition and reconstruction method based on single photon detection

    CN115615349A

  • Image reconstruction method and device, detection system and storage medium

    CN116883264A