Rapid non-vision field imaging system and method of area array time-of-flight camera

Abstract

The invention relates to a fast non-vision field imaging system and method of an area array time-of-flight camera. The fast non-vision field imaging system comprises a detection module, a data processing module and a data analysis module, the detection module is used for collecting a target image of a non-vision scene; the data processing module is used for performing depth information processing on the target image to obtain a depth image; and the data analysis module is used for performing reconstruction processing on the depth image and inputting the reconstructed image into a preset neural network for automatic identification and early warning. According to the method, the limitation that a traditional imaging system is limited by a WYSIWYG imaging scene is broken through, and non-vision-field target imaging recognition can be achieved by bypassing obstacles under the condition that the view field is limited; and the large-scale popularization of the non-vision field imaging technology can be effectively promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of non-line-of-sight imaging, and in particular to a fast non-line-of-sight imaging system and method for an area array time-of-flight camera. Background Art

[0002] Non-line-of-sight imaging, also known as corner imaging, is a technology that images hidden targets or scenes outside the line of sight. As the application conditions of imaging systems become increasingly complex and imaging scenes become increasingly diversified, traditional imaging technology is limited by the influence of the field of view and has been difficult to meet the needs of today's society. Non-line-of-sight imaging technology breaks through the imaging limitations of traditional optical systems and can play a huge role in scenarios with limited field of view, such as autonomous driving, post-disaster rescue, and battlefield environment exploration. It has broad application value and application potential.

[0003] Traditional non-line-of-sight imaging technologies are mostly based on highly sensitive photoelectric sensors to perform non-line-of-sight imaging of a point in the target area. When it is necessary to obtain three-dimensional information of the target area in a non-line-of-sight scenario, long-term scanning imaging and data fusion are required. The imaging speed is slow and the imaging system is expensive. It cannot be used as an imaging technology that can be promoted. Whether it is the imaging distance, imaging range, or imaging speed, it cannot meet the actual application conditions. It is still in the laboratory stage and cannot be applied on a large scale. Summary of the invention

[0004] The purpose of the present invention is to propose a fast non-line-of-sight imaging system and method for an area array time-of-flight camera to solve the problems existing in the above-mentioned prior art. The system aims to expand the distance, range and imaging speed of non-line-of-sight imaging by using area array time-of-flight cameras and technical means such as three-dimensional imaging and machine learning, so as to achieve real-time observation of the situation in the non-line-of-sight area when the field of view is limited, obtain the image of the non-line-of-sight target, obtain the light signal carrying the non-line-of-sight target information after triple scattering through an intermediate surface and reconstruct it, provide information such as the distance, three-dimensional structure and motion state of the non-line-of-sight target, thereby providing a feasible solution for the practical application of the technology.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A fast non-line-of-sight imaging system of an array time-of-flight camera, comprising: a detection module, a data processing module and a data analysis module;

[0007] The detection module is used to collect target images of non-viewing scenes;

[0008] The data processing module is used to perform depth information processing on the target image to obtain a depth image;

[0009] The data analysis module is used to reconstruct the depth image and input the reconstructed image into a preset neural network for automatic recognition and early warning.

[0010] Optionally, the detection module includes: an array time-of-flight camera, a light source device, a signal control unit and a preprocessing unit;

[0011] The area array time-of-flight camera is installed in the detection area of ​​the non-line-of-sight imaging scene, and is used to detect photons carrying target information scattered back by the intermediate surface after passing through the preset non-line-of-sight imaging light path, and obtain a target image of the non-line-of-sight scene;

[0012] The light source device is used to emit a modulated light signal for depth detection; wherein the light signal emitted by the light source device can illuminate the non-viewing area and the target therein after passing through the intermediate surface, and part of the light signal is scattered back to the intermediate surface through secondary scattering and scattered to the observation area through tertiary scattering;

[0013] The signal control unit is used to set the modulation light frequency of the light source signal, the exposure time of the area array time-of-flight camera, and the imaging range;

[0014] The pre-processing unit is used to perform preliminary processing on the acquired target image of the non-viewing scene, including: denoising and color correction.

[0015] Optionally, the data processing module includes: a signal transmission unit, a depth map conversion unit, and a transmission control unit;

[0016] The signal transmission unit is used for signal transmission between the detection module and the data analysis module;

[0017] The depth image conversion unit is used to modulate the target image using a four-step phase shift method to obtain a depth image, and filter and denoise the depth image according to a preset photon number threshold;

[0018] The transmission control unit is used to transmit the filtered and denoised depth image to the data analysis module at a regular interval.

[0019] Optionally, the data analysis module includes: an image reconstruction unit, an image recognition unit, an analysis unit, and a result display unit;

[0020] The image reconstruction unit is used to perform image denoising, downsampling, region division and extraction, and multiple image joint registration and enhancement processing on the received depth image to obtain the target image in the non-viewing scene;

[0021] The image recognition unit is used to automatically recognize the reconstructed target image using the trained YOLO neural network, and to alarm and mark the recognized target;

[0022] The analysis unit is used to analyze the imaging results of the non-viewing scene;

[0023] The result display unit is used to display the non-viewing area imaging result and the original imaging data to the user.

[0024] Optionally, the area array time-of-flight camera and the light source device are respectively installed on six-degree-of-freedom platforms arranged at different preset positions in the observation area.

[0025] A fast non-line-of-sight imaging method for an area array time-of-flight camera, the method comprising:

[0026] Arrange the area array time-of-flight camera and light source equipment based on preset positions;

[0027] Start the area array time-of-flight camera and light source equipment to collect target image information scattered back by non-viewing targets in non-viewing scenes;

[0028] Using a preset data processing module, depth information processing is performed on the target image information to obtain a depth image;

[0029] The depth image is reconstructed using a preset data analysis module, and the reconstructed image is input into a preset neural network for automatic recognition and early warning.

[0030] Optionally, arranging the area array time-of-flight camera and the light source device based on the preset position includes:

[0031] The area array time-of-flight camera and the light source device are respectively installed on six-degree-of-freedom platforms arranged at different preset positions in the observation area;

[0032] The modulation light frequency of the light source signal of the light source device, the exposure time of the area array flight time camera, and the imaging range are set; and the light signal emitted by the light source device can illuminate the non-viewing area and the target therein after passing through the intermediate surface, and part of the light signal is scattered back to the intermediate surface through secondary scattering and scattered tertiary to the observation area.

[0033] Optionally, before performing depth information processing on the target image information, the process further includes:

[0034] The acquired target image information of the non-viewing scene is preliminarily processed, including: denoising and color correction.

[0035] Optionally, performing depth information processing on the target image information includes:

[0036] The target image information is modulated by a four-step phase shift method to obtain a depth image, and the depth image is filtered and denoised according to a preset photon number threshold.

[0037] Optionally, reconstructing the depth image includes:

[0038] The depth image is subjected to image denoising, downsampling, region division and extraction, and multiple images are jointly registered and enhanced to reconstruct target image information in the non-viewing scene;

[0039] The trained YOLO neural network is used to automatically identify the reconstructed target image information, and the identified targets are alarmed and marked;

[0040] The imaging results of the non-line-of-sight scene are analyzed, and the non-line-of-sight imaging results and the original imaging data are presented to the user.

[0041] The beneficial effects of the present invention are:

[0042] On the one hand, compared with traditional imaging systems, the system of the present invention breaks through the imaging scene limitation of traditional imaging systems that is limited by "what you see is what you get", and can realize imaging and recognition of non-line-of-sight targets by bypassing obstacles when the field of view is limited; on the other hand, compared with the existing non-line-of-sight imaging system that relies on single-photon detection point scanning imaging, the imaging of the system of the present invention greatly improves the imaging speed by using an area array camera, and overcomes the noise interference caused by area array imaging. At the same time, the price of the time-of-flight camera required for imaging is much lower than that of the single-photon detector and streak tube camera used in the non-line-of-sight imaging of the single-photon detection system, and the complexity and price of the system are greatly improved. The imaging speed of one picture per second can basically meet the application requirements of most scenarios. Therefore, based on the above advantages, the present invention can effectively promote the large-scale promotion of non-line-of-sight imaging technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 It is a schematic structural diagram of a fast non-line-of-sight imaging system of an area array time-of-flight camera according to an embodiment of the present invention;

[0045] Figure 2 The figure is a schematic diagram of the entire process of the area array time-of-flight camera according to an embodiment of the present invention from acquiring non-viewing area target information to finally transmitting the data to a terminal. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1-Figure 2 As shown, this embodiment proposes a fast non-line-of-sight imaging system of a plane array time-of-flight camera, including: a detection module, a data processing module and a data analysis module;

[0049] A detection module, used to collect target images of non-viewing scenes;

[0050] A data processing module is used to process the depth information of the target image and obtain a depth image;

[0051] The data analysis module is used to reconstruct the depth image and input the reconstructed image into the preset neural network for automatic recognition and early warning.

[0052] The system of this embodiment includes a time-of-flight camera consisting of a laser source and a detector with a horizontal field of view of 59.3° and a vertical field of view of 74.4°, and a data processing software supporting it. The detector is responsible for the photons that are emitted by the light source and carry the target information after multiple scattering by the intermediate surface and the non-viewing target. The data processing software is responsible for providing a processed non-viewing target picture every 1 second, displaying and saving the image.

[0053] Furthermore, the detection module includes: an array time-of-flight camera, a light source device, a signal control unit and a preprocessing unit;

[0054] The area array time-of-flight camera is installed in the detection area of ​​the non-line-of-sight imaging scene to detect photons carrying target information scattered back by the intermediate surface after passing through the preset non-line-of-sight imaging optical path, and obtain the target image of the non-line-of-sight scene;

[0055] A light source device, used to emit a modulated light signal for depth detection; wherein the light signal emitted by the light source device can illuminate the non-viewing area and the target therein after passing through the intermediate surface, and part of the light signal is scattered back to the intermediate surface through secondary scattering and scattered to the observation area through tertiary scattering;

[0056] A signal control unit, used to set the modulation light frequency of the light source signal, the exposure time of the area array time-of-flight camera, and the imaging range;

[0057] The preprocessing unit is used to perform preliminary processing on the acquired target image of the non-viewing scene, including: denoising and color correction.

[0058] Specifically, in this embodiment, the photon detector installed in the detection area of ​​the non-line-of-sight imaging scene is a face diagnosis detector equipped with a 320*240 pixel CMOS array, which is used to detect photons carrying target information that are scattered three times back through the intermediate surface after the photons pass through the set non-line-of-sight imaging optical path, and obtain the image of the non-line-of-sight scene. The 850nm laser source installed in the detection area of ​​the non-line-of-sight imaging scene is a light source array composed of four diodes, and the diodes are controlled by the signal on the chip to emit the required modulated light signal.

[0059] The design of the area array time-of-flight camera enables it to simultaneously obtain all photon signals and extremely deep information in a large field of view, thereby avoiding the slow imaging speed defect brought by the point scanning non-line-of-sight imaging technology. At the same time, the combination of array light source illumination and area array detection method reduces the angle requirements when designing the optical path, and the optical path can be adjusted more conveniently to obtain non-line-of-sight imaging results.

[0060] Compared with the combination of a high-collimation laser source and a high-precision single-photon detector / streak tube camera, the system cost of a time-of-flight camera is greatly reduced, and the imaging system is also smaller and simpler.

[0061] Among them, the light source hardware: used to emit modulated light signals for depth detection, including light-emitting diodes and LED chips. Detector hardware: used to receive photon signals after multiple scattering, including 320*240CMOS array, 2.8-12mm manual zoom lens. Signal control unit: responsible for setting the modulated light frequency of the light source signal, the exposure time of the detector, the imaging range, and controlling the synchronization of the demodulated signal transmitted to the detector and the modulated light signal transmitted to the light source, including a timing generator and an analog-to-digital converter. Image preprocessing: preliminary processing of the captured original image, such as denoising, color correction, etc.

[0062] Further, the data processing module includes: a signal transmission unit, a depth map conversion unit, and a transmission control unit;

[0063] A signal transmission unit, used for signal transmission between the detection module and the data analysis module;

[0064] A depth image conversion unit, used to modulate the target image using a four-step phase shift method to obtain a depth image, and filter and denoise the depth image according to a preset photon number threshold;

[0065] The transmission control unit is used to transmit the filtered and denoised depth image to the data analysis module at a regular interval.

[0066] Specifically, in this embodiment, the data processing module is a signal control and transmission module. The digital-analog signal flight time sensing function on the module is combined with an optimized analog-to-digital converter (ADC) and a universal programmable timing generator (TG). The high-frequency modulated signal can be transmitted as a high and low level digital signal through the FC gray cable to the light source to control the emission of a modulated light signal, and receive the signal received by the detector. After the depth information is obtained through the time-of-flight principle processing, a depth image is obtained. The depth image data with a resolution of one quarter of a video graphics array (QVGA 320*240) can be provided to the data analysis terminal at a frame rate of up to 150 frames / second (600 readouts / second).

[0067] Among them, the FTP system is responsible for establishing a stable signal transmission channel between the light source-detector module and the terminal. The FPGA chip is responsible for processing the received light signal, using the four-step phase shift method to demodulate the light signal received by the detector to obtain the depth image, and performing simple filtering and denoising according to the set photon number threshold. Data transmission control ensures that the image data is transmitted on time every 15 minutes.

[0068] Furthermore, the data analysis module includes: an image reconstruction unit, an image recognition unit, an analysis unit, and a result display unit;

[0069] An image reconstruction unit is used to perform image denoising, downsampling, region division and extraction, and joint registration and enhancement processing of multiple images on the received depth image to obtain a target image in a non-viewing scene;

[0070] The image recognition unit is used to automatically recognize the reconstructed target image using the trained YOLO neural network, and to alarm and mark the recognized target;

[0071] An analysis unit, used for analyzing the imaging results of the non-viewing scene;

[0072] The result display unit is used to display the non-viewing area imaging results and the original imaging data to the user.

[0073] Specifically, in this embodiment, the data analysis module uses terminal analysis software to receive and analyze the transmitted depth image containing non-line-of-sight scene information, combines the optical path feature constraint analysis of the non-line-of-sight imaging scene, and filters out noise to obtain clear non-line-of-sight imaging results.

[0074] Combined with the camera's own high frame rate data output, the terminal analysis software can receive and analyze depth images. After filtering, multi-point cloud image enhancement, point cloud registration image enhancement, etc., it can achieve a non-line-of-sight image imaging speed of 320*240 pixels and >1FPS within a 5m imaging distance, improve the real-time performance of non-line-of-sight imaging, and provide support for its transition from laboratory to large-scale application.

[0075] Among them, data reception: responsible for receiving image data transmitted from the FTP system. Data processing: responsible for continuous processing of multiple received depth maps, denoising, filtering, key area extraction, registration and point cloud fusion. Image recognition: arrange the corresponding simulation scene according to the system usage scenario, collect image processing and conduct training before image recognition. The content of the training data should fully cover the target conditions in the usage scenario. After training, the network is responsible for automatically recognizing the processed images.

[0076] The analysis unit analyzes the imaging results of non-visual scenes in combination with actual application requirements, such as whether there are dangers in the non-visual scenes, the location of the target to be rescued, whether there are hidden dangers in the facilities in the scenes, and whether there are targets that need attention in the unrecognized images. For scenes with long-term application requirements, the movement patterns of the targets can be analyzed in combination with the long-term imaging results.

[0077] The result display unit shows the non-viewing area imaging results and the original imaging data to the user. The data control entrances such as imaging speed, platform adjustment of light source and detector, and parameters of the algorithm during image reconstruction are also integrated into the software interface, and users can make corresponding adjustments and optimizations based on feedback.

[0078] Furthermore, the area array time-of-flight camera and the light source device are respectively installed on six-degree-of-freedom platforms arranged at different preset positions in the observation area.

[0079] The following is a detailed description of the hardware and software improvement method of the system of the technical solution of this embodiment:

[0080] Non-line-of-sight depth image processing technology: Using computer vision algorithms, the original point cloud data acquired by the system is initially filtered, key area extracted and down-sampled to reduce the noise signal interference introduced by multiple scattering, and then the registration and point cloud fusion methods are used to combine multiple point cloud images to enhance the target information in the image, highlight the features, and perform secondary corrections on the image, so as to achieve the purpose of quickly obtaining non-line-of-sight images using an array time-of-flight camera.

[0081] Flexible and accurate optical path adjustment: The light source and detector of the time-of-flight camera are separated, and the platform carries the light source and detector hardware respectively. FC gray cables are used to extend the distance and transmit data between the two. The platform is used for fixing and can perform six-degree-of-freedom adjustment in a small range to achieve flexible adjustment of the light source and detector respectively, avoid interference from primary scattering, and establish a non-field of view imaging optical path with good imaging effect.

[0082] Automatic image recognition: Based on the deep learning network, the imaging scene is pre-designed, and image acquisition, feature extraction, image classification and network training are performed in advance for the targets that need to be imaged, and then used for image recognition.

[0083] Prior knowledge optimizes the system: Combined with historical data, some rules in actual usage scenarios can be obtained as prior data input into the system, such as the target distance range of imaging, the angle of the light source during imaging, etc., which can more accurately extract target information, reduce the time required for image processing, increase image quality, and obtain a better imaging system.

[0084] Interactive analysis interface: Design a user-friendly interactive analysis interface to retain and compare the original data and reconstructed images. Experts can find more information from the original data, and can also find out the shortcomings of the system and improve them by comparing the reconstructed images with the actual scenes. In addition, they can further combine the image data for scene analysis.

[0085] Feedback mechanism: Due to the diversity of information in actual application scenarios, unrecognizable images will be stored in a designated database. Users can optimize the neural network based on the database, such as adding recognition target categories, increasing training data and reducing false alarm rates.

[0086] Establish a cloud platform: Training set data for specific targets can be uploaded to the established cloud platform, such as human posture in non-visual scenes, the appearance of unmanned vehicles, etc., to facilitate users with similar application needs to directly use it or optimize their networks.

[0087] Through the above module improvements, fast, high-quality image restoration and high-precision image recognition of the fast non-line-of-sight imaging system of the area array time-of-flight camera can be achieved.

[0088] This embodiment also proposes a fast non-line-of-sight imaging method for an area array time-of-flight camera, including:

[0089] Arrange the area array time-of-flight camera and light source equipment based on preset positions;

[0090] Start the area array time-of-flight camera and light source equipment to collect target image information scattered back by non-viewing targets in non-viewing scenes;

[0091] Using a preset data processing module, depth information processing is performed on the target image information to obtain a depth image;

[0092] The preset data analysis module is used to reconstruct the depth image, and the reconstructed image is input into the preset neural network for automatic recognition and early warning.

[0093] Further, arranging the area array time-of-flight camera and the light source device based on the preset position includes:

[0094] The area array time-of-flight camera and the light source device are respectively installed on six-degree-of-freedom platforms arranged at different preset positions in the observation area;

[0095] The modulation light frequency of the light source signal of the light source device, the exposure time of the area array flight time camera, and the imaging range are set; and the light signal emitted by the light source device can illuminate the non-viewing area and the target therein after passing through the intermediate surface, and part of the light signal is scattered back to the intermediate surface through secondary scattering and scattered tertiary to the observation area.

[0096] Specifically, in this embodiment, the arrangement of the area array time-of-flight camera and the light source device includes:

[0097] In the non-line-of-sight imaging scenario, a six-degree-of-freedom platform is arranged at a suitable position in the observation area and the light source of the time-of-flight camera is fixed on it. The angle of the light source is adjusted appropriately so that the light signal emitted by the light source can illuminate the non-line-of-sight area and the target therein after passing through the intermediate surface, and part of the light signal can be scattered back to the intermediate surface through secondary scattering and scattered three times to the observation area, ensuring that the light signal carrying the non-line-of-sight target information can be found in the observation area. The observation range of the detector should include the area in the intermediate surface where the light signal can be scattered back to receive the light signal carrying the non-line-of-sight target information.

[0098] After the position of the light source is determined, a second six-degree-of-freedom platform is installed at a suitable location in the observation area and a time-of-flight camera is installed on it. The angle and height of the camera are adjusted through the six-degree-of-freedom platform to ensure that the camera's observation range covers the entire intermediary surface to ensure that all light signals scattered back from the intermediary surface are received.

[0099] A data transmission module is installed near the camera, including an FPGA chip and an FTP system. The FPGA chip is configured to control the modulation frequency and optical power of the light source, process the optical signal received by the detector to obtain a depth image and perform simple filtering, and the processed image is transmitted to the designated terminal through the FTP system.

[0100] Special analysis software is installed on the terminal device to receive and analyze the image data transmitted by the camera, restore the non-viewing target image from the original image, and perform automatic recognition and early warning in combination with the trained neural network.

[0101] Start the area array time-of-flight camera and light source equipment to collect the target image information scattered back by the non-viewing target in the non-viewing scene, including:

[0102] Start the light source and detector, and start capturing the image information scattered back by the non-viewing target in the non-viewing scene. The light source emits the corresponding modulated light signal according to the preset power and signal, and the camera detector will continuously collect data according to the preset shooting parameters.

[0103] The optical signal received by the detector is pre-processed by the FPGA chip, including signal demodulation to obtain depth information, denoising and filtering of the depth map, etc., to improve the subsequent processing speed.

[0104] The FPGA chip processes the depth image at a speed of 30 frames per second and performs simple preprocessing, and transmits it to the designated terminal through the FTP system to ensure the real-time data transmission.

[0105] Furthermore, before performing depth information processing on the target image information, the following steps are also included:

[0106] The acquired target image information of the non-viewing scene is preliminarily processed, including: denoising and color correction.

[0107] Furthermore, performing depth information processing on the target image information includes:

[0108] A four-step phase shift method is used to modulate the target image information to obtain the depth image, and the depth image is filtered and denoised according to a preset photon number threshold.

[0109] Specifically, in this embodiment, the terminal analysis software receives image data transmitted from the FTP system, and uses the automated program in the terminal software to perform processes such as image denoising, downsampling, region division and extraction, and joint registration and enhancement of multiple images to obtain images of targets in non-field of view scenes. The entire process of image acquisition, preprocessing, and reconstruction is automated, and the reconstruction speed reaches 1FPS.

[0110] Furthermore, the reconstruction process of the depth image includes:

[0111] For depth images, image denoising, downsampling, region division and extraction, and multiple images joint registration and enhancement processing are performed to reconstruct target image information in non-viewing scenes;

[0112] The trained YOLO neural network is used to automatically identify the reconstructed target image information, and the identified targets are alarmed and marked;

[0113] The imaging results of the non-line-of-sight scene are analyzed, and the non-line-of-sight imaging results and the original imaging data are presented to the user.

[0114] Specifically, in this embodiment, the trained yolo neural network is used to automatically identify the received images, and the identified targets are alarmed and marked to obtain the target images that the user needs to monitor. The unrecognized ones are stored in the unrecognized database for subsequent analysis, and the system and network are optimized based on the results. The training set used for identification and the data set added during subsequent optimization can be uploaded and downloaded on the cloud platform to optimize the neural network.

[0115] Combined with actual application needs, the imaging results of non-line-of-sight scenes are analyzed, such as whether there are dangers in the non-line-of-sight scenes, the location of the target to be rescued, whether there are hidden dangers in the facilities in the scene, and whether there are targets that need attention in the unrecognized images. For scenes with long-term application needs, the movement patterns of the targets can be analyzed in combination with long-term imaging results.

[0116] The non-viewing area imaging results and the original imaging data are presented to the user. The data control entrances such as imaging speed, platform adjustment of light source and detector, and parameters of the algorithm during image reconstruction are also integrated into the software interface, and the user can make corresponding adjustments and optimizations based on the feedback.

[0117] More specifically, the maintenance and management method of the system in this embodiment includes:

[0118] Regular inspection and maintenance: Regularly inspect and maintain the light source, detector, six-degree-of-freedom platform, data transmission module and terminal analysis software of the array time-of-flight camera to ensure its normal operation.

[0119] Data backup and recovery: Regularly back up the received and reconstructed images to prevent data loss. At the same time, establish a data recovery mechanism to ensure timely recovery when data is lost or damaged.

[0120] System upgrade and optimization: According to the changes in user requirements for imaging speed and distance, the imaging system's hardware and software are optimized. For different application scenarios, the automatic image recognition neural network is optimized in combination with the data set uploaded by the user on the cloud. This includes improving image reconstruction and recognition algorithms, optimizing data transmission modules, etc., to improve system performance and accuracy.

[0121] The application examples of this embodiment are:

[0122] In a certain rescue mission, there were hostages and robbers in the building where the hostages were held, but the situation was unclear and it was easy to be discovered using traditional detectors. Therefore, a non-line-of-sight imaging system was set up outside the house where the hostages were held, and the ground was used as the intermediate surface of the non-line-of-sight imaging system. The system used 850nm infrared light to indirectly observe the situation inside the house through the intermediate surface, which was not easy to be discovered.

[0123] After adjusting the angle, the signal emitted by the light source can illuminate all areas in the house through the tile intermediate surface, and the detector can also receive the scattered light signal. The signal can be processed by the FPGA chip to obtain its depth image. After the image is transmitted to the terminal through the FTP system, after the preset parameters, such as imaging speed, imaging range (known house size), reconstruction speed, etc., the situation inside the house can be reconstructed. The reconstructed image can observe the entire situation and the number of people in the house, and then the personnel can be identified through the automatic identification network to obtain a more accurate number and distribution of personnel. The automatic identification of weapons and equipment can also distinguish hostages and robbers. Then, by observing the movement of targets in the house for a period of time, excluding possible misidentified targets such as puppet models and coats hanging on hangers, the number, distribution, weapons and equipment of hostages and robbers in the house, obstacles in the house, etc. can be obtained. After reporting it to the commander, the rescue mission and plan can be better arranged.

[0124] Through the above specific implementation, the fast non-line-of-sight imaging system based on the area array time-of-flight camera can effectively realize long-distance and large-scale fast non-line-of-sight imaging. The system reduces the price and complexity of the non-line-of-sight system by taking advantage of the time-of-flight camera, and improves the image quality and ensures the imaging speed by reconstructing and optimizing the image in combination with the subsequent algorithm. It uses neural networks to realize automatic image recognition, providing strong support for the large-scale application of non-line-of-sight imaging technology, helping to obtain information in real time in scenes with limited field of view, and providing feasible technical solutions for observation needs in scenes with limited field of view such as automatic driving of cars, post-disaster rescue, and battlefield environment exploration.

[0125] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A fast non-line-of-sight imaging system for an area array time-of-flight camera, characterized in that: include: Detection module, data processing module and data analysis module; The detection module is used to collect target images of non-viewing scenes; The data processing module is used to perform depth information processing on the target image to obtain a depth image; The data analysis module is used to reconstruct the depth image and input the reconstructed image into a preset neural network for automatic recognition and early warning.

2. The fast non-line-of-sight imaging system of the area array time-of-flight camera according to claim 1, characterized in that: The detection module includes: an array time-of-flight camera, a light source device, a signal control unit and a preprocessing unit; The area array time-of-flight camera is installed in the detection area of ​​the non-line-of-sight imaging scene, and is used to detect photons carrying target information scattered back by the intermediate surface after passing through the preset non-line-of-sight imaging light path, and obtain a target image of the non-line-of-sight scene; The light source device is used to emit a modulated light signal for depth detection; wherein the light signal emitted by the light source device can illuminate the non-viewing area and the target therein after passing through the intermediate surface, and part of the light signal is scattered back to the intermediate surface through secondary scattering and scattered to the observation area through tertiary scattering; The signal control unit is used to set the modulation light frequency of the light source signal, the exposure time of the area array time-of-flight camera, and the imaging range; The preprocessing unit is used to perform preliminary processing on the acquired target image of the non-viewing scene.

3. The fast non-line-of-sight imaging system of the area array time-of-flight camera according to claim 1, characterized in that: The data processing module includes: a signal transmission unit, a depth map conversion unit, and a transmission control unit; The signal transmission unit is used for signal transmission between the detection module and the data analysis module; The depth image conversion unit is used to modulate the target image using a four-step phase shift method to obtain a depth image, and filter and denoise the depth image according to a preset photon number threshold; The transmission control unit is used to transmit the filtered and denoised depth image to the data analysis module at a regular interval.

4. The fast non-line-of-sight imaging system of the area array time-of-flight camera according to claim 1, characterized in that: The data analysis module includes: an image reconstruction unit, an image recognition unit, an analysis unit, and a result display unit; The image reconstruction unit is used to perform image denoising, downsampling, region division and extraction, and multiple image joint registration and enhancement processing on the received depth image to obtain the target image in the non-viewing scene; The image recognition unit is used to automatically recognize the reconstructed target image using the trained YOLO neural network, and to alarm and mark the recognized target; The analysis unit is used to analyze the imaging results of the non-viewing scene; The result display unit is used to display the non-viewing area imaging result and the original imaging data to the user.

5. The fast non-line-of-sight imaging system of the area array time-of-flight camera according to claim 2, characterized in that: The area array time-of-flight camera and the light source device are respectively installed on six-degree-of-freedom platforms arranged at different preset positions in the observation area.

6. A fast non-line-of-sight imaging method for an array time-of-flight camera, characterized in that: A fast non-line-of-sight imaging system for implementing the area array time-of-flight camera according to any one of claims 1 to 5, the method comprising: Arrange the area array time-of-flight camera and light source equipment based on preset positions; Start the area array time-of-flight camera and light source equipment to collect target image information scattered back by non-viewing targets in non-viewing scenes; Using a preset data processing module, depth information processing is performed on the target image information to obtain a depth image; The depth image is reconstructed using a preset data analysis module, and the reconstructed image is input into a preset neural network for automatic recognition and early warning.

7. The fast non-line-of-sight imaging method of the area array time-of-flight camera according to claim 6, characterized in that: Arrangement of the array time-of-flight camera and light source equipment based on preset positions includes: The area array time-of-flight camera and the light source device are respectively installed on six-degree-of-freedom platforms arranged at different preset positions in the observation area; The modulation light frequency of the light source signal of the light source device, the exposure time of the area array flight time camera, and the imaging range are set; and the light signal emitted by the light source device can illuminate the non-viewing area and the target therein after passing through the intermediate surface, and part of the light signal is scattered back to the intermediate surface through secondary scattering and scattered tertiary to the observation area.

8. The fast non-line-of-sight imaging method of the area array time-of-flight camera according to claim 6, characterized in that: Before processing the depth information of the target image information, the following steps are also included: The acquired target image information of the non-viewing scene is preliminarily processed, including: denoising and color correction.

9. The fast non-line-of-sight imaging method of the area array time-of-flight camera according to claim 6, characterized in that: The depth information processing of the target image information includes: The target image information is modulated by a four-step phase shift method to obtain a depth image, and the depth image is filtered and denoised according to a preset photon number threshold.

10. The fast non-line-of-sight imaging method of the area array time-of-flight camera according to claim 6, characterized in that: Reconstructing the depth image includes: The depth image is subjected to image denoising, downsampling, region division and extraction, and multiple images are jointly registered and enhanced to reconstruct target image information in the non-viewing scene; The trained YOLO neural network is used to automatically identify the reconstructed target image information, and the identified targets are alarmed and marked; The imaging results of the non-line-of-sight scene are analyzed, and the non-line-of-sight imaging results and the original imaging data are presented to the user.

Images