Depth measurement method and device based on active binocular event camera
Through the depth measurement method based on the active binocular event camera, the high time domain resolution of active infrared structured light and event cameras are used to solve the problem of low accuracy in textureless and dark environments, and real-time of high-speed scenes is achieved Depth measurement and high-precision measurement, suitable for applications such as drone obstacle avoidance and three-dimensional reconstruction.
Patent Information
- Application Number
- CN202410960882.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2044-07-17
AI Technical Summary
The existing three-dimensional depth measurement technology has low accuracy in textureless areas and dark environments. The depth frame rate of traditional active stereo cameras is insufficient, making it difficult to meet the depth perception needs of high-speed scenes, especially when drones fly at high speed to avoid obstacles. risk.
Using the depth measurement method based on the active binocular event camera, active infrared structured light is collected through the binocular event camera, and two event stream data are generated, and feature matching is performed using adaptive division and preset binocular parallax matcher, and one by one is calculated. Pixel parallax map, and the depth measurement value is finally calculated based on the parallax map and the baseline.
Real-time depth measurement in high-speed scenes, improve the measurement accuracy of low-light environments and weak texture scenes, and can effectively utilize the high time domain resolution of event cameras. It is suitable for drone system positioning and obstacle avoidance, three-dimensional reconstruction and other fields. .
Smart Images

Figure CN119152008A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of three-dimensional visual depth measurement, and in particular to a depth measurement method and device based on an active binocular event camera. Background Art
[0002] The three-dimensional scene depth measurement uses different light bands to measure the distance from the scene to the sensor. The main measurement methods now include: laser radar, millimeter wave radar, and visual sensor. The performance parameters of various sensor devices are different, and they are also suitable for scene depth measurement tasks with various detection accuracy and range in different environments. The depth measurement frequency of laser radar and millimeter wave radar is usually 30-90Hz. Its low frequency makes it difficult to meet high-speed obstacle avoidance tasks. At the same time, the volume and weight of these two types of sensors are not suitable for lightweight and agile drones. Using visual sensors for depth measurement is the most common method, which has the advantages of high-resolution real-scene recording, non-contact measurement, and little interference from magnetic fields or sensors. At present, the three-dimensional depth measurement based on the image frame paradigm is the main one, and the passive binocular depth measurement method has the defect of low accuracy in textureless areas and dark environments. In contrast, active stereo obtains a more accurate depth map than passive stereo by projecting infrared texture patterns. However, the imaging paradigm based on traditional frames limits the depth frame rate of traditional active stereo cameras (for example, 30FPS of Kinect V2 and 33FPS of RealSense D435). Traditional active stereo cameras face significant challenges in depth perception in high-speed scenarios. For example, when a drone is flying at high speed and avoiding obstacles, serious collision accidents may occur in the short time between two adjacent depth frames.
[0003] Event cameras, i.e., bio-inspired visual sensors, work differently from traditional frame-based cameras. Instead of capturing frames at a limited fixed rate, event cameras operate by detecting intensity changes at each pixel and providing asynchronous events with high temporal resolution (microseconds). Event streams are described by spatiotemporal sparse event stream data, which have advantages over traditional fixed frame rate cameras in terms of high temporal resolution, high dynamic range, and low power consumption. Therefore, event cameras provide a feasible solution for tasks such as unmanned visual sensors, drone visual sensors, and robot visual navigation and positioning, especially for high-speed depth measurement tasks in agile robots that need to process visual information quickly with low latency.
[0004] Currently, most event camera-based stereo depth measurements are passive stereo camera systems, which can lead to inaccurate depth maps in textureless areas and dark scenes. In fact, passive event stereo methods rely on feature matching, which can be difficult or impossible to match in areas with little texture or low contrast. Although some structured light systems with a single event camera attempt to achieve high-speed depth measurement, active monocular depth methods may not always match the accuracy of stereo depth systems in some scenarios.
[0005] Therefore, how to draw on the neural network and visual sampling and processing mechanism of the biological vision system, propose a depth measurement method based on active binocular events, and establish a new binocular event structured light device for high-speed visual depth measurement has become a core issue that needs to be urgently solved in the field of three-dimensional depth measurement.
[0006] Depth measurement methods based on stereo event cameras can be roughly divided into traditional model optimization methods and deep learning methods. Based on traditional model optimization methods, the matching disparity is optimized by finding the local or global relationship between the two event streams. Although the traditional model optimization method can obtain sparse or semi-dense depth maps in real time, it is difficult to obtain a global dense depth map. Based on deep learning methods, dense depth maps can be obtained from stereo event streams, but these methods are all passive binocular vision, which will lead to inaccurate depth maps in textureless areas and dark scenes. In addition, these deep learning-based methods improve the accuracy of depth measurement by considering the design of complex network structures, and rarely reduce the computational complexity. In other words, these deep learning algorithms require a large amount of GPU memory, making them impractical on mobile robots or mobile devices. Therefore, how to design a lightweight stereo matching neural network is an urgent problem to be solved based on stereo event camera depth measurement.
[0007] More and more works use event cameras combined with active infrared structured light for high-speed depth measurement. Active infrared structured light sources are generally divided into three types (i.e., point, line, and 2D dot speckle pattern). For example, Bramdli et al. first integrated a laser line projector and an event camera for 3D reconstruction. Martel et al. combined a laser light source with an event-based stereo setup. Muglikar et al. designed a structured light system that includes a laser point projector and an event camera for depth estimation. Huang et al. proposed a structured light system that uses an event camera and a laser pattern projector for high-speed 3D scanning. Although these systems based on traditional model optimization methods achieve high-speed depth measurement, they generate sparse depth maps with low accuracy. There are also methods that integrate deep learning models into event-based structured light systems to generate dense depth maps, but the computational complexity is too high. Therefore, there is an urgent need for a depth measurement method that uses an active event stereo structured light system and can generate dense depth maps using a lightweight depth matching model. Summary of the invention
[0008] The present invention provides a depth measurement method and device based on an active binocular event camera to solve the problems that although deep learning methods can obtain dense depth maps from binocular event streams, they are all passive binocular vision, which will lead to inaccurate depth maps in textureless areas and dark scenes and high calculation complexity.
[0009] The first aspect of the present invention provides a depth measurement method based on an active binocular event camera, comprising the following steps: using a binocular event camera to collect active infrared structured light emitted to a target three-dimensional scene to generate two event stream data; adaptively dividing the two event stream data to obtain two discrete event blocks; using a preset binocular disparity matcher to perform feature matching on the two discrete event blocks to calculate two pixel-by-pixel disparity maps; and calculating a depth measurement value based on the two pixel-by-pixel disparity maps and a preset baseline of the binocular event camera.
[0010] Optionally, before using a binocular event camera to collect active infrared structured light emitted to a target three-dimensional scene, the method further includes:
[0011] Determine the infrared spot pattern, infrared light band, infrared light emission frequency and amplitude of the active infrared structured light according to the target depth measurement task;
[0012] Based on the infrared light spot pattern, the infrared light band, the infrared light emission frequency and amplitude, an active infrared structured light emitter is used to emit the active infrared structured light to the target three-dimensional scene.
[0013] Optionally, the infrared spot pattern includes point-shaped active infrared structured light, line-shaped active infrared structured light and dot-matrix active infrared structured light.
[0014] Optionally, the using of a binocular event camera to collect active infrared structured light emitted to a target three-dimensional scene to generate two paths of event stream data includes:
[0015] Calibrate the transformation matrix of the binocular event camera by using a checkerboard to obtain a binocular event camera after baseline correction;
[0016] The baseline-corrected binocular event camera is triggered by hardware clock synchronization to collect active infrared structured light emitted to a target three-dimensional scene, so as to generate two channels of event stream data synchronized in time.
[0017] Optionally, the two event stream data are single-mode event streams or mixed multi-mode visual streams, wherein:
[0018] The single-mode event stream is the light intensity changes in the infrared band and the visible light band of the target three-dimensional scene perceived by the binocular event camera, and the light intensity change pixel points recorded as asynchronous events;
[0019] The mixed multimodal visual stream is the infrared structured light image and light intensity change of the target three-dimensional scene simultaneously perceived and recorded by the binocular event camera.
[0020] Optionally, the preset binocular disparity matcher is any one of an optimization matching solver, an artificial neural network solver and a pulse neural network solver.
[0021] The second aspect of the present invention provides a depth measurement device based on an active binocular event camera, comprising: an active infrared structured light emitter, for emitting active infrared structured light to a target three-dimensional scene; a binocular event camera, the binocular event camera comprising a left event camera and a right event camera, for collecting the active infrared structured light to generate two event stream data; a division module, for adaptively dividing the two event stream data to obtain two discrete event blocks; a binocular matching calculation module, for performing feature matching on the two discrete event blocks to calculate two pixel-by-pixel disparity maps; and an output module, for calculating a depth measurement value based on the two pixel-by-pixel disparity maps and a preset baseline of the binocular event camera.
[0022] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a depth measurement method based on an active binocular event camera as described in the above embodiment.
[0023] A fourth aspect of the present invention provides a computer program product, which implements the above-mentioned depth measurement method based on an active binocular event camera when the computer program / instructions are executed by a processor.
[0024] A fifth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the depth measurement method based on an active binocular event camera as described above.
[0025] The depth measurement method and device based on the active binocular event camera proposed in the embodiment of the present invention can effectively utilize the high temporal resolution provided by the event camera to perform real-time depth measurement of high-speed scenes; at the same time, active structured light has higher measurement accuracy in low-light environments and weak-texture scenes than passive vision; and has broad application potential in fields involving high-speed depth measurement such as unmanned aerial vehicle system positioning and obstacle avoidance, three-dimensional reconstruction, and virtual reality.
[0026] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0028] Figure 1 A flow chart of a depth measurement method based on an active binocular event camera provided by an embodiment of the present invention;
[0029] Figure 2 A schematic diagram of the structure of a binocular event camera provided in an embodiment of the present invention collecting active infrared structured light emitted to a target three-dimensional scene;
[0030] Figure 3 A schematic diagram of the structure of a binocular event structured light prototype device for high-speed visual depth measurement provided by an embodiment of the present invention;
[0031] Figure 4 A schematic diagram of the principle of a depth measurement method for an active binocular event camera provided in an embodiment of the present invention;
[0032] Figure 5 A schematic diagram of a lightweight event stereo matching neural network provided by an embodiment of the present invention;
[0033] Figure 6 A performance diagram of each Active-Event-Stereo test sequence provided by an embodiment of the present invention;
[0034] Figure 7A schematic diagram showing a performance comparison of various binocular stereo matching algorithms provided by an embodiment of the present invention;
[0035] Figure 8 A schematic diagram of the visualization results of each Active-Event-Stereo test sequence provided in an embodiment of the present invention;
[0036] Fig. 9 A schematic diagram showing a comparison of visualization results of an embodiment of the present invention compared with other methods;
[0037] Fig.10 The visualization results of the depth map of the embodiment of the present invention and Realsense D455;
[0038] Fig.11 A block diagram of a depth measurement device based on an active binocular event camera provided by an embodiment of the present invention;
[0039] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0040] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0041] The following describes a depth measurement method and device based on an active binocular event camera according to an embodiment of the present invention with reference to the accompanying drawings.
[0042] Figure 1 A schematic flow chart of a depth measurement method based on an active binocular event camera provided in an embodiment of the present invention.
[0043] like Figure 1 As shown, the depth measurement method based on the active binocular event camera includes the following steps:
[0044] In step S101, a binocular event camera is used to collect active infrared structured light emitted to a target three-dimensional scene to generate two channels of event stream data.
[0045] In some embodiments, before using a binocular event camera to collect active infrared structured light emitted to a target three-dimensional scene, the method further includes:
[0046] Determine the infrared spot pattern, infrared light band, infrared light emission frequency and amplitude of the active infrared structured light according to the target depth measurement task;
[0047] Based on the infrared spot pattern, infrared light band, infrared light emission frequency and amplitude, an active infrared structured light emitter is used to emit active infrared structured light to the target three-dimensional scene.
[0048] In some embodiments, a binocular event camera is used to collect active infrared structured light emitted to a target three-dimensional scene to generate two event stream data, including:
[0049] Calibrate the transformation matrix of the binocular event camera by using a checkerboard to obtain a binocular event camera after baseline correction;
[0050] The baseline-corrected binocular event camera is triggered by hardware clock synchronization to collect active infrared structured light emitted to a target three-dimensional scene, so as to generate two channels of event stream data synchronized in time.
[0051] In the actual implementation process, Figure 2 As shown, the embodiment of the present invention preselects a binocular event camera composed of two event cameras capable of sensing infrared structured light effects and an active infrared structured light emitter as an active binocular event acquisition device. The spectral perception range of the light quantum effect curve of each event camera is generally 300nm-1000nm, which means that it can sense both visible light and infrared spectrum. The spectrum of active infrared structured light is generally around 850nm (similar to the infrared projector of Intel RealSense series cameras). Active infrared structured light sources are generally divided into three types (i.e., points, lines, and 2D dot speckle patterns). The infrared light band, infrared light emission frequency, and amplitude are selected according to the actual depth measurement task. Therefore, in principle, the active infrared structured light emitted by the active infrared structured light emitter to the target three-dimensional scene includes but is not limited to: infrared spot mode, which can emit active infrared structured light in the form of points, lines, or dot matrix according to the range of the scene depth measurement area; infrared light band, which selects the spectral working band of the active infrared structured light according to the scene depth measurement working environment; infrared light emission frequency and amplitude, which sets the amplitude and frequency of the active infrared structured light according to the output frequency of the scene depth measurement to meet the needs of dynamic light intensity changes of the event camera.
[0052] Furthermore, after determining the infrared spot pattern, infrared light band, infrared light emission frequency and amplitude of the active infrared structured light according to the target depth measurement task, the transformation matrix of the binocular event camera is calibrated by the checkerboard to obtain the binocular event camera after baseline correction.
[0053] Furthermore, active infrared structured light with determined infrared spot pattern, infrared light band, infrared light emission frequency and amplitude is emitted to the target three-dimensional scene, and the binocular event camera after baseline correction is triggered by hardware clock synchronization to collect active infrared structured light to generate two event stream data.
[0054] For example, Figure 3 As shown, the active binocular event acquisition device can be mainly integrated with two DAVIS346 event cameras and an infrared dot speckle projector. The acquisition device is only a prototype system for active binocular event data acquisition and measurement. The system can be further integrated and miniaturized. The active binocular event acquisition device can sense static scenes and generate dynamic events by adjusting the frequency or intensity of the laser signal generator. In addition, in order to enable the active binocular event acquisition device of the embodiment of the present invention to measure the depth in high-speed motion scenes, the current flagship binocular depth camera RealSense D455 can also be used. In order to meet the comparative analysis of different cameras under the same perspective, different sensors need to be synchronized in time and space. For time calibration, the active binocular event acquisition device of the embodiment of the present invention can use the timestamp synchronization event camera and RealSense D455 sensor of the ROS system. For spatial calibration, the active binocular event acquisition device of the embodiment of the present invention can use a standard checkerboard to perform baseline correction on the DAVIS346 camera, and perform spatial registration between the left event camera and the RealSense D455 sensor.
[0055] In step S102, two channels of event stream data are adaptively divided to obtain two channels of discrete event blocks.
[0056] In the actual execution process, the continuous event streams in the two event stream data are adaptively divided according to the speed of scene movement to obtain two discrete event blocks, where the two event stream data are unimodal event streams or mixed multimodal visual streams. The unimodal event stream is the infrared band and visible light band light intensity changes of the target three-dimensional scene perceived by the binocular event camera, and the light intensity change pixel points are recorded as asynchronous events. The mixed multimodal visual stream is the infrared structured light image and light intensity changes of the target three-dimensional scene simultaneously perceived and recorded by the binocular event camera.
[0057] In step S103, a preset binocular disparity matcher is used to perform feature matching on two discrete event blocks to calculate two pixel-by-pixel disparity maps.
[0058] Among them, the preset binocular disparity matcher is any one of an optimization matching solver, an artificial neural network solver and a pulse neural network solver.
[0059] In the actual execution process, any of the optimization matching solvers, artificial neural network solvers and pulse neural network solvers can be selected to perform feature matching on the two discrete event blocks, and then calculate the two pixel-by-pixel disparity maps. Among them, when the traditional model optimization matching solver is selected, the feature expression of the two event camera blocks is performed, and then the classic local dense matching, global dense matching and other algorithms are used to optimize the disparity solution; when the artificial neural network solver is selected, an end-to-end deep artificial neural network is designed, and the disparity is learned by inference by building a binocular structured light disparity estimation data driver; when the pulse neural network solver is selected, a biologically inspired low-power pulse neural network is designed, and the disparity is learned by inference by mining the spatiotemporal relationship of the two event data.
[0060] For example, Figure 4-5 As shown in the figure, a lightweight event stereo matching neural network (i.e., a preset binocular disparity matcher) is designed, which mainly includes four modules: event representation module, lightweight feature extraction module, dynamic interaction construction and calculation matching cost volume module, and encoding and decoding output module, among which the lightweight feature extraction module and the dynamic interaction construction and calculation matching cost volume module are the main modules. Specifically, two event stream data are collected, and the two event stream data are adaptively divided according to the measurement task requirements to obtain discrete event blocks (event bins), and event features are represented for each event block and event tensors are output; a lightweight MobileNet is designed as the backbone network of the event stereo matching neural network, with compact and powerful features in advance; the matching cost of the two feature streams is calculated by designing a channel interaction module, that is, the cost of matching the corresponding pixels between the two event streams at slightly different viewpoints; finally, the constructed matching cost volume is output to the encoder-decoder output module to predict the dense depth map.
[0061] Among them, the feature representation module: the asynchronous spatiotemporal event number presents a sparse discrete dot matrix in the three-dimensional space in the time domain and the space domain, which is different from the structured image frame and cannot be directly compatible with the existing mainstream deep learning binocular matcher. Existing neuromorphic vision algorithms generally need to take two operations: (1) The continuous event stream is divided into discrete pulse blocks in the time domain according to a fixed time domain window length or a fixed number of events. For example, a fixed time domain length of 30 milliseconds is equivalent to an image sequence of about 33 Hz; (2) An asynchronous event signal representation method is designed to encode the asynchronous event signal in the discrete pulse block into a feature that is compatible with the existing mainstream deep learning algorithm. The asynchronous event signal representation can be formally defined as:
[0062]
[0063] In the formula, S L (x n ,yn ,t n ) represents the asynchronous event stream output by the event camera on the left. is the event vector. k(x,y,t) is the kernel function, which can be a manually designed function or can be learned and represented by an artificial neural network or a pulse neural network.
[0064] In short, event representation is to make asynchronous events compatible with the binocular matching algorithm of the existing image frame paradigm. The current mainstream event representation methods include event image, voxel grid and event embedding.
[0065] Lightweight feature extraction module: In order to ensure the trade-off between the accuracy and speed of depth measurement, the embodiment of the present invention uses lightweight MobileNet as the backbone network of the binocular event matching network. Currently, MobileNet v1, MobileNet v2 and MobileNet v3 are lightweight modules used in the mainstream, and the three do not use traditional convolution operations. For example, MobileNet v1 uses depth-separable convolution, followed by point-by-point convolution, to achieve the spatial dimension of standard convolution while reducing calculations. Compared with standard convolution operations, depth-separable convolution reduces the amount of calculation by 8 to 9 times by effectively utilizing depth-separable convolution with 3x3 convolution kernels. In order to further provide the accuracy of MobileNet v1 while maintaining considerable computational complexity, MobileNet v2 introduces linear bottlenecks and reverse residuals. The entire structure of MobileNet v3 uses NAS neural network search, not artificial a priori design, and the improvement of downstream task effects is limited. Therefore, the present invention uses lightweight MobileNet v1 or MobileNet v2 operations to replace standard 2D convolutions in the feature extraction module and encoder-decoder module of the lightweight binocular event matching network.
[0066] Module for calculating matching cost volume: The matching cost volume represents the matching cost between pixels under different disparities, and is also a key step in the process of calculating matching disparity based on deep learning methods. Usually, the matching volume is a 3D or 4D tensor constructed based on the similarity between the left and right features. For example, the matching cost volume of a 3D tensor can be formally defined as:
[0067]
[0068] Where d is the disparity of binocular stereo matching, is a similarity measurement function. In the era of deep learning, Usually operations such as aggregation or calculating correlation are used.L and F R They are the features output by the left and right backbone networks respectively.
[0069] Finally, a dynamic interaction strategy is designed to construct the matching cost volume by L and F R Dynamic interaction is performed during training, and then the two features are interacted and The aggregation process can be formally described as:
[0070]
[0071]
[0072] In the formula, and They are the dynamic interaction features of the left and right visual streams, W R and W L are the weight parameters corresponding to the left and right visual streams respectively. The above dynamic interaction is based on the channel threshold parameters after the normalization operation of the two visual features to determine whether to perform feature interaction.
[0073] The matching cost volume C generated above d It is directly input into the encoder-decoder module, and the disparity map of each pixel is predicted by regression. The disparity map of each pixel can have multiple depth map modes, which can be selected according to the task requirements, including: dense depth map, which obtains the depth information of each pixel in the scene, which can meet the task of high-precision dense measurement and reconstruction of the scene; semi-dense depth map, which obtains the depth information of local pixels in the scene, which meets the trade-off between accuracy and inference speed of specific scenes; sparse depth map, which obtains the depth information of a few changing pixels in the scene, which meets the task of asynchronous high-speed depth measurement.
[0074] In step S104, a depth measurement value is calculated based on the two pixel-by-pixel disparity maps and a preset baseline of the binocular event camera.
[0075] The depth measurement method based on the active binocular event camera proposed in the embodiment of the present invention is quantitatively and qualitatively evaluated below to demonstrate its beneficial effects.
[0076] Experimental data preparation: A stereo matching dataset for active event cameras, namely Active-Event-Stereo, was established. First, 119 infrared video sequences were collected, taking into account factors such as speed distribution, lighting conditions, and scene diversity. The dataset provides event streams of stereo pairs and 23.8k synchronized true labels. Finally, the dataset is divided into 16,000 for training, 3,800 for validation, and 400 for testing.
[0077] Experimental parameter settings: The maximum disparity is set to 192, and all networks are trained for 50 epochs using the Adam optimizer on the NVIDIA GeForce RTX 3090 GPU with a learning rate of 0.001. For training loss, L1 loss is used to measure the absolute difference between the predicted map and the label, and gradient loss is used to promote the smoothness of the predicted map. In addition, the average pixel error (EPE), root mean square error (RMSE), the percentage of pixel disparity errors less than 3 pixels and 0.05 true values (D1-all), and the percentage of pixel disparity errors greater than 1 pixel, 2 pixels, and 3 pixels are used to evaluate the accuracy in the stereo matching task. Model parameters (#Params) and running time (ms) are used to evaluate the computational complexity.
[0078] Quantitative experimental results:
[0079] 1) Performance of each sequence in the dataset. Figure 6 As shown, the performance of each Active-Event-Stereo test sequence is demonstrated. It can be seen that the average pixel error (EPE) of the embodiment of the present invention is 1.223, and the inference speed reaches 21.9ms on the NVIDIA GeForce RTX 3090 GPU.
[0080] 2) Performance comparison of each method. Figure 7 As shown, the comparison results between the embodiment of the present invention and various existing binocular stereo matching algorithms are shown. It can be seen that the embodiment of the present invention has strong advantages in both disparity estimation accuracy and reasoning speed.
[0081] Qualitative visualization results:
[0082] 1) Visualization results of each sequence in the dataset. Figure 8 As shown, the solution with structured light in the embodiment of the present invention can obtain high-quality dense disparity maps in indoor and outdoor scenes, and performs well under changing lighting conditions. Especially in dark environments, the active event stereo solution with structured light in the embodiment of the present invention outperforms the passive stereo solution.
[0083] 2) Visualization comparison results of each method. Fig. 9 As shown, the SGM based on the traditional matching optimization model can only generate sparse disparity maps. The embodiments of the present invention can excellently obtain high-quality dense disparity maps in both indoor and outdoor scenes, even when the lighting conditions change.
[0084] 3) Depth measurement results in high-speed scenes. In order to verify the solution of the embodiment of the present invention for high-speed depth perception, the camera prototype of the embodiment of the present invention was compared with a traditional active stereo camera (i.e., RealSense D455). Fig.10 As shown, the camera prototype of the embodiment of the present invention can obtain high-quality disparity maps in high-speed scenes, while the RealSense D455 obviously does not perform well at 90FPS. This is because the RGB image of the RealSense D455 may be motion blurred. In contrast, the embodiment of the present invention achieves 150FPS in disparity estimation at a resolution of 346×260.
[0085] The depth measurement method based on the active binocular event camera proposed in the embodiment of the present invention can effectively utilize the high temporal resolution provided by the event camera to perform real-time depth measurement of high-speed scenes; at the same time, active structured light has higher measurement accuracy in low-light environments and weak-texture scenes than passive vision; and it has broad application potential in fields involving high-speed depth measurement such as unmanned aerial vehicle system positioning and obstacle avoidance, three-dimensional reconstruction, and virtual reality.
[0086] Next, a depth measurement device based on an active binocular event camera according to an embodiment of the present invention will be described with reference to the accompanying drawings.
[0087] Fig.11 4 is a block diagram of a depth measurement device based on an active binocular event camera according to an embodiment of the present invention.
[0088] like Fig.11 As shown, the depth measurement device 110 based on the active binocular event camera includes: an active infrared structured light emitter 1101 , a binocular event camera 1102 , a division module 1103 , a binocular matching calculation module 1104 and an output module 1105 .
[0089] Among them, the active infrared structured light emitter 1101 is used to emit active infrared structured light to the target three-dimensional scene. The binocular event camera 1102 binocular event camera includes a left event camera and a right event camera, which are used to collect active infrared structured light to generate two event stream data. The division module 1103 is used to adaptively divide the two event stream data to obtain two discrete event blocks. The binocular matching calculation module 1104 is used to perform feature matching on the two discrete event blocks to calculate two pixel-by-pixel disparity maps. The output module 1105 is used to calculate the depth measurement value based on the two pixel-by-pixel disparity maps and the preset baseline of the binocular event camera.
[0090] It should be noted that the aforementioned explanation of the embodiment of the depth measurement method based on the active binocular event camera is also applicable to the depth measurement device based on the active binocular event camera in this embodiment, which will not be repeated here.
[0091] The depth measurement device based on the active binocular event camera proposed in the embodiment of the present invention can effectively utilize the high temporal resolution provided by the event camera to perform real-time depth measurement of high-speed scenes; at the same time, active structured light has higher measurement accuracy than passive vision in low-light environments and weak-texture scenes; it has broad application potential in fields involving high-speed depth measurement such as unmanned aerial vehicle system positioning and obstacle avoidance, three-dimensional reconstruction, and virtual reality.
[0092] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:
[0093] A memory 1201 , a processor 1202 , and a computer program stored in the memory 1201 and executable on the processor 1202 .
[0094] When the processor 1202 executes the program, the depth measurement method based on the active binocular event camera provided in the above embodiment is implemented.
[0095] Furthermore, the electronic device further comprises:
[0096] The communication interface 1203 is used for communication between the memory 1201 and the processor 1202 .
[0097] The memory 1201 is used to store computer programs that can be executed on the processor 1202 .
[0098] The memory 1201 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0099] If the memory 1201, the processor 1202 and the communication interface 1203 are implemented independently, the communication interface 1203, the memory 1201 and the processor 1202 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0100] Optionally, in a specific implementation, if the memory 1201, the processor 1202 and the communication interface 1203 are integrated on a chip, the memory 1201, the processor 1202 and the communication interface 1203 can communicate with each other through an internal interface.
[0101] The processor 1202 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.
[0102] An embodiment of the present invention further provides a computer program product, and when the computer program / instructions are executed by a processor, the depth measurement method based on an active binocular event camera as described above is implemented.
[0103] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the depth measurement method based on an active binocular event camera as described above is implemented.
[0104] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0105] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0106] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.
[0107] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.
[0108] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0109] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0110] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0111] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.
Claims
1. A depth measurement method based on an active binocular event camera, characterized in that: The following steps are involved: The binocular event camera is used to collect active infrared structured light emitted to the target three-dimensional scene to generate two event stream data; Adaptively dividing the two event stream data to obtain two discrete event blocks; Performing feature matching on the two discrete event blocks using a preset binocular disparity matcher to calculate two pixel-by-pixel disparity maps; Depth measurements are calculated based on the two pixel-by-pixel disparity maps and a preset baseline of the binocular event camera.
2. The depth measurement method based on an active binocular event camera according to claim 1, characterized in that: Before using a binocular event camera to collect active infrared structured light emitted to a target three-dimensional scene, it also includes: Determine the infrared spot pattern, infrared light band, infrared light emission frequency and amplitude of the active infrared structured light according to the target depth measurement task; Based on the infrared light spot pattern, the infrared light band, the infrared light emission frequency and amplitude, an active infrared structured light emitter is used to emit the active infrared structured light to the target three-dimensional scene.
3. The depth measurement method based on an active binocular event camera according to claim 2, characterized in that: The infrared light spot patterns include point-shaped active infrared structured light, line-shaped active infrared structured light and dot-matrix active infrared structured light.
4. The depth measurement method based on an active binocular event camera according to claim 1, characterized in that: The method of using a binocular event camera to collect active infrared structured light emitted to a target three-dimensional scene to generate two event stream data includes: Calibrate the transformation matrix of the binocular event camera by using a checkerboard to obtain a binocular event camera after baseline correction; The baseline-corrected binocular event camera is triggered by hardware clock synchronization to collect active infrared structured light emitted to a target three-dimensional scene, so as to generate two channels of event stream data synchronized in time.
5. The depth measurement method based on an active binocular event camera according to claim 1, characterized in that: The two event stream data are single-mode event streams or mixed multi-mode visual streams, wherein: The single-mode event stream is the light intensity changes in the infrared band and the visible light band of the target three-dimensional scene perceived by the binocular event camera, and the light intensity change pixel points recorded as asynchronous events; The mixed multimodal visual stream is the infrared structured light image and light intensity change of the target three-dimensional scene simultaneously perceived and recorded by the binocular event camera.
6. The depth measurement method based on an active binocular event camera according to claim 1, characterized in that: The preset binocular disparity matcher is any one of an optimization matching solver, an artificial neural network solver and a pulse neural network solver.
7. A depth measurement device based on an active binocular event camera, characterized in that: include: An active infrared structured light emitter, used to emit active infrared structured light to a target three-dimensional scene; A binocular event camera, comprising a left event camera and a right event camera, for collecting the active infrared structured light to generate two event stream data; A partitioning module, used for adaptively partitioning the two event stream data to obtain two discrete event blocks; A binocular matching calculation module is used to perform feature matching on the two discrete event blocks to calculate two pixel-by-pixel disparity maps; An output module is used to calculate a depth measurement value according to the two pixel-by-pixel disparity maps and a preset baseline of the binocular event camera.
8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the depth measurement method based on an active binocular event camera as described in any one of claims 1 to 6.
9. A computer program product, characterized in that When the computer program / instruction is executed by a processor, the depth measurement method based on an active binocular event camera according to any one of claims 1 to 6 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the depth measurement method based on an active binocular event camera as described in any one of claims 1 to 6.
Citation Information
Patent Citations
A measuring method based on active type binocular stereoscopic vision and an RGBD camera system
CN107917701A
Image depth information generation method and device and storage medium
CN112927279A
Photoelectric calculation light field intelligent depth classification method and device
CN113780258A
Event camera-based binocular depth estimation method
CN118196160A
Cited By
Binocular stereo matching method for non-polar correction
CN119672378A
