Indoor depth estimation method based on event stream and image frame combination

Abstract

The application relates to the technical field of depth estimation, in particular to an indoor depth estimation method based on joint event stream and image frame, wherein the method comprises the following steps: fixing a depth camera and an event camera; performing time registration and space registration on the depth camera and the event camera, mapping an image collected by the depth camera to an image collected by the event camera, and obtaining a time-space aligned image frame data set; for each frame of the image collected by the event camera, selecting a real depth map with a collection time interval less than a preset interval as a real depth label; training a depth estimation network by using the image frame data set to obtain a trained depth estimation network; inputting an event stream image and an image sequence of an indoor scene into the trained depth estimation network; and outputting a depth estimation result of the indoor scene. Thus, the problems in the related art that the indoor scene environment is various and complex in structure, the indoor monocular depth estimation accuracy is low, and the indoor monocular depth estimation cannot meet the actual use requirements are solved.

Description

Technical Field

[0001] This application relates to the field of depth estimation technology, and in particular to an indoor depth estimation method based on the combination of event stream and image frames. Background Technology

[0002] Indoor monocular depth estimation has important applications in robot navigation, 3D reconstruction, and object grasping. One related technology uses an event-based camera-based depth estimation method, primarily focused on outdoor scenes in autonomous driving. In outdoor scenes, the camera can move smoothly along the road, and the depth distribution of the scene can be consistent from near to far along the road. However, when shooting indoor scenes, due to the diverse and complex structures of the indoor environment, it cannot handle indoor depth estimation and cannot meet the needs of practical applications. Summary of the Invention

[0003] This application provides an indoor depth estimation method, apparatus, electronic device, and storage medium based on the combination of event stream and image frames, in order to solve the problems in related technologies such as the diverse and complex indoor scene environments, which lead to low accuracy of indoor monocular depth estimation and failure to meet practical needs.

[0004] The first aspect of this application provides an indoor depth estimation method based on the joint use of event streams and image frames, comprising the following steps: fixing a depth camera and an event camera, wherein the depth camera is used to acquire indoor true depth maps, and the event camera is used to acquire indoor aligned event stream images and image sequences; performing temporal and spatial registration on the depth camera and the event camera, mapping the images acquired by the depth camera onto the images acquired by the event camera, to obtain a spatiotemporally aligned image frame dataset, wherein for each frame of the image acquired by the event camera, a true depth map with a time interval less than a preset interval is selected as the true depth label; training a depth estimation network using the image frame dataset to obtain a trained depth estimation network; inputting the indoor event stream images and image sequences into the trained depth estimation network, and outputting the indoor depth estimation result.

[0005] Optionally, the depth estimation network includes a convolutional encoder network, a feature temporal fusion network, and a decoder network. The convolutional encoder network takes the event stream image and the image sequence as input and outputs joint loss voxel features. The feature temporal fusion network is used to perform temporal fusion on the joint loss voxel features to obtain fused features. The decoder network is used to perform depth estimation based on the fused features to obtain the depth estimation result.

[0006] Optionally, the convolutional encoder network is further configured to: extract first to fourth deep features from adjacent event stream images and adjacent image frames in the image sequence, respectively; discretize multiple depth planes with calculated loss errors from a preset depth range; extract first and second features corresponding to the second and fourth deep features, respectively; calculate the first voxel loss of the event stream image based on the first deep feature and the first feature; calculate the second voxel loss of the image frame based on the third deep feature and the second feature; calculate the loss voxel of the depth plane based on the first voxel loss and the second voxel loss; and concatenate the loss voxels of each depth plane to obtain the joint loss voxel features of the event stream image and the image sequence.

[0007] Optionally, the loss function of the convolutional encoder network is:

[0008] L depth =w1L s +w2L g +w3L l

[0009] Among them, L l For the structural line loss, N is extracted using event streams and image frames. l A line is formed, and multiple points are randomly selected on each line to construct the structure line loss L. l Gradient loss L g To ensure gradient consistency between the predicted depth map and the true depth map; w1, w2, and w3 are 1.0, 0.5, and 0.01, respectively; scale-invariant loss L s The calculation method is as follows:

[0010]

[0011] Here, Mean represents the averaging operation. and D p Let represent the predicted depth and the true depth of pixel p, respectively, and let λ control the effect of scale ambiguity on the loss function.

[0012] Optionally, the feature temporal fusion network is further configured to: update the latent features of the current frame based on the joint loss voxel features of the current frame and the latent features of the previous frame, such that the joint loss voxel features contain long-sequence temporal information, wherein the update process includes:

[0013] z t =σ(W z *Concat(F t H t-1 ))

[0014] r t =σ(W r *Concat(Ft H t-1 ))

[0015]

[0016]

[0017] Among them, W * F represents the network weights, Concat represents the feature concatenation operation, σ represents the non-linear activation function, and F... * H represents the features extracted by the pre-network module. * The hidden z-axis represents the maintenance and update of the recurrent network. * This indicates updating the gate weights, which are determined by F. * With H * It is calculated.

[0018] A second aspect of this application provides an indoor depth estimation method based on the joint use of event streams and image frames, comprising: acquiring indoor event stream images and image sequences; inputting the indoor event stream images and image sequences into a trained depth estimation network, and outputting the indoor depth estimation result. The depth estimation network includes a convolutional encoder network, a feature temporal fusion network, and a decoder network. The convolutional encoder network takes the event stream images and the image sequences as input and outputs joint loss voxel features. The feature temporal fusion network is used to temporally fuse the joint loss voxel features to obtain fused features. The decoder network is used to perform depth estimation based on the fused features to obtain the depth estimation result.

[0019] A third aspect of this application provides an indoor depth estimation device based on the joint use of event streams and image frames, comprising: a camera module including a fixed depth camera and an event camera, wherein the depth camera is used to acquire indoor true depth maps, and the event camera is used to acquire indoor aligned event stream images and image sequences; a mapping module, used to perform temporal and spatial registration on the depth camera and the event camera, mapping the images acquired by the depth camera onto the images acquired by the event camera to obtain a spatiotemporally aligned image frame dataset, wherein for each frame of the image acquired by the event camera, a true depth map with a time interval less than a preset interval is selected as the true depth label; and a training module, used to train a depth estimation network using the image frame dataset to obtain a trained depth estimation network, inputting indoor event stream images and image sequences into the trained depth estimation network, and outputting the indoor depth estimation result.

[0020] A fourth aspect of this application provides an indoor depth estimation device based on the joint use of event streams and image frames, comprising: an acquisition module for acquiring indoor event stream images and image sequences; and an output module for inputting the indoor event stream images and image sequences into a trained depth estimation network and outputting the indoor depth estimation result. The depth estimation network includes a convolutional encoder network, a feature temporal fusion network, and a decoder network. The convolutional encoder network takes the event stream images and the image sequences as input and outputs joint loss voxel features. The feature temporal fusion network performs temporal fusion on the joint loss voxel features to obtain fused features. The decoder network performs depth estimation based on the fused features to obtain the depth estimation result.

[0021] A fifth aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the indoor depth estimation method based on the combination of event stream and image frames as described in the above embodiments.

[0022] A sixth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the indoor depth estimation method based on the combination of event streams and image frames as described in the above embodiments.

[0023] Therefore, this application has at least the following beneficial effects:

[0024] This invention can acquire indoor event stream images and image sequences, perform spatiotemporal registration, and establish a depth estimation network that incorporates joint loss voxel representation and multimodal temporal information, using a standard indoor dataset as an evaluation benchmark. The training of this depth estimation network is completed, and the indoor depth estimation results are obtained by taking indoor event stream images and image sequences as input. Therefore, this embodiment can effectively integrate the spatiotemporal information of neighboring frames of event stream image frames, providing more effective information for indoor depth estimation tasks. This makes depth estimation more adaptable to complex indoor environments, enabling depth estimation tasks to be completed both offline and online. The processing is more intelligent, enhancing the accuracy of depth estimation results and meeting practical needs. Thus, it solves the technical problems in related technologies, such as the diverse and complex indoor scene environments, low accuracy of indoor monocular depth estimation, and inability to meet practical needs.

[0025] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0026] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0027] Figure 1 This is a flowchart of an indoor depth estimation method based on the combination of event stream and image frames, according to an embodiment of this application.

[0028] Figure 2 A flowchart illustrating the steps for establishing a dataset according to embodiments of this application;

[0029] Figure 3 This is a schematic diagram of a data acquisition device provided according to an embodiment of this application;

[0030] Figure 4 This is an example diagram of a dataset provided according to an embodiment of this application;

[0031] Figure 5 This is a schematic diagram of the overall structure of the PARNet (Parallel Networks) network provided according to an embodiment of this application;

[0032] Figure 6 This is a schematic diagram illustrating the visual comparison results provided according to the embodiments of this application;

[0033] Figure 7 This is a flowchart of another indoor depth estimation method based on the combination of event stream and image frames, according to an embodiment of this application;

[0034] Figure 8 This is an example diagram of an indoor depth estimation device based on the combination of event stream and image frames, according to an embodiment of this application.

[0035] Figure 9 This is an example diagram of another indoor depth estimation device based on the combination of event stream and image frames provided according to an embodiment of this application;

[0036] Figure 10 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0037] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0038] Indoor monocular depth estimation, a long-standing and challenging problem, has important applications in robot navigation, 3D reconstruction, and object grasping. While traditional image-frame-based indoor depth estimation algorithms have developed rapidly, the performance of these algorithms degrades in complex and variable indoor environments, particularly in extreme scenarios such as high-speed blurring and low light. This is fundamentally because the camera sensor cannot effectively collect scene information under these extreme conditions, hindering the algorithm's proper functioning. Consequently, event-based computer vision tasks have seen rapid development. Event cameras, as bio-inspired neuromorphic sensors, output event streams with extremely high temporal resolution and a wide dynamic range, performing well even in low-light, high-speed scenes, thus helping to solve various computer vision tasks in extreme environments.

[0039] Among related technologies, event-based camera-based depth estimation methods can be used, mainly focusing on outdoor scenes for autonomous driving. Compared to outdoor scenes, indoor scenes have more diverse geometric structures. The depth distribution in outdoor scenes is mostly along the road from near to far, while indoor environments are very diverse and do not have such geometric consistency. Moreover, when shooting outdoor scenes, the camera basically moves smoothly along the road, while when shooting indoor scenes, the camera's position and pose change more diversely, which further increases the complexity of the scene.

[0040] The following are some of the event-stream-based depth estimation algorithms and related datasets:

[0041] (1) Event Stream Depth Datasets: Due to the relatively late development of event cameras, only some depth datasets contain event stream data. For example, the real-world event stream and discrete depth map datasets MVSEC (The Multi Vehicle Stereo Event Camera Dataset) and DSEC (Driving Stereo Event Camera Dataset) obtained using event cameras and LiDAR (Light Detection and Ranging) are real-world datasets. The virtual dataset DENSE (Depth Estimation on Synthetic Events) obtained using the CARLA (Car Learning to Act) simulator is a virtual dataset that simulates event streams and depth maps. However, these datasets only focus on depth data of outdoor scenes and contain only a very small number of indoor scenes, making them unsuitable for algorithm evaluation. Furthermore, some works use event flow simulators to simulate event flows from existing indoor image frame depth datasets or image frame 3D reconstruction datasets. However, these methods cannot reflect real low-light, high-speed scenes, and the simulation results deviate from reality. Essentially, they still infer event flows based on existing image frames without introducing more information. In summary, there is currently a lack of a standard real-world indoor scene event flow depth dataset as a benchmark for evaluating related algorithms, which limits the development of this field.

[0042] (2) Single-modal event stream depth estimation algorithms: These methods rely solely on event streams for depth estimation, primarily using neural networks to directly estimate dense depth maps from various representations of the event stream (including but not limited to event images, event spatiotemporal voxels, etc.). For example, E2Depth was the first to utilize neural networks to directly recover depth from event streams, achieving reasonable results. DTL, on the other hand, leverages transfer learning to utilize prior knowledge from image models to enhance event stream depth estimation. However, compared to traditional image frames, event cameras lack rich texture information from the scene and cannot acquire event streams in static scenes, which limits the capabilities of these algorithms. Furthermore, they have only been trained and tested in outdoor scenes and have not been further explored and experimented with in more complex and diverse indoor scenes.

[0043] (3) Multimodal depth estimation algorithms combining event streams and image frames: These methods rely on both event streams and image frames as inputs to predict depth maps. Event streams have high dynamic range and high temporal resolution, while image frames provide richer information and can handle static scenes. The complementary nature of these two modalities makes these methods more robust and accurate than single-modal depth estimation algorithms. For example, RAMNet uses a recurrent neural network to asynchronously fuse event stream and image frame modalities, improving the network's depth estimation accuracy in nighttime outdoor scenes. However, due to limitations in current depth datasets, these methods have not been trained and evaluated in indoor scenes. Furthermore, these methods directly use event images or event voxel representations as inputs, resulting in a coarse processing of event stream temporal information and failing to consider the multi-view geometric relationships inherent in the depth estimation task itself, thus losing some effective spatiotemporal information.

[0044] In view of the problem mentioned in the background art that the indoor scene environment is diverse and the structure is complex, resulting in low accuracy of indoor monocular depth estimation and failing to meet the needs of practical use, this application provides an indoor depth estimation method, device, electronic device and storage medium based on the combination of event stream and image frame. The following describes the indoor depth estimation method, device, electronic device and storage medium based on the combination of event stream and image frame of this application with reference to the accompanying drawings.

[0045] Specifically, Figure 1 This is a flowchart illustrating an indoor depth estimation method based on the combination of event stream and image frames, provided in an embodiment of this application.

[0046] like Figure 1 As shown, this indoor depth estimation method based on the joint analysis of event streams and image frames includes the following steps:

[0047] In step S101, a depth camera and an event camera are fixed, wherein the depth camera is used to acquire a true indoor depth map, and the event camera is used to acquire indoor aligned event stream images and image sequences.

[0048] It is understandable that, such as Figure 2 As shown, in this embodiment of the application, a depth camera and an event camera can be used as sensors for installation. First, the depth camera that collects the true depth of the room and the event camera that collects aligned event stream images and image sequences of the room are fixed. In this embodiment of the application, at least one method can be used for fixing, such as using a handheld bracket, etc., without specific limitation.

[0049] It should be noted that, since the infrared light emitted by the depth camera can severely affect the data set of the event camera, this embodiment of the application can add a visible light filter in front of the event camera lens to remove the infrared light and ensure the proper operation of the event camera; and the fixing of the depth camera and the event camera in this embodiment of the application needs to ensure that their relative spatial positions do not change.

[0050] For example, such as Figure 3 As shown, to ensure the accuracy of depth camera acquisition and the versatility of event camera applications, this embodiment of the application can use an Azure Kinect depth camera and a DAVIS346 event camera as data acquisition sensors. In use, the Azure Kinect is set to output a depth map at 30 FPS with a resolution of 1280*720. The depth map data will be transformed to the event camera coordinates through the spatiotemporal registration process described below (details omitted here). The DAVIS346 is then configured to output image frames and event streams at 25 FPS with a resolution of 346*260.

[0051] In step S102, temporal and spatial registration is performed on the depth camera and the event camera, and the image acquired by the depth camera is mapped onto the image acquired by the event camera to obtain a temporally and spatially aligned image frame dataset. For each frame of the image acquired by the event camera, the real depth map with a time interval less than a preset interval is selected as the real depth label.

[0052] The preset interval time can be set according to the actual situation, such as 0.02s, and there is no specific limitation on it.

[0053] It is understood that the embodiments of this application can register the depth camera and the event camera in both time and space to ensure that the obtained depth camera images and event camera images are registered in both time and space; after obtaining the registered image frame data, a dataset of image frames that are aligned in both time and space is mapped. Specifically, the embodiments of this application can use at least one possible method to achieve the time registration of the depth camera and the event camera, and can use at least one possible method to achieve the spatial registration of the depth camera and the event camera, without specific limitations; the specific method of the embodiments of this application for the time-space registration of the depth camera and the event camera and obtaining the dataset is as follows:

[0054] (1) Spatiotemporal registration of depth camera and event camera:

[0055] First, the embodiments of this application can match the time of the depth camera and the event camera: when shooting indoor scenes, the movement speed is low. Therefore, as a possible implementation, for each frame of the event camera image, the embodiment of this application can select the depth map in the depth camera data that is closest to its time as its true depth label to construct a dataset, which reduces costs and still ensures the accuracy of registration.

[0056] For example, in this embodiment of the application, the above-mentioned time soft alignment method can be used to search for the closest time depth camera data for each event camera image frame as a depth label, and the maximum time difference is set to 0.02s, that is, the preset interval is 0.02s. When the time difference between the closest depth data and the image frame is greater than 0.02s, it can be considered that there is no suitable time alignment data at this time, and thus the frame can be ignored.

[0057] Secondly, the embodiments of this application can match the space of the depth camera and the event camera: as one possible implementation, the embodiments of this application can use a Zhang Zhengyou checkerboard calibration method to complete the spatial registration of the depth camera and the event camera, wherein the checkerboard size is different to reduce the registration error within all reasonable depth ranges; after the calibration data is collected from the checkerboard, the embodiments of this application can use these data to optimize and obtain the final calibration parameters.

[0058] For example, when performing multi-grid calibration, embodiments of this application may use the grid shown in Table 1 to complete the calibration of the depth camera and the event camera; wherein, Table 1 is a grid size table for spatial registration.

[0059] Table 1

[0060] chessboard numbering Number of lines Number of columns Grid size 1 9 6 2.50cm 2 9 6 3.60cm 3 10 8 7.95cm

[0061] (2) Obtain a spatiotemporally aligned image frame dataset:

[0062] In this embodiment, a method can be used to map images acquired by a depth camera onto images acquired by an event camera to obtain a spatiotemporally aligned image frame dataset. This embodiment can acquire the spatiotemporally aligned image frame data using at least one method, such as using a ROS (Robot Operating System) package; no specific limitation is made. Examples of datasets in this embodiment include... Figure 4 As shown, the left side of both images in the figure is an image frame using the aforementioned DAVIS346, and the right side of both images is the corresponding depth map obtained after registration. The two can be aligned well, and the spatiotemporal registration result of the embodiment of this application is accurate.

[0063] It should be noted that the embodiments of this application can also clean and segment the dataset to ensure the cleanliness of the depth data. The embodiments of this application can use a simple median filter to remove noise from the depth data. After cleaning, the embodiments of this application can segment the dataset into a training set and a test set.

[0064] In step S103, a depth estimation network is trained using the image frame dataset to obtain the trained depth estimation network. The indoor event stream image and image sequence are input into the trained depth estimation network to output the indoor depth estimation result.

[0065] It is understood that, in this embodiment of the application, the depth estimation network can first be trained using the image frame dataset obtained in step S102, and then the indoor event stream image and image sequence obtained in step S101 can be input into the trained depth estimation network, thereby outputting the indoor depth estimation result. This embodiment of the application can use at least one method to train the depth estimation network, and is not specifically limited thereto. The specific steps of training the depth estimation network using the image frame dataset to obtain the trained depth estimation network in this embodiment of the application are as follows:

[0066] In this embodiment, the depth estimation network includes a convolutional encoder network, a feature temporal fusion network, and a decoder network. The convolutional encoder network takes event stream images and image sequences as input and outputs joint loss voxel features. The feature temporal fusion network is used to perform temporal fusion on the joint loss voxel features to obtain fused features. The decoder network is used to perform depth estimation based on the fused features to obtain the depth estimation result.

[0067] It is understood that the embodiments of this application can use a depth estimation network comprising a convolutional encoder network, a feature temporal fusion network, and a decoder network to achieve depth estimation for indoor spaces. The depth estimation network can be constructed in at least one manner. For example, the embodiments of this application can use a PyTorch framework to implement the network and modify it based on RestNet (Residual Network) as the backbone network to obtain a PARNet depth estimation network comprising the above three parts, etc., without specific limitations. The convolutional encoder network and feature temporal fusion network of the embodiments of this application will be described below, wherein the overall network structure can be as follows: Figure 5 As shown, the image sequence I = {I1, ..., I...} N} and the segmented event stream image S = {S1, ..., S2} N Given the input}, estimate the corresponding depth map. Specifically as follows:

[0068] (1) Convolutional encoder network:

[0069] In this embodiment, the convolutional encoder network is further configured to: extract first to fourth deep features from adjacent event stream images and adjacent image frames in the image sequence, respectively; discretize multiple depth planes with calculated loss errors from a preset depth range; extract first and second features corresponding to the second and fourth deep features, respectively; calculate the first voxel loss of the event stream image based on the first deep features and the first features; calculate the second voxel loss of the image frame based on the third and second deep features; calculate the loss voxel of the depth plane based on the first and second voxel losses; and concatenate the loss voxels of each depth plane to obtain the joint loss voxel features of the event stream image and the image sequence.

[0070] It is understood that the embodiments of this application can discretize multiple depth planes for calculating loss errors to obtain corresponding feature values, and calculate the loss voxels of the depth planes to obtain the joint loss voxel features of the event stream image and the image sequence. Therefore, the embodiments of this application can deeply consider the explicit spatial relationships existing in the depth estimation task, and better extract the spatiotemporal information contained in the event camera and depth camera data. The preset depth range can be set according to actual conditions, and is not specifically limited thereto.

[0071] Specifically, embodiments of this application can establish a joint loss voxel representation based on multi-view geometry, which calculates loss errors at different depth ranges between neighboring frames according to multi-view geometric relationships; embodiments of this application can simultaneously calculate loss errors using event streams and image frame features to obtain a joint loss voxel representation: when calculating loss voxels, neighboring frames I are required. t I t+1 The relative camera pose T between t，t+1 Using a pose estimation network θ pose Predicting T t，t+1 When constructing the joint loss voxel representation, embodiments of this application can first utilize a convolutional neural network to extract data from the event image S. t S t+1 With image frame I t I t+1 Extracting deep features and and from the preset depth range {d min d max Discretization N D Depth plane of the loss error to be calculated For each d i The embodiments of this application can utilize T t，t+1 Extract corresponding features from neighboring frames Then, the loss voxel corresponding to the depth plane is constructed as follows:

[0072]

[0073]

[0074]

[0075] That is, perform pixel-level inner product to obtain the individual loss errors, then add them together, and finally perform joint loss voxel representation V. t ={V t，1 , ..., V t，N}For each d i The lost voxels are spliced ​​together, and this joint representation, together with the original representations of each modality, constitutes the final feature F. t Together they serve as subsequent input.

[0076] It should be noted that, in the embodiments of this application, the true depth D, camera intrinsic parameters K, and image frame data I can also be used to construct the reprojection error for supervising pose; and the scale-invariant loss L can be used. s As the main loss function, gradient loss L is used. g and structural line loss L l As a regularization loss function.

[0077] Specifically, for two frames I t I t+n The network outputs their relative poses T. t，t+n Combined with true depth D t Based on the geometric relationships of multiple views, embodiments of this application can make I t+n Image projected onto I t On the frame, that is

[0078]

[0079] Where Π represents the reprojection process, through comparison with I t The consistency is used to construct the final error.

[0080]

[0081] Therefore, embodiments of this application can use SSIM consistency and L1 loss function to measure... with I t Consistency.

[0082] In this embodiment, the loss function of the convolutional encoder network is:

[0083] L depth =w1L s +w2L g +w3L l

[0084] Among them, L l For the structural line loss, N is extracted using event streams and image frames. l A line is formed, and multiple points are randomly selected on each line to construct the structure line loss L. l Gradient loss L g To ensure gradient consistency between the predicted depth map and the true depth map; w1, w2, and w3 are 1.0, 0.5, and 0.01, respectively; scale-invariant loss L s The calculation method is as follows:

[0085]

[0086] Here, Mean represents the averaging operation. and D p λ represents the predicted depth and the true depth of pixel p, respectively. λ controls the influence of scale ambiguity on the loss function and can be set according to the actual situation, such as 0.5, etc. There is no specific limitation on it.

[0087] Gradient loss L g To ensure gradient consistency between the predicted depth map and the true depth map, a structure line loss L is constructed. l In this embodiment, N can be extracted first using event streams and image frames. l Draw a line, and then randomly select three points A, B, and C on each line. Construct an L using these three points. l ,Right now

[0088] L l =∑|A i B i ×A i C i |

[0089] (2) Feature-time fusion network

[0090] It is understood that the joint loss voxel representation contains spatiotemporal information from neighboring frames, but it does not have the ability to fuse long-sequence temporal information. Therefore, embodiments of this application can use a feature temporal fusion network to enable the joint loss voxel features to contain long-sequence temporal information. Specifically:

[0091] In this embodiment, the feature temporal fusion network is further used to: update the latent features of the current frame based on the joint loss voxel features of the current frame and the latent features of the previous frame, such that the joint loss voxel features contain long-sequence temporal information, wherein the update process includes:

[0092] z t =σ(W z *Concat(F t Ht-1 ))

[0093] r t =σ(W r *Concat(F t H t-1 ))

[0094]

[0095]

[0096] Among them, W * F represents the network weights, Concat represents the feature concatenation operation, σ represents the non-linear activation function, and F... * H represents the features extracted by the pre-network module. * The hidden z-axis represents the maintenance and update of the recurrent network. * This indicates updating the gate weights, which are determined by F. * With H * It is calculated. The feature temporal fusion network of this application embodiment can, during each temporal update, use z... t r t Two gates are used to determine the information that needs to be updated and forgotten, thereby effectively extracting temporal features and providing more effective information for depth prediction of the current frame.

[0097] The indoor depth estimation method based on the joint analysis of event streams and image frames of this application will be illustrated below through a specific embodiment:

[0098] 1. In this embodiment of the application, an event camera and a depth camera can be installed as sensors. In this embodiment of the application, an Azure Kinect depth camera and a DAVIS346 event camera can be selected as data acquisition sensors. When using them, the Azure Kinect can be set to output a depth map with a resolution of 1280*720 at 30 FPS, and the DAVIS346 can be configured to output image frames and event streams with a resolution of 346*260 at 25 FPS. Figure 3 The data acquisition equipment has been installed.

[0099] 2. This application embodiment can perform sensor temporal registration and spatial registration. This application embodiment can use a temporal soft alignment method to search for the closest temporally related depth camera data for each event camera image frame as its depth label, and set a maximum time difference. When the time difference between the closest depth data and the image frame is greater than 0.02 seconds, this application embodiment can consider that there is no suitable temporal alignment data and will ignore the frame; thus, the temporal registration ends, and this application embodiment then performs sensor spatial registration; when performing multi-chessboard calibration, this application embodiment can select three chessboards, and the size of each chessboard is shown in Table 1 above.

[0100] 3. This application embodiment can perform data acquisition, and clean and segment the data. This application embodiment acquired 131 indoor scene sequences, each sequence containing an aligned event stream, image frames, and a depth map. Figure 4 Several sample data points were presented. Through analysis and practical experiments, this embodiment of the application can use simple median filtering to remove noise in the depth data and filter out the following types of dirty data: Frames with excessive depth loss can be filtered out; due to the imaging principle of the depth camera, frames with excessive depth loss will always have some depth loss; Sequences where the depth map and image frame are misaligned can be directly removed; during the acquisition process, due to the instability of the depth camera, the depth map may sometimes be delayed in output, resulting in the depth map and image frame not being perfectly aligned; Error frames, where some frames' data is corrupted during transmission and cannot be used. After cleaning, this embodiment of the application can divide the dataset into training and test sets for subsequent evaluation by various algorithms. After data cleaning, this embodiment of the application retains 111 valid sequences, of which 20 sequences are used as the test set, and the remaining 91 sequences are used as the training set.

[0101] 4. In this embodiment, the PyTorch framework can be used to modify the ResNet18 backbone network to extract features from the event stream and image frames respectively.

[0102] (1) Joint loss voxel characterization

[0103] like Figure 3 As shown, in order to balance the computational cost and performance improvement brought by voxel representation, this embodiment of the application can use the feature maps extracted by the second layer of ResNet18 to construct voxels. When constructing voxels, this embodiment of the application can set d min =0.25m, d max =10.0m, N D =32, meaning 32 candidate depth planes are selected for error calculation. Voxel feature V t will be respectively with The fused features are obtained by splicing them together and used as input for subsequent ResNet18 network layers.

[0104] (2) Time-series cyclical update

[0105] The encoder portion of this application provides multi-scale feature map output. When performing time-series cyclic updates, in order to balance computational cost and performance improvement, this application embodiment can simultaneously apply the update at a low resolution and a high resolution to ensure that time-series information at different scales is preserved.

[0106] (3) Loss function and training settings

[0107] This application embodiment can use self-collected datasets for training and testing. During training, PoseNet is first trained to output reasonable pose estimation results. The Adam optimizer is used for 15 training epochs to optimize the loss function L. pose After convergence, PoseNet weights are fixed, and DepthNet (a deep network) is trained for 25 epochs using the Adam optimizer and a loss function L. depth =w1L s +w2L g +w3L l The weights of each term are w1 = 1.0, w2 = 0.5, and w3 = 0.001. During training, the learning rate in this embodiment can be uniformly set to 0.0001. The visualization comparison results of this embodiment are shown in the figure below. Figure 6 As shown.

[0108] Therefore, as shown in Table 2, the method of this application embodiment still has the best overall results under low light and motion blur input, and its predicted depth map is closer to the true depth and the details are more complete; wherein, Table 2 is a quantitative comparison result table.

[0109] Table 2

[0110]

[0111] In summary, the indoor depth estimation method based on the joint event stream and image frame proposed in this application can acquire indoor event stream images and image sequences, perform spatiotemporal registration, establish a depth estimation network including joint loss voxel representation and multimodal temporal information using a standard indoor dataset as an evaluation benchmark, complete the training of the depth estimation network, and obtain indoor depth estimation results using indoor event stream images and image sequences as input. Therefore, this application embodiment can effectively integrate the spatiotemporal information of neighboring frames of event stream image frames in an offline state, providing more effective information for indoor depth estimation tasks, completing depth estimation offline, making depth estimation more adaptable to complex indoor environments, processing more intelligently, enhancing the accuracy of depth estimation results, and meeting practical application needs.

[0112] Based on the offline depth estimation of the previous embodiment, this application provides another indoor depth estimation method based on the joint analysis of event streams and image frames, to illustrate how to perform online depth estimation. This embodiment and the previous embodiment have different focuses in their descriptions, and steps not fully described in each embodiment can be referred to mutually. The following describes, with reference to the accompanying drawings, another indoor depth estimation method based on the joint analysis of event streams and image frames proposed according to an embodiment of this application.

[0113] like Figure 7 As shown, this indoor depth estimation method based on the joint analysis of event streams and image frames includes the following steps:

[0114] In step S201, the indoor event stream image and image sequence are acquired.

[0115] In step S202, the indoor event stream image and image sequence are input into the trained depth estimation network, and the indoor depth estimation result is output. The depth estimation network includes a convolutional encoder network, a feature temporal fusion network, and a decoder network. The convolutional encoder network takes the event stream image and image sequence as input and outputs joint loss voxel features. The feature temporal fusion network is used to perform temporal fusion on the joint loss voxel features to obtain fused features. The decoder network is used to perform depth estimation based on the fused features to obtain the depth estimation result.

[0116] It is understood that, in the embodiments of this application, the obtained event stream image and image sequence can also be directly input into the trained depth estimation network. In this embodiment, the spatiotemporal information of neighboring frames can be integrated by joint loss voxel representation, and the multimodal temporal information of long sequences can be preserved by feature temporal fusion network.

[0117] The indoor depth estimation method based on the joint event stream and image frame proposed in this application can acquire indoor event stream images and image sequences. Using a standard indoor dataset as an evaluation benchmark, a depth estimation network including joint loss voxel representation and multimodal temporal information is established. The training of the depth estimation network is completed, and the indoor depth estimation result is obtained by taking the indoor event stream images and image sequences as input. Thus, this application embodiment can effectively integrate the spatiotemporal information of neighboring frames of event stream image frames in an online state, providing more effective information for the indoor depth estimation task, completing the depth estimation online, making the depth estimation more adaptable to complex indoor environments, processing more intelligently, enhancing the accuracy of the depth estimation results, and meeting the needs of practical use.

[0118] The following description, with reference to the accompanying drawings, describes an indoor depth estimation apparatus based on the combination of event stream and image frames according to an embodiment of this application.

[0119] Figure 8 This is a block diagram of an indoor depth estimation device based on the combination of event stream and image frames according to an embodiment of this application.

[0120] like Figure 8 As shown, the indoor depth estimation device 10 based on the combination of event stream and image frames includes: a camera module 110, a mapping module 120 and a training module 130.

[0121] The camera module 110 includes a fixed depth camera and an event camera. The depth camera is used to acquire indoor true depth maps, and the event camera is used to acquire indoor aligned event stream images and image sequences. The mapping module 120 is used to perform temporal and spatial registration between the depth camera and the event camera, mapping the images acquired by the depth camera onto the images acquired by the event camera to obtain a temporally and spatially aligned image frame dataset. For each frame of the image acquired by the event camera, the true depth map with a time interval less than a preset interval is selected as the true depth label. The training module 130 is used to train a depth estimation network using the image frame dataset to obtain the trained depth estimation network. The indoor event stream images and image sequences are input into the trained depth estimation network, and the indoor depth estimation results are output.

[0122] In this embodiment of the application, the training module 130 is further used for: the depth estimation network includes a convolutional encoder network, a feature temporal fusion network and a decoder network, wherein the convolutional encoder network takes event stream images and image sequences as input and outputs joint loss voxel features, the feature temporal fusion network is used to perform temporal fusion on the joint loss voxel features to obtain fused features, and the decoder network is used to perform depth estimation based on the fused features to obtain depth estimation results.

[0123] In this embodiment, the training module 130 is further configured to: extract first to fourth deep features from adjacent event stream images and adjacent image frames in the image sequence, respectively; discretize multiple depth planes with uncalculated loss errors from a preset depth range; extract first and second features corresponding to the second and fourth deep features respectively; calculate the first voxel loss of the event stream image based on the first deep features and the first features; calculate the second voxel loss of the image frame based on the third and second deep features; calculate the loss voxel of the depth plane based on the first and second voxel losses; and concatenate the loss voxels of each depth plane to obtain the joint loss voxel features of the event stream image and the image sequence.

[0124] In this embodiment, the loss function of the convolutional encoder network is:

[0125] L depth =w1L s +w2L g +w3L l

[0126] Among them, L l For the structural line loss, N is extracted using event streams and image frames. l A line is formed, and multiple points are randomly selected on each line to construct the structure line loss L. l Gradient loss L g To ensure gradient consistency between the predicted depth map and the true depth map; w1, w2, and w3 are 1.0, 0.5, and 0.01, respectively; scale-invariant loss L s The calculation method is as follows:

[0127]

[0128] Here, Mean represents the averaging operation. and D p Let represent the predicted depth and the true depth of pixel p, respectively, and let λ control the effect of scale ambiguity on the loss function.

[0129] In this embodiment, the training module 130 is further configured to: update the latent features of the current frame based on the joint loss voxel features of the current frame and the latent features of the previous frame, such that the joint loss voxel features contain temporal information of a long sequence, wherein the update process includes:

[0130] z t =σ(W z *Concat(F t H t-1 ))

[0131] r t =σ(W r *Concat(F tH t-1 ))

[0132]

[0133]

[0134] Among them, W * F represents the network weights, Concat represents the feature concatenation operation, σ represents the non-linear activation function, and F... * H represents the features extracted by the pre-network module. * The hidden z-axis represents the maintenance and update of the recurrent network. * This indicates updating the gate weights, which are determined by F. * With H * It is calculated.

[0135] It should be noted that the foregoing explanation of the indoor depth estimation method based on the combination of event stream and image frame also applies to the indoor depth estimation device based on the combination of event stream and image frame in this embodiment, and will not be repeated here.

[0136] The indoor depth estimation device based on the joint event stream and image frame proposed in this application can acquire indoor event stream images and image sequences, perform spatiotemporal registration, establish a depth estimation network including joint loss voxel representation and multimodal temporal information using a standard indoor dataset as an evaluation benchmark, complete the training of the depth estimation network, and obtain the indoor depth estimation result using indoor event stream images and image sequences as input. Thus, this application embodiment can effectively integrate the spatiotemporal information of neighboring frames of event stream image frames in an offline state, providing more effective information for the indoor depth estimation task, completing the depth estimation offline, making the depth estimation more adaptable to complex indoor environments, processing more intelligently, enhancing the accuracy of the depth estimation result, and meeting practical needs.

[0137] Another indoor depth estimation device based on the combination of event stream and image frames, according to an embodiment of this application, is described below with reference to the accompanying drawings.

[0138] Figure 9 This is a block diagram of an indoor depth estimation device based on the combination of event stream and image frames according to an embodiment of this application.

[0139] like Figure 9 As shown, the indoor depth estimation device 20 based on the combination of event stream and image frames includes: an acquisition module 210 and an output module 220.

[0140] The acquisition module 210 is used to acquire indoor event stream images and image sequences; the output module 220 is used to input the indoor event stream images and image sequences into the trained depth estimation network and output the indoor depth estimation result. The depth estimation network includes a convolutional encoder network, a feature temporal fusion network, and a decoder network. The convolutional encoder network takes the event stream images and image sequences as input and outputs joint loss voxel features. The feature temporal fusion network is used to perform temporal fusion on the joint loss voxel features to obtain fused features. The decoder network is used to perform depth estimation based on the fused features to obtain the depth estimation result.

[0141] It should be noted that the foregoing explanation of the indoor depth estimation method based on the combination of event stream and image frame also applies to the indoor depth estimation device based on the combination of event stream and image frame in this embodiment, and will not be repeated here.

[0142] The indoor depth estimation device based on the joint event stream and image frame proposed in this application can acquire indoor event stream images and image sequences. Using a standard indoor dataset as an evaluation benchmark, a depth estimation network including joint loss voxel representation and multimodal temporal information is established. The training of the depth estimation network is completed, and the indoor depth estimation result is obtained by taking the indoor event stream images and image sequences as input. Thus, this application embodiment can effectively integrate the spatiotemporal information of neighboring frames of event stream image frames in an online state, providing more effective information for the indoor depth estimation task, completing the depth estimation online, making the depth estimation more adaptable to complex indoor environments, processing more intelligently, enhancing the accuracy of the depth estimation results, and meeting practical needs.

[0143] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0144] The memory 1001, the processor 1002, and the computer program stored on the memory 1001 and capable of running on the processor 1002.

[0145] When the processor 1002 executes the program, it implements the indoor depth estimation method based on the combination of event stream and image frame provided in the above embodiments.

[0146] Furthermore, electronic devices also include:

[0147] Communication interface 1003 is used for communication between memory 1001 and processor 1002.

[0148] The memory 1001 is used to store computer programs that can run on the processor 1002.

[0149] The memory 1001 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0150] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, then the communication interface 1003, memory 1001, and processor 1002 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0151] Optionally, in a specific implementation, if the memory 1001, processor 1002, and communication interface 1003 are integrated on a single chip, then the memory 1001, processor 1002, and communication interface 1003 can communicate with each other through an internal interface.

[0152] The processor 1002 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0153] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described indoor depth estimation method based on the combination of event streams and image frames.

[0154] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0155] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0156] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0157] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0158] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

Claims

1. An indoor depth estimation method based on the joint analysis of event streams and image frames, characterized in that, Includes the following steps: A fixed depth camera and an event camera, wherein the depth camera is used to acquire a true depth map of the room, and the event camera is used to acquire aligned event stream images and image sequences of the room; Temporal and spatial registration are performed on the depth camera and the event camera to map the image acquired by the depth camera onto the image acquired by the event camera, thereby obtaining a temporally and spatially aligned image frame dataset. For each frame of the image acquired by the event camera, a real depth map with a time interval less than a preset interval is selected as the real depth label. A depth estimation network is trained using the image frame dataset to obtain a trained depth estimation network, which includes a convolutional encoder network, a feature temporal fusion network, and a decoder network. Indoor event stream images and image sequences are input into the trained depth estimation network, which outputs the indoor depth estimation result. Specifically, the convolutional encoder network takes the event stream images and image sequences as input and outputs joint loss voxel features. The feature temporal fusion network performs temporal fusion on the joint loss voxel features to obtain fused features. The decoder network performs depth estimation based on the fused features to obtain the depth estimation result. The convolutional encoder network is further configured to: extract first to fourth deep features from adjacent event stream images and adjacent image frames in the image sequence, respectively; discretize multiple depth planes with calculated loss errors from a preset depth range; extract first and second features corresponding to the second and fourth deep features, respectively; calculate the first voxel loss of the event stream image based on the first deep feature and the first feature; calculate the second voxel loss of the image frame based on the third deep feature and the second feature; calculate the loss voxel of the depth plane based on the first voxel loss and the second voxel loss; and concatenate the loss voxels of each depth plane to obtain the joint loss voxel features of the event stream image and the image sequence.

2. The indoor depth estimation method based on the joint analysis of event stream and image frames according to claim 1, characterized in that, The loss function of the convolutional encoder network is: in, For structural line loss, event streams and image frames are used for extraction. A line is formed, and multiple points are randomly selected on each line to construct the structure line loss. Gradient loss To ensure gradient consistency between the predicted depth map and the true depth map, , , The values ​​are 1.0, 0.5, and 0.01 respectively; scale-invariant loss. The calculation method is as follows: in, This indicates an averaging operation. and Representing pixels The predicted depth and the actual depth, The impact of scale ambiguity on the loss function.

3. An indoor depth estimation method based on the joint analysis of event streams and image frames, characterized in that, Includes the following steps: Acquire indoor event stream images and image sequences; An indoor event stream image and image sequence are input into a trained depth estimation network, which outputs the depth estimation result of the indoor space. The depth estimation network includes a convolutional encoder network, a feature temporal fusion network, and a decoder network. The convolutional encoder network takes the event stream image and the image sequence as input and outputs joint loss voxel features. The feature temporal fusion network is used to perform temporal fusion on the joint loss voxel features to obtain fused features. The decoder network is used to perform depth estimation based on the fused features to obtain the depth estimation result. The convolutional encoder network is further configured to: extract first to fourth deep features from adjacent event stream images and adjacent image frames in the image sequence, respectively; discretize multiple depth planes with calculated loss errors from a preset depth range; extract first and second features corresponding to the second and fourth deep features, respectively; calculate the first voxel loss of the event stream image based on the first deep feature and the first feature; calculate the second voxel loss of the image frame based on the third deep feature and the second feature; calculate the loss voxel of the depth plane based on the first voxel loss and the second voxel loss; and concatenate the loss voxels of each depth plane to obtain the joint loss voxel features of the event stream image and the image sequence.

4. An indoor depth estimation device based on the combination of event stream and image frames, characterized in that, include: The camera module includes a fixed depth camera and an event camera, wherein the depth camera is used to acquire a true depth map of the room, and the event camera is used to acquire aligned event stream images and image sequences of the room. The mapping module is used to perform temporal and spatial registration between the depth camera and the event camera, mapping the images acquired by the depth camera onto the images acquired by the event camera to obtain a temporally and spatially aligned image frame dataset. For each frame of the image acquired by the event camera, a real depth map with a time interval less than a preset interval is selected as the real depth label. The training module is used to train a depth estimation network using the image frame dataset to obtain a trained depth estimation network. The depth estimation network includes a convolutional encoder network, a feature temporal fusion network, and a decoder network. Indoor event stream images and image sequences are input into the trained depth estimation network, and the depth estimation result of the indoor environment is output. Specifically, the convolutional encoder network takes the event stream images and image sequences as input and outputs joint loss voxel features. The feature temporal fusion network is used to perform temporal fusion on the joint loss voxel features to obtain fused features. The decoder network is used to perform depth estimation based on the fused features to obtain the depth estimation result. The convolutional encoder network is further configured to: extract first to fourth deep features from adjacent event stream images and adjacent image frames in the image sequence, respectively; discretize multiple depth planes with calculated loss errors from a preset depth range; extract first and second features corresponding to the second and fourth deep features, respectively; calculate the first voxel loss of the event stream image based on the first deep feature and the first feature; calculate the second voxel loss of the image frame based on the third deep feature and the second feature; calculate the loss voxel of the depth plane based on the first voxel loss and the second voxel loss; and concatenate the loss voxels of each depth plane to obtain the joint loss voxel features of the event stream image and the image sequence.

5. An indoor depth estimation device based on the combination of event stream and image frames, characterized in that, include: The acquisition module is used to acquire indoor event stream images and image sequences; The output module is used to input indoor event stream images and image sequences into a trained depth estimation network and output the depth estimation result of the indoor environment. The depth estimation network includes a convolutional encoder network, a feature temporal fusion network, and a decoder network. The convolutional encoder network takes the event stream images and image sequences as input and outputs joint loss voxel features. The feature temporal fusion network is used to perform temporal fusion on the joint loss voxel features to obtain fused features. The decoder network is used to perform depth estimation based on the fused features to obtain the depth estimation result. The convolutional encoder network is further configured to: extract first to fourth deep features from adjacent event stream images and adjacent image frames in the image sequence, respectively; discretize multiple depth planes with calculated loss errors from a preset depth range; extract first and second features corresponding to the second and fourth deep features, respectively; calculate the first voxel loss of the event stream image based on the first deep feature and the first feature; calculate the second voxel loss of the image frame based on the third deep feature and the second feature; calculate the loss voxel of the depth plane based on the first voxel loss and the second voxel loss; and concatenate the loss voxels of each depth plane to obtain the joint loss voxel features of the event stream image and the image sequence.

6. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the indoor depth estimation method based on the combination of event stream and image frames as described in any one of claims 1-3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the indoor depth estimation method based on the combination of event stream and image frames as described in any one of claims 1-3.

Images