Monocular depth estimation method and device for pulse camera

Abstract

The present application provides a monocular depth estimation method and device for a pulse camera, the method comprising: obtaining an unstructured continuous pulse stream output by a pulse camera; inputting the continuous pulse stream into a preset monocular depth estimation model, so that the monocular depth estimation model mines spatial and temporal features from the continuous pulse stream, and outputs a depth estimation result corresponding to the continuous pulse stream. The present application can mine spatial and temporal features from the continuous pulse stream output by a pulse camera at the same time, can effectively improve the reliability of the monocular depth estimation prediction process for a pulse camera, and can improve the accuracy and effectiveness of the depth estimation results, thereby further improving the reliability and intelligence of the application of pulse cameras.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a monocular depth estimation method and device for a pulse camera. Background Art

[0002] Conventional frame-based cameras operate at a fixed rate, providing a stroboscopically synchronized sequence of images via snapshots. The concept of exposure time windows in frame-based cameras limits their use in some challenging scenarios, such as high-speed scenes and high dynamic range environments, resulting in motion blur or over / underexposure. Pulse cameras are biomimetic visual sensors with a completely different sensing approach compared to conventional cameras. Instead of capturing visual signals in the exposure interval via snapshots, each pixel on the pulse camera sensor captures incoming photons independently and continuously, and triggers a pulse only when the accumulated photons reach a certain threshold. Therefore, pulse cameras can produce a continuous stream of pulses with very high temporal resolution. The recorded spatiotemporal pulse stream can be used to reconstruct dynamic scenes at any given moment. Unlike event cameras (also known as dynamic vision sensors) that only record the relative brightness changes of each pixel, pulse cameras record absolute light intensity, providing both static and dynamic scene information. Thanks to their superior performance such as full-time imaging and free dynamic range, pulse cameras have great potential in fields such as autonomous driving, drones, and robots.

[0003] Depth estimation is a fundamental task in computer vision. State-of-the-art depth estimation works focus more on standard frame-based cameras. Recently, significant progress has been made in depth estimation based on event cameras. However, there is no research on monocular depth estimation for pulse cameras. Existing depth estimation models that only record traditional images of static scenes are not applicable to the pulse streams generated by pulse cameras due to different sampling mechanisms and cannot simultaneously mine spatial and temporal features from the continuous pulse stream. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a monocular depth estimation method, device, electronic device and computer-readable storage medium for a pulse camera to eliminate or improve one or more defects existing in the prior art.

[0005] The first aspect of the present application provides a monocular depth estimation method for a pulse camera, comprising:

[0006] Get the unstructured continuous pulse stream output by the pulse camera;

[0007] The continuous pulse stream is input into a preset monocular depth estimation model, so that the monocular depth estimation model mines spatial and temporal features from the continuous pulse stream and outputs a depth estimation result corresponding to the continuous pulse stream.

[0008] In some embodiments of the present application, it also includes:

[0009] A preset spatiotemporal Transformer network model is trained using a training data set to obtain a monocular depth estimation model for mining spatial and temporal features from a continuous pulse stream in the training data set and outputting a depth estimation result corresponding to the continuous pulse stream.

[0010] In some embodiments of the present application, the training data set is a synthetic pulse data set including a pulse stream and corresponding real depth data.

[0011] In some embodiments of the present application, it also includes:

[0012] Acquire a real pulse sequence containing various scenes collected in advance by the pulse camera to form a corresponding real data set;

[0013] The real data set is used to verify the generalization of the monocular depth estimation model obtained by training.

[0014] In some embodiments of the present application, the monocular depth estimation model includes: a pulse embedding module and a Transformer network model;

[0015] The pulse embedding module is used to extract the time and space characteristics of the pulse stream to obtain corresponding multiple pulse feature embeddings;

[0016] The Transformer network model is used to obtain the spatiotemporal features of each pulse embedding and perform feature fusion on each spatiotemporal feature to predict the depth estimation result data of the pulse stream.

[0017] In some embodiments of the present application, the Transformer network model includes: a spatiotemporal Transformer encoder and a convolutional decoder;

[0018] The spatiotemporal Transformer encoder is used to extract spatiotemporal features of each pulse embedding through a multi-layer self-attention mechanism to obtain the spatiotemporal features of each pulse embedding;

[0019] The convolution decoder is used to perform feature fusion and up-sampling on spatiotemporal features of different scales to predict the depth estimation result data of the pulse stream.

[0020] In some embodiments of the present application, the pulse embedding module includes: a time partition, a feature extraction layer, a space partition and a feature merging layer, wherein the time partition is provided with a multi-scale time window;

[0021] Correspondingly, the multi-scale time window is used to divide the pulse stream input into the pulse embedding module into a plurality of non-overlapping blocks;

[0022] The feature extraction layer is used to extract feature data of each non-overlapping block, and the feature data includes: local features and global features;

[0023] The spatial partition is used to split the feature data in the spatial domain to obtain spatiotemporal feature data;

[0024] The feature merging layer is used to merge the global features and local features in the spatiotemporal feature data to obtain multiple spatiotemporal blocks corresponding to the pulse stream, and determine each of the spatiotemporal blocks as a pulse embedding to be input into the Transformer network model.

[0025] In some embodiments of the present application, the spatiotemporal Transformer encoder includes multiple stages, and each stage includes multiple spatiotemporal Transformer blocks, and a residual connection layer is provided between every two adjacent stages;

[0026] The spatiotemporal Transformer block includes: a multi-head self-attention MSA module equipped with a 3D shift window, and a feed-forward neural network FFN composed of multiple MLP layers;

[0027] A GELU layer is provided between each of the MSA modules and the FFN, and a normalization layer LN is provided before each of the MSA modules and the FFN.

[0028] In some embodiments of the present application, the convolutional decoder includes: multiple residual blocks, multiple feature fusion layers and a prediction head.

[0029] The second aspect of the present application also provides a monocular depth estimation device for a pulse camera, comprising:

[0030] A data acquisition module, used for acquiring an unstructured continuous pulse stream output by a pulse camera;

[0031] The depth estimation module is used to input the continuous pulse stream into a preset monocular depth estimation model so that the monocular depth estimation model mines spatial and temporal features from the continuous pulse stream and outputs a target depth estimation result corresponding to the continuous pulse stream.

[0032] The third aspect of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the monocular depth estimation method for a pulse camera when executing the computer program.

[0033] The third aspect of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the monocular depth estimation method for a pulse camera is implemented.

[0034] The monocular depth estimation method for a pulse camera provided in the present application obtains an unstructured continuous pulse stream output by the pulse camera; the continuous pulse stream is input into a preset monocular depth estimation model, so that the monocular depth estimation model mines spatial and temporal features from the continuous pulse stream, and outputs depth estimation result data corresponding to the continuous pulse stream; by designing a monocular depth estimation model for mining spatial and temporal features from the continuous pulse stream and outputting the depth estimation result corresponding to the continuous pulse stream, the problem that there is currently no standard network that can mine spatial and temporal features from a continuous pulse stream at the same time can be effectively solved, the reliability of the monocular depth estimation prediction process for the pulse camera can be effectively improved, and the accuracy and effectiveness of the depth estimation result can be improved, thereby further improving the application reliability and intelligence of the pulse camera.

[0035] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0036] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0038] Figure 1 It is a flow chart of the monocular depth estimation method for a pulse camera provided in this application.

[0039] Figure 2 It is a flowchart of the monocular depth estimation model training provided in this application.

[0040] Figure 3 It is a structural diagram of the monocular depth estimation model provided in this application.

[0041] Figure 4 It is a schematic diagram of the specific model framework of the monocular depth estimation model provided in this application.

[0042] Figure 5 It is a schematic diagram of the structure of a monocular depth estimation device for a pulse camera provided in this application.

[0043] Figure 6 This is a schematic diagram of the framework of the space-time transformation SST encoder provided in this application.

[0044] Figure 7-1 It is the comparison result of the absolute relative error (Abs Rel. Difference) under different distance ranges provided by this application; among them, the three curves from bottom to top represent Spike-T, E2Depth and U-Net of the application example of this application respectively.

[0045] Figure 7-2 It is the comparison result of the mean depth error (Mean Depth Error) under different clipping distances provided by this application; among them, the three curves from bottom to top represent Spike-T, E2Depth and U-Net of the application example of this application respectively.

[0046] Figure 8 This is the visualization result of the synthetic dataset “DENSE-spike” provided by this application.

[0047] Fig. 9 The visualization results of the real data set provided by this application (from "Outdoor-spike") are shown in Figure 2. (a) The pulse frame at the predicted timestamp. (b) The reconstructed image. (c) The predicted depth map of Spike-T using U-Net, E2Depth and the application example of this application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0049] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0050] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0051] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0052] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0053] Since the current depth estimation for pulse cameras lacks a unified pulse data architecture, that is, in binary and irregular data structures, continuous pulse streams capture dynamic scenes with very high temporal resolution, and there is currently no standard network that can simultaneously mine spatial and temporal features from continuous pulse streams. Therefore, the present application provides an embodiment of a monocular depth estimation method for pulse cameras, by designing a monocular depth estimation model for mining spatial and temporal features from the continuous pulse stream and outputting depth estimation result data corresponding to the continuous pulse stream, it can effectively solve the problem that there is currently no standard network that can simultaneously mine spatial and temporal features from a continuous pulse stream, can effectively improve the reliability of the monocular depth estimation prediction process for pulse cameras, and can improve the accuracy and effectiveness of the depth estimation results, thereby further improving the reliability and intelligence of the application of pulse cameras.

[0054] In one or more embodiments of the present application, a spike camera is a biomimetic visual sensor that shows great potential in capturing high-speed dynamic scenes at a sampling rate of 40,000 Hz. Unlike traditional digital cameras, spike cameras continuously capture photons and output asynchronous binary pulses that encode time, position, and light intensity. Due to the different sampling mechanisms, off-the-shelf image-based digital camera algorithms are not applicable to the pulse streams generated by spike cameras. Therefore, it is particularly important to develop novel pulse perception algorithms for common computer vision tasks. In this application, we focus on the task of depth estimation, which is challenging due to the natural properties of the pulse stream, such as irregularity, continuity, and spatiotemporal correlation, and has not yet been explored for spike cameras.

[0055] Understandably, the pulse camera, also known as the Vidar camera, is a biomimetic visual sensor that mimics the sampling mechanism of the fovea of ​​the primate retina, 1,000 times faster than its traditional frame-based counterparts. Due to its unique working principle, the spiked camera can theoretically record the texture of the scene continuously. Given its great potential in many applications such as traffic monitoring and suspect identification, there have been studies on pulse-based visual tasks. Some researchers have proposed an efficient pulse camera encoding method by calculating the time interval of the pulse; others have constructed a three-layer pulse neural network SNN to reconstruct high-quality visual images of natural scenes; others have improved the smoothness of the reconstructed image by using optical flow to align scenes at different moments; and others have introduced an image reconstruction model through the brain's short-term plasticity STP mechanism. Some researchers have also constructed a hierarchical CNN architecture to reconstruct dynamic scenes, gradually exploiting the temporal correlation of the pulse stream. At present, some people have proposed a deep learning pipeline to estimate the optical flow from a continuous pulse stream, in which the predicted optical flow can reduce motion blur. Previous work has made significant progress in the development of pulse cameras. However, one of the basic visual tasks, depth estimation, has not been fully studied. Therefore, the improvement focus of this application is how to effectively estimate the depth of the scene from a continuous pulse stream.

[0056] In addition, the specific descriptions of image-based and event-based monocular depth estimation are as follows:

[0057] Image-based monocular depth estimation aims to generate dense depth maps containing 3D structural information from single-view images. Early image-based depth estimation works were mainly based on hand-crafted features related to image depth cues, such as texture density and object size. In recent years, deep learning-based depth estimation models have gained attention. They usually utilize encoder-decoder architectures with skip connections to learn depth-related priors directly from training data, achieving impressive depth estimation performance compared to their hand-crafted counterparts.

[0058] Recently, event-based monocular depth estimation has attracted increasing attention due to its unique characteristics, especially for low-latency obstacle avoidance and fast path planning. Researchers have developed a unified contrast maximization framework to solve several event-based vision problems, such as depth estimation and optical flow estimation, which can find point trajectories that are best aligned with event-based data on the image plane. Others have proposed appropriate event representations in the form of discrete volumes and used encoder-decoder mechanisms to integrate multiple clues from event streams. In the existing approach, recurrent convolutional neural networks can also be used to learn monocular depth by exploiting the temporal consistency presented in the event stream. Other scholars have proposed a recursive asynchronous multimodal network that estimates monocular depth by combining events and frames. The network generalizes traditional RNNs to learn asynchronous event-based data from multiple sensors. In this application, unlike event-based cameras that focus more on motion edges, pulse cameras can capture both stationary and moving objects. Therefore, this application needs to study pulse-based vision problems in a different way from its event-based counterparts.

[0059] The details are described in detail through the following embodiments.

[0060] (I) Embodiment of the monocular depth estimation method for pulse camera

[0061] The present invention provides a monocular depth estimation method for a pulse camera, see Figure 1 , the monocular depth estimation method for pulse camera specifically includes the following contents:

[0062] Step 100: Acquire an unstructured continuous pulse stream output by a pulse camera.

[0063] It can be understood that, inspired by the sampling mechanism of the fovea of ​​the primate retina, the pulse camera records intensity information with spatiotemporal characteristics. It outputs a binary stream in a pulse format, with only 0 or 1 representing the data. The pulse camera mainly consists of three components, photoreceptors, batteries, and comparators. Specifically, the photosensitive pixel array is spatially arranged on the photoreceptor of the pulse camera to continuously capture photons. Secondly, the battery continuously converts the light signal into an electrical signal to increase the voltage of each unit. The comparator detects whether the accumulated voltage reaches the scheduling voltage threshold θ. When the threshold is reached, a pulse is triggered and the voltage is reset to the preset value. To describe the pulse generation mechanism, the process on a pixel can be expressed as formula (1):

[0064]

[0065] where I(t) describes the light intensity, t i and t i-1respectively represent the trigger times of the i-th and (i-1)-th pulses. α is the photoelectric conversion rate. Due to the limitations of circuit technology, the units in the output circuit periodically read out the pulses as discrete-time signals s(x, y, n) within a fixed interval Δt = 25 us. If the pixel at the spatial coordinates (x, y) triggers a pulse at time t, the read-out pulse s(x, y, n) = 1 (n = 1, 2,...), where (n - 1)Δt < t ≤ nΔt. Otherwise, s(x, y, n) = 0 is read. The sensor uses high-speed polling to generate a pulse frame of size H×W at each discrete time stamp n. Within a fixed time interval Δt·T, the camera generates a binary pulse stream with a size of H×W×T.

[0066] Step 200: Input the continuous pulse stream into a preset monocular depth estimation model, so that the monocular depth estimation model mines spatial and temporal features from the continuous pulse stream and outputs the depth estimation result corresponding to the continuous pulse stream.

[0067] In step 200, the preset monocular depth estimation model can be a monocular depth estimation model trained or updated before step 100, or a monocular depth estimation model directly retrieved by the device performing steps 100 and 200 from other servers or databases, etc. For example, the online model application process and the offline model training process are carried out on different servers, and specific settings can be made according to the actual application situation.

[0068] It can be understood that the monocular depth estimation model mentioned in step 200 can generally refer to a machine learning model that can mine spatial and temporal features from the continuous pulse stream and output the depth estimation result corresponding to the continuous pulse stream, preferably a model improved based on the Transformer network.

[0069] As can be seen from the above description, the monocular depth estimation method for a pulse camera provided by the embodiments of the present application can effectively solve the problem that there is no standard network that can simultaneously mine spatial and temporal features from a continuous pulse stream by designing a monocular depth estimation model for mining spatial and temporal features from the continuous pulse stream and outputting the depth estimation result data corresponding to the continuous pulse stream, can effectively improve the reliability of the monocular depth estimation prediction process for the pulse camera, and can improve the accuracy and effectiveness of the depth estimation result, thereby further improving the application reliability and intelligence level of the pulse camera.

[0070] In order to further mine spatial and temporal features from the continuous pulse stream, in a monocular depth estimation method for a pulse camera provided by the embodiments of the present application, see Figure 2 , before step 100 in the monocular depth estimation method for the pulse camera, the following specific content is further included:

[0071] Step 010: Use the training data set to train the preset spatiotemporal Transformer network model to obtain a monocular depth estimation model for mining spatial and temporal features from the continuous pulse stream in the training data set and outputting the depth estimation result corresponding to the continuous pulse stream.

[0072] Specifically, self-attention based models have recently become the dominant backbone architecture in natural language processing (NLP). It also has significant advantages, including massively parallel computation, long-range properties, and minimal inductive bias. For dense prediction tasks, Transformer has a global receptive field at each stage and can work at a constant and relatively high resolution. These attractive properties can naturally lead to fine-grained and globally coherent dense predictions. Transformer-based networks have been intensively studied for dense prediction. Some have applied ViT as the encoder backbone to estimate monocular depth. Compared with the CNN backbone, it shows that more coherent predictions can be learned due to the global receptive field of Transformer. Others have used additional ResNet projection layers and attention gates in the decoder to induce the spatial locality of CNN for monocular depth estimation. Others have used self-attention blocks to explore general contextual information and applied discrete disparity to regularize the training process. Others have studied self-supervised monocular depth estimation using visual transformers. The results show that compared with CNN-based architectures, Transformer achieves comparable performance while being more robust and general. Both NLP and image tasks have demonstrated the structural superiority of Transformer. Previous dense prediction work has also demonstrated the depth estimation ability of Transformer. This application designs a spatiotemporal Transformer network for monocular depth estimation from continuous and unstructured pulse streams.

[0073] From the above description, it can be seen that the monocular depth estimation method for pulse cameras provided in the embodiment of the present application achieves comparable performance to the CNN-based architecture, while being more robust and versatile. Both NLP and image tasks have demonstrated the structural superiority of the Transformer. Previous intensive prediction work has also demonstrated the depth estimation capability of the Transformer. The present application designs a spatiotemporal Transformer network for monocular depth estimation from continuous and unstructured pulse streams; by adopting a pre-involved spatiotemporal Transformer network model, the depth estimation capability of the trained monocular depth estimation model can be guaranteed, thereby further improving the reliability and effectiveness of the monocular depth estimation model used for pulse cameras.

[0074] Considering the problem of lack of pulse depth dataset in the prior art, namely: since calibrating the imaging window and synchronizing the timestamps between the pulse and depth cameras are quite complicated, there is currently no well-annotated dataset containing the pulse stream and the corresponding ground truth depth; this application considers collecting and generating a synthetic dataset, which is specifically described as follows.

[0075] In order to further improve the training reliability and application effectiveness of the monocular depth estimation model, in a monocular depth estimation method for a pulse camera provided in an embodiment of the present application, the training data set in the monocular depth estimation method for a pulse camera is a synthetic pulse data set including a pulse stream and corresponding real depth data.

[0076] Training a monocular depth estimation model in a supervised manner requires a large-scale training dataset in the form of a stream of spikes and the corresponding synchronized depth maps. However, constructing a realistic dataset consisting of pulse streams, grayscale images, and corresponding depth maps is complex. Moreover, calibrating the imaging window and synchronizing timestamps between the pulse camera, frame-based camera, and depth camera is quite complex. For this reason, we constructed a synthetic spike dataset.

[0077] Specifically, to unleash the potential of spike cameras for high-speed depth estimation, this application first collects and generates a synthetic dataset, which can be denoted as "DENSE-spike", which includes spike streams and corresponding ground-truth depth maps.

[0078] From the above description, it can be seen that the monocular depth estimation method for pulse cameras provided in the embodiments of the present application can effectively solve the problem that there is currently no well-annotated data set containing pulse streams and corresponding ground-truth depths due to the complexity of calibrating the imaging window and synchronizing the timestamps between the pulse and depth cameras, thereby effectively improving the training reliability and application effectiveness of the monocular depth estimation model.

[0079] In order to effectively verify the generalization of the monocular depth estimation model obtained by training, in a monocular depth estimation method for a pulse camera provided in an embodiment of the present application, see Figure 2 , step 010 and step 100 in the monocular depth estimation method for a pulse camera directly further specifically include the following contents:

[0080] Step 020: Acquire a real pulse sequence containing various scenes collected in advance by the pulse camera to form a corresponding real data set;

[0081] Step 030: Use the real data set to verify the generalization of the trained monocular depth estimation model.

[0082] Specifically, to verify the generalization of the proposed model, we further captured some real pulse sequences using a pulse camera. The spatial resolution of the pulse camera is 400×250, and the temporal resolution is about 1000 times higher than that of a standard frame-based camera. The real captured pulse streams record urban streets and road scenes. We named this real dataset "Outdoor-Spike" and it is only used for inference testing due to the lack of corresponding real depth data.

[0083] From the above description, it can be seen that the monocular depth estimation method for pulse cameras provided in the embodiment of the present application, by forming a corresponding real data set using a real pulse sequence containing various scenes captured by a pulse camera, can effectively verify the generalization of the monocular depth estimation model obtained by training, and thus can further verify the application reliability and effectiveness of the monocular depth estimation model.

[0084] In order to further effectively extract the temporal and spatial features of the pulse stream from the continuous pulse stream at the same time, in a monocular depth estimation method for a pulse camera provided in an embodiment of the present application, see Figure 3 , the monocular depth estimation model includes: a pulse embedding module and a Transformer network model;

[0085] The pulse embedding module is used to extract the time and space characteristics of the pulse stream to obtain corresponding multiple pulse embeddings;

[0086] The Transformer network model is used to obtain the spatiotemporal features of each pulse embedding and perform feature fusion on each spatiotemporal feature to predict the depth estimation result data of the pulse stream.

[0087] It can be understood that the network architecture of the monocular depth estimation model and the spatiotemporal Transformer network model is the same. The two are two forms before and after model training for the same network architecture, and can be written as: Spike-T.

[0088] From the above description, it can be seen that the monocular depth estimation method for pulse cameras provided in the embodiment of the present application, by adopting a pulse embedding module, can effectively realize the simultaneous extraction of the temporal and spatial characteristics of the pulse stream from the continuous pulse stream, thereby further improving the reliability of the monocular depth estimation prediction process for the pulse camera, and can improve the accuracy and effectiveness of the depth estimation results.

[0089] In order to further improve the reliability and effectiveness of learning the spatiotemporal features of pulse embedding and gradually fusing the hierarchical features from the encoder for final depth estimation, in a monocular depth estimation method for a pulse camera provided in an embodiment of the present application, see Figure 3, the Transformer network model includes: a spatiotemporal Transformer (transform) encoder and a convolutional decoder;

[0090] The spatiotemporal Transformer encoder is used to extract spatiotemporal features of each pulse embedding through a multi-layer self-attention mechanism to obtain the spatiotemporal features of each pulse embedding;

[0091] It can be understood that the spike embedding mentioned in one or more embodiments of the present application refers to the projection of the spike data in a high-dimensional space.

[0092] The convolution decoder is used to perform feature fusion and up-sampling on spatiotemporal features of different scales to predict the depth estimation result data of the pulse stream.

[0093] Specifically, we first feed the input pulse stream into the pulse embedding module to obtain several pulse embedding features that retain the spatiotemporal (time and space) features required for the input Transformer. Subsequently, we use several STT blocks to learn the spatiotemporal features of the pulse embedding, and use the self-attention mechanism to gradually fuse the hierarchical features from the encoder for the final depth estimation.

[0094] From the above description, it can be seen that the monocular depth estimation method for pulse cameras provided in the embodiment of the present application, by adopting a spatiotemporal Transformer (STT) encoder and a convolutional CNN-based decoder, can further improve the reliability and effectiveness of learning the spatiotemporal features of pulse embedding and fusing the hierarchical features from the encoder layer by layer for the final depth estimation on the basis of effectively maintaining the overall encoder-decoder architecture with a hierarchical structure, thereby further improving the reliability of the monocular depth estimation prediction process for the pulse camera and improving the accuracy and effectiveness of the depth estimation results.

[0095] In order to effectively maintain more time information and further realize the reliability and effectiveness of extracting the time and space features of the pulse stream from the continuous pulse stream at the same time, in a monocular depth estimation method for a pulse camera provided in an embodiment of the present application, the pulse embedding module includes: time partition, feature extraction layer, space partition and feature merging layer, wherein the time partition is provided with a multi-scale time window;

[0096] Correspondingly, the multi-scale time window is used to divide the pulse stream input into the pulse embedding module into a plurality of non-overlapping blocks;

[0097] The feature extraction layer is used to extract feature data of each non-overlapping block, and the feature data includes: local features and global features;

[0098] The spatial partition is used to split the feature data in the spatial domain to obtain spatiotemporal feature data;

[0099] The feature merging layer is used to merge the global features and local features in the spatiotemporal feature data to obtain multiple spatiotemporal blocks corresponding to the pulse stream, and determine each of the spatiotemporal blocks as a pulse embedding to be input into the Transformer network model.

[0100] Among them, a series of pulse frames The scene radiance at each timestamp t is recorded. Features along the time axis are crucial for reconstructing the depth map. Therefore, a multi-scale time window is introduced to utilize more temporal information.

[0101] From the above description, it can be seen that the monocular depth estimation method for a pulse camera provided in an embodiment of the present application, by setting time partitioning, feature extraction layer, spatial partitioning and feature merging layer, and providing a multi-scale time window in the time partition, can effectively utilize more time information and further realize the reliability and effectiveness of extracting the time and spatial characteristics of the pulse stream from the continuous pulse stream at the same time.

[0102] In order to ensure the core performance of the monocular depth estimation model, in a monocular depth estimation method for a pulse camera provided in an embodiment of the present application, the spatiotemporal Transformer encoder includes multiple stages, and each stage includes multiple spatiotemporal Transformer blocks, and a residual connection layer is provided between every two adjacent stages;

[0103] The spatiotemporal Transformer block (STT block) includes: a multi-head self-attention MSA module equipped with a 3D shift window, and a feed-forward neural network FFN composed of multiple MLP layers;

[0104] A GELU layer is provided between each of the MSA modules and the FFN, and a normalization layer LN is provided before each of the MSA modules and the FFN.

[0105] It can be understood that the STT block is a key component of Spike-T. Each STT block consists of a multi-head self-attention MSA module equipped with a 3D shift window, followed by a feed-forward network FFN consisting of 2 layers of MLP. Between each MSA module and FFN, a GELU layer is used, and a normalization layer LN is used before each MSA and FFN. A residual connection is applied to each module.

[0106] From the above description, it can be seen that the monocular depth estimation method for pulse cameras provided in the embodiment of the present application can effectively guarantee the core performance of the monocular depth estimation model by designing a spatiotemporal Transformer encoder based on the spatiotemporal Transformer block, thereby further ensuring the reliability of the monocular depth estimation prediction process for the pulse camera, and can improve the accuracy and effectiveness of the depth estimation results.

[0107] In order to further ensure the reliability of the monocular depth estimation prediction process for a pulse camera, in a monocular depth estimation method for a pulse camera provided in an embodiment of the present application, the convolution decoder includes: multiple residual blocks, multiple feature fusion layers and a prediction head.

[0108] Specifically, the decoder can be composed of two residual blocks, three feature fusion layers, and a prediction head. The output feature map size from each encoder stage is H i ×W i ×T×C i , connect along the time axis and reshape to H i ×W i ×TC i Afterwards, a convolutional layer is used to project the reconstructed features back to H i ×W i ×C i The last feature map has a size of First, two residual blocks with kernel size 3 and stride 1 are passed through. Then, the features of the previous layer are upsampled by bilinear interpolation and fused with the features encoded in this layer through a feature fusion layer. The prediction head consisting of one convolutional layer is finally used to generate a H×W×1 depth map.

[0109] From the above description, it can be seen that the monocular depth estimation method for a pulse camera provided in an embodiment of the present application, by designing a decoder based on a convolutional layer, can further ensure the reliability of the monocular depth estimation prediction process for a pulse camera and can improve the accuracy and effectiveness of the depth estimation results.

[0110] (II) Example of Monocular Depth Estimation Model Training Method

[0111] This application embodiment provides a monocular depth estimation model training method, see Figure 2The monocular depth estimation model training method is as shown in the aforementioned step 010, or as shown in steps 010 to 030, the monocular depth estimation method for a pulse camera provided in an embodiment of the present application trains a spatiotemporal Transformer network for performing monocular depth estimation from a continuous and unstructured pulse stream; the depth estimation capability of the trained monocular depth estimation model can be ensured, thereby further improving the reliability and effectiveness of monocular depth estimation for a pulse camera using the monocular depth estimation model.

[0112] The embodiment of the monocular depth estimation model training method provided in the present application can specifically refer to the processing flow except step 100 and step 200 in the above-mentioned embodiment of the monocular depth estimation method for a pulse camera. Its functions are not repeated here, and the detailed description except step 100 and step 200 in the above-mentioned embodiment of the monocular depth estimation method for a pulse camera can be referred to.

[0113] (III) Implementation of Monocular Depth Estimation Model

[0114] This application embodiment provides a monocular depth estimation model, see Figure 3 and Figure 4 The monocular depth estimation model is trained based on the above-mentioned monocular depth estimation model training method. The processing flow except step 100 and step 200 in the above-mentioned embodiment of the monocular depth estimation method for a pulse camera can be referred to. Its function is not repeated here. The detailed description except step 100 and step 200 in the above-mentioned embodiment of the monocular depth estimation method for a pulse camera can be referred to.

[0115] (IV) Embodiment of a monocular depth estimation device for a pulse camera

[0116] From the software level, the present application also provides a monocular depth estimation device for a pulse camera for executing all or part of the monocular depth estimation method for a pulse camera, see Figure 5 , the monocular depth estimation device for pulse camera specifically includes the following contents:

[0117] A data acquisition module 10 is used to acquire an unstructured continuous pulse stream output by a pulse camera;

[0118] The depth estimation module 20 is used to input the continuous pulse stream into a preset monocular depth estimation model so that the monocular depth estimation model mines spatial and temporal features from the continuous pulse stream and outputs a depth estimation result corresponding to the continuous pulse stream.

[0119] The embodiment of the monocular depth estimation device for a pulse camera provided in the present application can be specifically used to execute the processing flow of the embodiment of the monocular depth estimation method for a pulse camera in the above-mentioned embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above-mentioned embodiment of the monocular depth estimation method for a pulse camera.

[0120] From the above description, it can be seen that the monocular depth estimation device for a pulse camera provided in an embodiment of the present application, by designing a monocular depth estimation model for mining spatial and temporal features from the continuous pulse stream and outputting depth estimation result data corresponding to the continuous pulse stream, can effectively solve the problem that there is currently no standard network that can simultaneously mine spatial and temporal features from a continuous pulse stream, can effectively improve the reliability of the monocular depth estimation prediction process for a pulse camera, and can improve the accuracy and effectiveness of the depth estimation results, thereby further improving the application reliability and intelligence of the pulse camera.

[0121] The part of the monocular depth estimation device for a pulse camera that performs monocular depth estimation for a pulse camera can be executed in a server, and in another practical application scenario, all operations can also be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for specific processing of the monocular depth estimation for the pulse camera.

[0122] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0123] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0124] (V) Embodiments of electronic equipment

[0125] The embodiment of the present application also provides a computer device (i.e., an electronic device), which may include a processor, a memory, a receiver, and a transmitter, wherein the processor is used to execute the monocular depth estimation method for a pulse camera mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, such as a bus connection. The receiver may be connected to the processor and the memory via a wired or wireless manner. The computer device is communicatively connected to the monocular depth estimation device for a pulse camera to receive real-time motion data from a sensor in the wireless multimedia sensor network, and to receive an original video sequence from the video acquisition device.

[0126] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0127] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the monocular depth estimation for the pulse camera in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, realizing the monocular depth estimation for the pulse camera in the above method embodiment.

[0128] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0129] The one or more modules are stored in the memory, and when executed by the processor, perform the monocular depth estimation method for a pulse camera in the embodiment.

[0130] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0131] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0132] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0133] (VI) Embodiments of Computer Readable Storage Medium

[0134] The embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the steps of the aforementioned monocular depth estimation method for a pulse camera. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0135] (VII) Application Examples

[0136] To further illustrate the present solution, the present application also provides a specific application example of a monocular depth estimation method for a pulse camera. The present application example proposes Spike Transformer (Spike-T), which is a new paradigm for learning pulse data and estimating monocular depth from a continuous pulse stream. In order to fit the pulse data to the Transformer, the present application example proposes an input pulse embedding module equipped with a spatiotemporal partitioning module to maintain pulse features from the spatial and temporal domains. In addition, the present application example constructs two pulse-based depth datasets. One is simulated and synthesized, and the other is captured by a real pulse camera. Experimental results show that the proposed Spike-T can advantageously predict the depth of the scene and consistently outperforms its direct competitors. More importantly, the representation learned by Spike-T can be well transferred to real pulse data that has never been seen, indicating that Spike-T can be generalized to real-world scenes. As far as the present application example is aware, this is the first time that depth estimation directly from a pulse stream has become possible.

[0137] Inspired by previous studies on modeling spatiotemporal correlations in videos using Transformers, we first try to explore the possibility of Transformers learning spatiotemporal features from irregular pulse data. Although Transformers have been successfully applied to NLP, image, and point cloud processing, when targeting binary pulse data, the current approach is to convert the pulse stream into a video consisting of continuous intensity frames so that mature image-based algorithms can be used to learn the pulse stream. However, as the temporal resolution increases, this conversion from pulses to frames increases the computational cost and may also lose some of the temporal properties of the pulse stream.

[0138] Based on this, this work focuses on dense monocular depth estimation (MDE) from raw pulse streams. There are two key points:

[0139] 1) How to mine spatiotemporal features from binary, irregular, and continuous pulse streams;

[0140] 2) How to make full use of Transformer on unstructured spike data.

[0141] Based on this, the application example of this application proposes a new scheme called Spike Transformer (Spike-T) to mine the spatial and temporal characteristics of pulses and estimate depth from continuous pulse streams. Depth estimation is achieved using only pulse streams. In order to unleash the potential of pulse cameras in depth estimation of moving scenes, the application example of this application first collects and generates a synthetic dataset, which can be represented as "DENSE-spike", which includes pulse streams and corresponding real depth maps. And use the pulse camera to further collect a real dataset that can be named "Outdoor-spike", which includes various scenes of traffic roads and city streets.

[0142] Experimental results show that the proposed Spike-T performs well on the synthetic dataset of this application example and can reliably predict depth maps on real data. In summary, the main contributions of this application example include:

[0143] – The first work on monocular depth estimation from continuous spike streams. We first develop a synthetic and a real captured spike-based depth dataset.

[0144] – Proposed the Spike Transformer (Spike-T), which adopts the spatiotemporal Transformer architecture to learn unstructured spike data and mine the spatiotemporal features of spike streams.

[0145] –To fit spike data to Transformer, this application example proposes an input spike embedding equipped with a spatiotemporal partitioning module to preserve features from both spatial and temporal domains.

[0146] -Qualitative and quantitative evaluations on synthetic datasets show that the proposed Spike-T reliably predicts the depth of the scene, and the representations learned by Spike-T transfer well to unseen real data, demonstrating the generalization of the proposed model to real scenes.

[0147] See also Figure 4 , the Spike-T model proposed in the application example of this application can be a U-shaped network consisting of three components:

[0148] (a) Pulse embedding module;

[0149] (b) Spatial-Temporal Transformer Block (STT) encoder;

[0150] (c) Convolutional decoder.

[0151] The application example of this application first divides the input pulse stream into several non-overlapping blocks through multi-scale time windows. Then, a spatial partitioning layer and a local / global feature extractor (Feature Extractor, FE) are used to obtain a series of pulse embeddings. The encoder of the application example of this application is built by several STT blocks, which implements the attention mechanism along the time, height and width axes. The decoder includes multiple feature fusion layers, in which the hierarchical features are gradually fused and finally used to estimate the depth map. The specific instructions are as follows:

[0152] 1. Spike-T for monocular depth estimation

[0153] The pulse camera outputs pulses independently and asynchronously on each pixel. For simplicity, this application example uses S∈{0,1} H×W×T Denote a pulse stream and use D∈R H×W Represents the depth map of a poll. The purpose of monocular depth estimation of a pulse camera is to predict the original depth map D from a continuous binary pulse stream S. To this end, this application example proposes a Spike Transformer (Spike-T) for monocular depth estimation. The entire framework consists of three components: (a) a pulse embedding module, (b) a spatiotemporal transformer (STT) encoder, and (c) a CNN-based decoder. The framework is as follows Figure 4 As shown, it maintains an overall encoder-decoder architecture with a hierarchical structure. Specifically, the pulse stream is first input into the pulse embedding module to obtain several pulse embedding feature blocks, which retain the spatiotemporal (time and space) features of the Transformer input. Subsequently, the application example of this application uses several STT blocks to learn the spatiotemporal features of pulse embedding using an adaptive attention mechanism, and finally uses a CNN-based decoder to gradually fuse the hierarchical features from the encoder for final depth estimation.

[0154] (1) Pulse embedding

[0155] The pulse embedding module consists of three steps: time partitioning, feature extraction, and spatial partitioning. A series of pulse frames The scene radiance at each timestamp t is recorded. Features along the time axis are crucial for reconstructing the depth map. Therefore, a multi-scale time window is introduced to maintain more temporal information.

[0156] Specifically, for each original input S, the application example of this application first uses a sliding window of length Tn to divide it into n non-overlapping pulse blocks along the time axis. Each shape is The impulse blocks carry different local temporal features in the interval T. Then a lightweight feature extractor FE consisting of four residual blocks is used to project each block onto a feature map of size H×W×C. In theory, the length of the time window is It can be set to any positive integer not exceeding T. Different time scale windows carry time information of different scales. Therefore, the application example of this application can use multiple time scale windows, such as To capture multi-level temporal features.

[0157] In this application example, n is set to 4. The features of the time window can be considered as Then, in the present application example, n is set to 1, and the local features of The features of the time window can be regarded as global features. Similarly, other time windows can be used. In the setting of the application example of this application, only Local and Global features. In this case, the four pulse blocks are passed through the shared local FE module, and the full-length pulse stream is fed to the global FE module. Afterwards, the application example of this application splits each feature map in the FE module into By combining local and global FE features, the application example of this application can obtain H2×W2×4 time-robust feature maps. Therefore, the input pulse stream of shape H×W×T can be divided into This application example uses each ST block of size The ST block is considered as a token. Following the practice of Transformer in NLP and image-based tasks, this application example calls these labeled features pulse embeddings, so they can be used as input to the Transformer network.

[0158] (2) Spatiotemporal Transformer Encoder (also called Spatiotemporal Transformer Encoder or STT Encoder)

[0159] The overall architecture of the spatiotemporal Transformer encoder is as follows Figure 6 As shown in Fig. 1, it is adapted from the SwinTransformer architecture. The features of the spike embedding module are input into the Transformer-based encoder, which consists of three stages. Each stage consists of 2, 2, and 6 STT blocks, respectively. A residual connection layer is added between two adjacent stages.

[0160] Spatial partition merging: In order to retain more local temporal features, following previous work, this application example only implements downsampling operations in the spatial domain, maintaining the number of tokens in the temporal domain. More specifically, the features from each 2×2 spatially adjacent block are first connected along the C channel to form a merged feature map with 4C dimensions. Such a 4C-dimensional feature map is then projected to a 2C-dimensional feature map using a linear layer. Therefore, after each stage, the feature size along the C channel will double.

[0161] STT Block: The key component of Spike-T is the STT block. Each STT block consists of a multi-head self-attention MSA module equipped with a 3D shift window, followed by a feed-forward network FFN consisting of 2 layers of MLP. Between each MSA module and FFN, a GELU layer is used, and a normalization layer LN is used before each MSA and FFN. A residual connection is applied to each module.

[0162] In particular, for two consecutive STT blocks in a Transformer encoder stage, the MSA module in the previous block acts on T x ×H x ×W x Then use a space-time token of size W t ×W h ×W w The 3D window divides these tokens evenly into Non-overlapping windows. In the implementation of the application example of this application, T x Set to the same as the number of time partitions (n=4). x and W x is the current spatial token size. The 3D window size is set to 2×7×7. For the MSA module in the subsequent SST block, the application instance of this application shifts the window by (1,3,3) tokens from the previous STT block along the time, height, and width axes, and then performs attention between windows. The application instance of this application refers to the first window-based MSA as 3DW-MSA, which uses a conventional window partitioning configuration. The second window-based MSA is denoted as 3DSW-MSA, which applies a shifted window partitioning mechanism. The above two consecutive SST blocks can be formalized as A.

[0163]

[0164]

[0165]

[0166]

[0167] in and zm They represent the output features of block m in a stage of the 3D(S)W-MSA module and the FFN module. For each stage of the STT block, the application examples of this application use 3, 6, and 12 attention heads respectively. In this way, the features from each Transformer encoder stage can form a tuple with sizes of and It can therefore be used for downstream spike-based vision tasks.

[0168] (3) Depth Estimation Decoder

[0169] The decoder of the application example of this application consists of two residual blocks, three feature fusion layers and a prediction head. The output feature map size from each encoder stage is H i ×W i ×T×C i , connect along the time axis and reshape to H i ×W i ×TC i Afterwards, a convolutional layer is used to project the reconstructed features back to H i ×W i ×C i The last feature map has a size of First, two residual blocks with kernel size 3 and stride 1 are passed through. Then, the features of the previous layer are upsampled by bilinear interpolation and fused with the features encoded in this layer through a feature fusion layer. The prediction head consisting of one convolutional layer is finally used to generate a H×W×1 depth map.

[0170] (4) Loss Function

[0171] The application example of this application uses scale-invariant loss to train the depth estimation network of the application example of this application in a supervised manner. For the k-th pulse stream, the application example of this application represents the predicted depth map, the actual depth map and its residual as D k and R k . The scale-invariant loss is then defined as:

[0172]

[0173] in n is the number of valid live pixels p. Given a batch size L of live depth maps {D k}, so the total loss is

[0174] 2. Experiment

[0175] (1) Dataset

[0176] Synthetic Dataset: The application example of this application trains the Spike-T of the application example of this application in a supervised manner, which requires a large-scale training dataset in the form of a pulse stream and the corresponding synchronized depth map. However, it is complicated to construct a real dataset consisting of pulse data, grayscale images and corresponding depth maps. In addition, it is quite complicated to calibrate the imaging window and synchronize timestamps between pulse cameras, frame-based cameras and depth cameras. For this reason, the application example of this application constructs a synthetic pulse dataset. Specifically, the application example of this application first selects the dataset named DENSE proposed in as the database of the application example of this application. The DENSE dataset is generated by the CARLA simulator and includes clear depth maps and intensity frames at 30FPS under various weather and lighting conditions. In order to obtain a continuous pulse stream with very high temporal resolution, the application example of this application uses a video interpolation method to generate intermediate RGB frames between adjacent 30-FPS frames. With the help of the absolute intensity information between RGB frames, each sensor pixel can continuously accumulate light intensity through the pulse generation mechanism introduced in Section 3, generating a pulse stream with high temporal resolution (128×30=3840FPS), which is 128 times the temporal resolution of video frames. The "spike" version of the DENSE dataset (i.e., DENSE-spike) contains seven sequences, five for training, two for validation, and one for testing. Each sequence consists of 999 samples, each of which is a tuple of an RGB image, a depth map, and a pulse stream. Each pulse stream is simulated between two consecutive images to generate a binary sequence of 128 pulse frames (size 346×260), describing the continuous process of a dynamic scene.

[0177] Real data set: In order to verify the generalization of the proposed model, the application example of this application further collected some natural pulse sequences using a pulse camera. The spatial resolution of the pulse camera is 400×250, and the temporal resolution is about 1000 times faster than that of a standard frame-based camera. The real captured pulse stream records the scenes of urban streets and roads. The application example of this application represents this real data set as "Outdoor-Spike", which is only used for testing due to the lack of corresponding real-life depth. In the "Outdoor-Spike" data set of the application example of this application, 33 outdoor scene sequences were captured from the first-person perspective in a driving car. Each sequence contains 20,000 pulse frames.

[0178] (2) Implementation details

[0179] Depth representation: This application example uses the original depth D k,abs Convert to logarithmic depth map D k , can be calculated as:

[0180]

[0181] Among them, D max is the maximum depth in the dataset, and β is a hyperparameter set empirically to make the minimum depth close to 0. For the synthetic dataset “DENSE-spike” of the application example of this application, this application example sets β = 5.7, D max =1000m.

[0182] Training settings: The network is implemented in PyTorch. During training, the ADAM optimizer is used to optimize the network, and the initial learning rate λ is set to 0.0003. During training, the model of the application example of this application is trained on 2 NVIDIAA100-SXM4-80GB GPUs for 200 rounds with a batch size of 16. In addition, the application example of this application uses an exponential learning rate scheduler to adjust the learning rate after the 100th round, and γ is set to 0.5.

[0183] Metrics: In order to comprehensively evaluate the proposed model, several important indicators are introduced in the application examples of this application, including absolute relative error (Abs Rel.), square relative error (Sq Rel.), mean absolute depth error (MAE), root mean square logarithmic error (RMSE log) and accuracy index (Acc.δ).

[0184] (3) Experimental results

[0185] The application example of this application evaluates the performance of the method of the application example of this application on synthetic datasets and real captured datasets, and compares the Spike-T of the application example of this application with two model architectures U-Net and E2Depth. All three models were trained on the synthetic "DENSE-spike" dataset. In order to verify the generalization and transferability of the Spike-T of the application example of this application, the application example of this application is further tested using the real dataset "Outdoor-Spike" and qualitative visualization results are given. Finally, the ablation study experiment of Spike-T is introduced.

[0186] A. Qualitative and quantitative comparisons. The application example of this application first compares the Spike-T of the application example of this application with two dense prediction networks, namely U-Net and E2Depth. They and the Spike-T of the application example of this application both follow the encoder-decoder architecture of the multi-scale fusion method, but use different encoding mechanisms. In particular, U-Net uses 2D convolutional layers as its encoder and focuses on spatial feature extraction, while E2Depth applies the ConvLSTM layer combining CNN and LSTM to capture spatial and temporal features. In contrast, the Spike-T of the application example of this application adopts converter-based blocks to simultaneously learn spatiotemporal features. Therefore, both U-Net and E2Depth can be regarded as direct competitors of the application example of this application.

[0187] Table 1 reports the quantitative comparison on the "Dense-spike" dataset. On both the validation set and the test set, the proposed Spike-T consistently outperforms the other two methods in all metrics. In addition, the method of the application example of this application achieves significant improvements in the Abs Rel. metric, which is the most convincing metric in the depth estimation task. The main difference between the three methods lies in the encoder architecture. These experimental results show that the Spike-T with a Transformer-based encoder of the application example of this application is more effective in capturing spatiotemporal features from irregular continuous pulse streams.

[0188] Table 1 shows the quantitative comparison of the DENSE-spike dataset with U-Net and E2Depth. The application examples in this application show the results of the validation set and the test set. "↓" means the lower the better, and "↑" means the higher the better.

[0189] Table 1

[0190]

[0191] The application example of the present application also evaluates the method of the application example of the present application at depths of 10 meters, 20 meters, 30 meters, 80 meters, 250 meters, and 500 meters. Figure 7-1 and Figure 7-2 It shows how the absolute relative error and the average depth error vary with respect to the depth of the validation sequence. The results show that the method of the application example of the present application performs more accurate depth estimation at all distances, especially at larger distances.

[0192] Qualitative comparison Figure 8As shown. For a clear comparison, the present application example marks some cyan boxes on fine-grained objects. As can be seen in the present application example, more details, including tiny structures, sharp edges and contours, can be estimated by the pulse-T of the present application example. Quantitative and qualitative results show that the method of the present application example is more suitable for the continuous pulse stream generated by the pulse camera, and can learn effective and robust features from both spatial and temporal domains.

[0193] B. Evaluation on real-world datasets. We evaluate our method by training the network on a synthetic dataset and testing it on a real dataset, “Outdoor-Spike”. This is a more challenging dataset that is captured from outdoor scenes with various motions and noises from the real world. Fig. 9 Some examples with real impulses, grayscale images and predicted depth compared to baselines U-Net and E2Depth are shown. The visualization verifies that acceptable depth estimation results are achieved in real-world scenarios.

[0194] from Fig. 9 As can be seen in (c), the depth maps predicted by U-Net and E2Depth include blurring artifacts and lose some details, resulting in blurring between the foreground and background. In contrast, the depth map predicted by Spike-T is better, with more contour details, and can provide more accurate depth changes. Overall, despite the domain gap between synthetic data and real data, Spike-T in the application example of this application can reasonably predict the depth of real scenes, indicating the transferability of the model to real-world scenes.

[0195] (4) Ablation studies

[0196] A. The impact of multi-scale time windows. The application example of this application introduces a multi-scale time window in the pulse embedding stage to retain more time information. In order to verify its effectiveness, the application example of this application implemented three ablation studies, namely "global time", "local time" and "multi-scale time". Specifically, "global time" refers to feature embedding with only global features (number of time partitions n=1), which pays more attention to spatial features. "Local Temporal" means feature embedding using only local features (n=4), while "Multi-scale Temporal" means feature embedding from local (n=4) and global (n=1) features. The comparison results are shown in Table 2 for ablation studies of different time window mechanisms. On the most critical metric AbsRel., the pulse embedding using only "Local Temporal" is better than the embedding using only "Global Temporal", which indicates that the spatiotemporal correlations involved in the local features are more informative than the spatiotemporal correlations contained in the global part. In addition, the multi-scale time window combining local and global features is better than the above two settings. It shows that more spatiotemporal features can be learned from unstructured and continuous spike streams through a multi-scale time window mechanism.

[0197] Table 2

[0198]

[0199] Effect of BS-T patch partitioning. The block partitioning embedding operation in the standard Video Swin Transformer is implemented through a 3D convolutional layer. It divides the original input into several 3D blocks. The application example of this application conducted an ablation study on the block partitioning embedding method. By replacing the pulse embedding module with the Conv3D layer, the ST partitioning of the application example of this application was compared with the standard Conv3D-based partitioning. The quantitative results are listed in the ablation results of different block partitioning embedding methods shown in Table 3. The ST block partitioning embedding method of the application example of this application performed better in all indicators, which shows that ST partitioning is more suitable for extracting features from the pulse stream.

[0200] Table 3

[0201]

[0202] In summary, the application example of this application proposes Spike Transformer for monocular depth estimation of pulse cameras. In order to better apply Transformer to pulse data, an effective pulse representation called pulse embedding is first proposed. Then a modified Swin Transformer architecture is used to learn spatiotemporal pulse features.

[0203] In addition, two spike-based depth datasets are constructed. Experiments on synthetic and real datasets show that Spike-T can reliably predict depth maps and outperforms its direct competitors.

[0204] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0205] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0206] In the present application, features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0207] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A monocular depth estimation method for a pulse camera, characterized in that: include: Get the unstructured continuous pulse stream output by the pulse camera; Inputting the continuous pulse stream into a preset monocular depth estimation model, so that the monocular depth estimation model mines spatial and temporal features from the continuous pulse stream and outputs a depth estimation result corresponding to the continuous pulse stream; The monocular depth estimation model includes: a pulse embedding module and a Transformer network model; The pulse embedding module is used to extract the time and space characteristics of the pulse stream to obtain corresponding multiple pulse feature embeddings; The Transformer network model is used to obtain the spatiotemporal features of each pulse embedding and perform feature fusion on each of the spatiotemporal features to predict the depth estimation result data of the continuous pulse stream; The Transformer network model includes: a spatiotemporal Transformer encoder and a convolutional decoder; The spatiotemporal Transformer encoder is used to extract spatiotemporal features of each of the pulse embeddings through a multi-layer self-attention mechanism to obtain the spatiotemporal features of each of the pulse embeddings; The convolution decoder is used to perform feature fusion and up-sampling on the spatiotemporal features of different scales to predict the depth estimation result data of the continuous pulse stream; The pulse embedding module includes: time partition, feature extraction layer, space partition and feature merging layer, wherein the time partition is provided with a multi-scale time window; Correspondingly, the multi-scale time window is used to divide the pulse stream input into the pulse embedding module into a plurality of non-overlapping blocks; The feature extraction layer is used to extract feature data of each non-overlapping block, and the feature data includes: local features and global features; The spatial partition is used to split the feature data in the spatial domain to obtain spatiotemporal feature data; The feature merging layer is used to merge the global features and local features in the spatiotemporal feature data to obtain a plurality of spatiotemporal blocks corresponding to the continuous pulse stream, and determine each of the spatiotemporal blocks as a pulse embedding to be input into the Transformer network model; The spatiotemporal Transformer encoder includes multiple stages, each of which includes multiple spatiotemporal Transformer blocks, and a residual connection layer is provided between every two adjacent stages; The spatiotemporal Transformer block includes: a multi-head self-attention MSA module equipped with a 3D shift window, and a feed-forward neural network FFN composed of multiple MLP layers; A GELU layer is provided between each of the MSA modules and the FFN, and a normalization layer LN is provided before each of the MSA modules and the FFN; The convolution decoder includes: multiple residual blocks, multiple feature fusion layers and a prediction head.

2. The monocular depth estimation method for a pulse camera according to claim 1, characterized in that: Also includes: A preset spatiotemporal Transformer network model is trained using a training data set to obtain a monocular depth estimation model for mining spatial and temporal features from a continuous pulse stream in the training data set and outputting a depth estimation result corresponding to the continuous pulse stream.

3. The monocular depth estimation method for a pulse camera according to claim 2, characterized in that: The training data set is a synthetic pulse data set including a pulse stream and corresponding real depth data.

4. The monocular depth estimation method for a pulse camera according to claim 2, characterized in that: Also includes: Acquire a real pulse sequence containing various scenes collected in advance by the pulse camera to form a corresponding real data set; The real data set is used to verify the generalization of the monocular depth estimation model obtained by training.

5. A monocular depth estimation device for a pulse camera, characterized in that: include: A data acquisition module, used for acquiring an unstructured continuous pulse stream output by a pulse camera; A depth estimation module, used for inputting the continuous pulse stream into a preset monocular depth estimation model, so that the monocular depth estimation model mines spatial and temporal features from the continuous pulse stream and outputs a target depth estimation result corresponding to the continuous pulse stream; The monocular depth estimation model includes: a pulse embedding module and a Transformer network model; The pulse embedding module is used to extract the time and space characteristics of the pulse stream to obtain corresponding multiple pulse feature embeddings; The Transformer network model is used to obtain the spatiotemporal features of each pulse embedding and perform feature fusion on each of the spatiotemporal features to predict the depth estimation result data of the continuous pulse stream; The Transformer network model includes: a spatiotemporal Transformer encoder and a convolutional decoder; The spatiotemporal Transformer encoder is used to extract spatiotemporal features of each of the pulse embeddings through a multi-layer self-attention mechanism to obtain the spatiotemporal features of each of the pulse embeddings; The convolution decoder is used to perform feature fusion and up-sampling on the spatiotemporal features of different scales to predict the depth estimation result data of the continuous pulse stream; The pulse embedding module includes: time partition, feature extraction layer, space partition and feature merging layer, wherein the time partition is provided with a multi-scale time window; Correspondingly, the multi-scale time window is used to divide the pulse stream input into the pulse embedding module into a plurality of non-overlapping blocks; The feature extraction layer is used to extract feature data of each non-overlapping block, and the feature data includes: local features and global features; The spatial partition is used to split the feature data in the spatial domain to obtain spatiotemporal feature data; The feature merging layer is used to merge the global features and local features in the spatiotemporal feature data to obtain a plurality of spatiotemporal blocks corresponding to the continuous pulse stream, and determine each of the spatiotemporal blocks as a pulse embedding to be input into the Transformer network model; The spatiotemporal Transformer encoder includes multiple stages, each of which includes multiple spatiotemporal Transformer blocks, and a residual connection layer is provided between every two adjacent stages; The spatiotemporal Transformer block includes: a multi-head self-attention MSA module equipped with a 3D shift window, and a feed-forward neural network FFN composed of multiple MLP layers; A GELU layer is provided between each of the MSA modules and the FFN, and a normalization layer LN is provided before each of the MSA modules and the FFN; The convolution decoder includes: multiple residual blocks, multiple feature fusion layers and a prediction head.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the monocular depth estimation method for a pulse camera is implemented as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the monocular depth estimation method for a pulse camera according to any one of claims 1 to 4 is implemented.

Images