Three-dimensional depth estimation method based on pulse stream

Abstract

The invention provides a three-dimensional depth estimation method based on pulse streams, and belongs to the technical field of three-dimensional computer vision and neuromorphic imaging. The method comprises the following steps: (1) constructing a training and testing data set of a synthetic pulse stream; (2) constructing a pulse stream data set of a real scene; (3) constructing a three-dimensional depth estimation network model based on the pulse flow; (4) training a three-dimensional depth estimation network model based on the pulse flow; and (5) performing depth prediction by adopting the trained pulse stream-based stereo depth estimation network model. According to the invention, depth estimation can be directly carried out from the original pulse stream data, and a high-definition depth image is obtained without sequence framing or other post-processing; compared with an existing method, the method has better interpretability and higher prediction precision under a pulse flow data test set through iterative updating of the multi-layer and multi-scale cyclic pulse neural network and an algorithm of the pulse neural network.

Description

Technical Field

[0001] This invention relates to the fields of three-dimensional computer vision and neuromorphic imaging technology, and in particular to a stereo depth estimation method based on a binocular pulse camera, used to estimate depth images from raw stereo pulse streams. Background Technology

[0002] In real-time perception scenarios such as autonomous driving, drone obstacle avoidance, and robot navigation, it is necessary to simultaneously meet three indicators: "sub-millisecond latency," "high dynamic range," and "low power consumption." Traditional frame-based cameras, limited by fixed frame rates and exposure modes, struggle to stably output high-quality stereo images in high-speed or bright-light environments.

[0003] Pulse cameras employ an asynchronous event sampling mechanism: when the cumulative photon count of a single pixel reaches a threshold, a binary pulse is output, naturally possessing microsecond-level temporal resolution and a dynamic range greater than 130 dB. Although it provides an ideal sensing foundation for dynamic scenes, pulse data is characterized by sparseness, asynchronous nature, and high throughput, making it difficult for existing frame-based stereo matching algorithms and datasets to directly adapt to it, severely restricting the widespread application of pulse vision in dense depth estimation.

[0004] Existing stereo reconstruction methods often rely on monocular event cameras or frame pulse sequences before applying convolutional networks, leading to loss of temporal information and decreased depth accuracy and real-time performance. Furthermore, the limited size and inconsistent annotation methods of publicly available datasets also hinder reproducible research. Therefore, there is an urgent need for an end-to-end dense stereo reconstruction scheme that utilizes binocular pulse streams and balances high accuracy with high efficiency.

[0005] The significance of solving the above problems and defects is that: (1) it improves the quality of depth map estimation from stereo pulse streams and promotes research on neuromorphic vision; (2) it explores new ways of feature fusion and provides inspiration for other visual processing work; (3) as a method of using spiking neural networks, it can create greater value when combined with other work. Summary of the Invention

[0006] The purpose of this invention is to provide a pulse flow-based stereo depth estimation method that can directly perform end-to-end spatiotemporal analysis on the pulse flow, output high-precision parallax / depth images in milliseconds, and fully leverage the high dynamic range and sparsity characteristics of pulse imaging to significantly improve the accuracy and stability of 3D perception.

[0007] To achieve the technical objective of this invention, the following technical solution is adopted.

[0008] A stereo depth estimation method based on pulse flow includes the following steps: Step S1: Construct training and testing datasets for synthetic pulse streams: In order to train and test the stereo depth estimation neural network, a synthetic dataset is constructed. This dataset consists of multiple data pairs. Each data pair consists of a set of pulse stream data from the left and right viewpoints and their corresponding depth maps, and contains different scenes and objects. Step S2: Construct a real-world pulse flow dataset: In order to train and test the stereo depth estimation neural network, a real-world dataset is constructed. This dataset is acquired by a pulse camera and a depth camera. Each data pair consists of a set of real pulse flow data from the left and right viewpoints and their corresponding real depth maps, and includes different objects and lighting. Step S3 constructs a pulse flow-based stereo depth estimation network: The network model includes a pulse acquisition module, a feature extraction module, a correlation volume construction module, an iterative optimization module, an upsampling module, and a depth transformation module. It mainly integrates the visual information and correlation features contained in the pulse flow data adaptively to generate high-quality disparity information images with clearer texture details.

[0009] Step S4: Training process of the stereo depth estimation network based on pulse flow: Based on the above synthetic training dataset, given the loss function, the parameters in the stereo depth estimation network are continuously optimized using backpropagation and gradient descent algorithms over time until the loss function value stabilizes. Step S5 uses the pulse-flow-based stereo depth estimation network trained in step S4 to predict the depth: the test set in the synthetic dataset and the dataset of the real scene are selected as test sets for testing. The original pulse-flow data in the test set is used as the input of the stereo depth estimation network, and the output of the network is the predicted depth image.

[0010] Furthermore, in the pulse-stream-based stereo depth estimation network constructed in step S3, the pulse acquisition module is used to read from synthetic or real pulse-stream data files, convert the information in the data files into three-dimensional information tensors, and use them as input to the network; the feature extraction module is used to perform context and multi-scale feature extraction on the read raw pulse-stream data, thereby generating spatiotemporal feature maps of different resolutions, which are used as input to the iterative optimization module; the correlation volume construction module is used to construct the left and right feature maps of the pulse-stream data from the left and right perspectives, thereby obtaining the correlation cost volume and the multi-scale correlation pyramid, which are used to calculate the correlation features of the left and right perspectives, and are used as input to the iterative optimization module. Another input is the iterative optimization module, which uses a multi-layer, multi-scale recurrent spiking neural network to simultaneously receive contextual and relevance features from the two modules mentioned above. These features are combined and iteratively optimized for disparity estimation. By dynamically and adaptively adjusting neuron membrane potentials, firing thresholds, and resetting membrane potentials, it captures spatiotemporal dynamics, thereby generating a low-resolution and clear disparity image. The upsampling module receives the low-resolution disparity image output by the iterative optimization module and obtains the original resolution disparity image through interpolation and mapping upsampling. The depth transformation module receives the obtained original resolution disparity image and, combined with camera intrinsic and extrinsic parameters, estimates the original resolution depth image using a stereo matching mathematical method.

[0011] Furthermore, in the multi-layer, multi-scale recurrent spiking neural network used in the iterative optimization module, each layer contains intra-layer recurrent connections and inter-layer feedforward connections, supporting hierarchical temporal feature processing. Low-resolution layers are used to receive contextual features and synaptic currents of the same and previous layer pulse states, expanding the receptive field and promoting cross-scale feature propagation. High-resolution layers are used to additionally input the current disparity estimate and local correlation cost volume, combining matching cost and depth prediction for joint optimization. Each neuron in the recurrent spiking neural network employs an adaptive leak-integration-emission model, dynamically adjusting membrane potential, firing threshold, and reset mechanism, and using dynamic parameter updates to enable adaptive adjustment of relevant parameters.

[0012] Furthermore, in the pulse flow-based stereo depth estimation network in step S4, the gradient of the loss function with respect to each parameter in the network is calculated using the backpropagation algorithm over time. Then, the gradient descent algorithm is used to optimize the parameters of the entire network layer based on the training dataset until the loss function value is stable, thereby obtaining the optimal parameters of the stereo depth estimation network.

[0013] Due to the adoption of the above technical solutions, the three-dimensional depth estimation method based on pulse flow of the present invention has the following beneficial effects: 1. This invention constructs a pulse stream-based stereo depth estimation network, which can directly estimate depth from the original pulse stream data without performing sequence framing or other post-processing on the pulse stream data, thus obtaining a high-resolution depth image; 2. The large-scale synthetic and real stereo pulse flow dataset constructed in this invention covers spatiotemporal information with multiple scenes and high dynamic range, filling the gap in high-quality data in this field, providing reliable training and verification support for neuromorphic stereo depth estimation, and strongly promoting the optimization of related algorithms and the implementation of technologies. 3. This invention, through iterative updates of a multi-layer, multi-scale recurrent spiking neural network and the spiking neural network algorithm, achieves better interpretability and higher prediction accuracy compared to existing depth estimation algorithms on the pulse flow data test set. Specifically, it achieves the best estimation accuracy across all evaluation metrics (bad1.0, bad2.0, bad3.0, AvgErr). Attached Figure Description

[0014] Figure 1 This is a flowchart of the pulse flow-based stereo depth estimation method of the present invention; Figure 2 This is a physical structural diagram of a pulse-depth hybrid camera provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the system for the pulse flow-based stereo depth estimation method of the present invention; Figure 4 This is a detailed block diagram of the stereo depth estimation network of the pulse flow-based stereo depth estimation method of the present invention, wherein (a) is the feature extraction module, (b) is the iterative optimization module, and (c) is the meaning of the specific identifiers in the module; Figure 5 This is a visual comparison of the depth estimation method based on pulse flow of the present invention with the depth estimation of other existing algorithms on the test set. Detailed Implementation

[0015] To better understand this technical solution, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The accompanying drawings and embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore for illustrative purposes only, and should not be used to limit the scope of protection of the present invention.

[0016] Figure 1 This is a flowchart of the pulse flow-based stereo depth estimation method of the present invention, which includes the following steps: Step S1 constructs the training and testing datasets for the synthesized pulse stream and saves them as data files. The generation of the synthetic dataset is based on simulating an active stereo vision system using the Blender rendering engine, and includes the following steps: Step S11 Scene Construction: Randomly generate a 3D scene containing different materials (specular, transparent, diffuse), textures and layouts. The camera samples uniformly along a virtual sphere with a radius of 0.6 meters (3° interval) to simulate the real shooting trajectory and obtain the scene image and the RGB images of the left and right camera views.

[0017] Step S12 Depth Truth Generation: A depth map is generated using an infrared projection and stereo matching algorithm. The formula is as follows: , in, Baseline distance, For camera focal length, For parallax, The horizontal offset of the principal points of the left and right cameras.

[0018] Step S13: Pulse Stream Synthesis: The rendered video undergoes 50x frame interpolation (using a pre-trained optical flow model), and is then converted into a pulse stream using a time integration algorithm. The time integration algorithm for pulse stream conversion specifically includes the following steps: Step S131: Normalize: Convert the RGB frame to grayscale values ​​of [0, 1].

[0019] Step S132: Membrane potential accumulation: , for Membrane potential at time t, For membrane resistance, for Input current at any given time Exceeding the threshold It emits a pulse and then resets.

[0020] Step S133 Noise Simulation: Thermal Noise Caused by Injected Dark Current The formula is: , in, , Let be the coordinates of a pixel at a certain point in the image. and This is the integral over that time period. For that moment The current intensity at the pixel.

[0021] Step S134 Frame Interpolation: Using a multi-level feature extraction and residual refinement network based on optical flow, the video frame rate is increased by 50 times to generate intermediate frames with high temporal resolution (e.g., 2000 Hz).

[0022] Step S135 Pulse Coding: The continuous frames are converted into a binary pulse stream using a time integration algorithm, which supports adjustable noise levels and threshold parameters and is compatible with different hardware configurations.

[0023] Step S2: Construct a pulse flow dataset of the real-world scene and save it as a data file. The hardware devices used for data acquisition include... Figure 2 As shown, dynamic scenes and dense real-world depths are acquired through simultaneous pulse and depth camera acquisition. Existing real-world datasets contain multiple scenes, and a domain adaptation method is introduced into the real-world dataset to enhance the model's generalization ability in real-world scenes.

[0024] Step S3: Construct a stereo depth estimation network based on pulse flow.

[0025] A schematic diagram of the system for the pulse flow-based stereo depth estimation method of this invention is shown below. Figure 3 As shown, pulse flow data is acquired through a stereo pulse system and input into a stereo pulse model (i.e., a stereo depth estimation network based on pulse flow). The model generates high-quality disparity information images through iterative updates of a recurrent spiking neural network and related pyramid calculations, which are then further converted into depth images.

[0026] The pulse-stream-based stereo depth estimation network model includes pulse acquisition, feature extraction, correlation volume construction, iterative optimization, upsampling, and depth transformation modules. A detailed module diagram of the stereo depth estimation network is shown below. Figure 4 As shown, where Figure 4 (a) is the feature extraction module. Figure 4 (b) is the iterative optimization module. Figure 4 (c) defines the meaning of specific identifiers within the module; The pulse acquisition module is used to load and read the original pulse stream data storage file and convert it into a three-dimensional floating-point tensor with left and right dual perspectives, which is then used as input for subsequent network modules.

[0027] The feature extraction module includes a context network and a feature network, used for contextual and multi-scale pulse feature extraction. In this embodiment of the invention, the feature extraction module adopts a biologically inspired hierarchical structure to simulate the feature hierarchical processing mechanism in the visual pathway, such as... Figure 4 As shown in (a).

[0028] First, the input is preprocessed and downsampled, and the pulse streams for the left and right viewpoints are generated. ( N For the number of frames in the pulse stream, H The height of the pulse flow, W The pulse width is first passed through a convolutional layer for downsampling, reducing the spatial resolution to half, thus reducing computation while preserving the main spatial structure.

[0029] Residual blocks and multi-scale feature generation are employed. Residual blocks are used for feature extraction, with each block containing batch normalization and ReLU activation functions to enhance the network's ability to express deep features. After processing through multiple residual blocks, high-resolution feature maps are obtained through further downsampling.

[0030] Then, to capture multi-scale spatiotemporal features, left and right feature maps with resolutions of 1 / 4, 1 / 8, and 1 / 16 were further generated. ( (Among them, the left and right feature maps at 1 / 4 resolution) The feature maps at all scales are used to construct the matching cost volume, while the feature maps at all scales serve as guiding signals for the 3D regularization network.

[0031] In this feature extraction module, the context network and the feature network share the backbone structure. The context network generates multi-scale context features through additional downsampling layers. ( c The number of feature channels is taken in this embodiment. c = 128 channels), used to initialize the following recurrent spiking neural network module and inject spatiotemporal context information to enhance the temporal continuity and spatial consistency of features.

[0032] The correlation body construction module is used to calculate the cross-view correlation of features from the left and right perspectives and to calculate the matching cost, which is the core step of stereo matching.

[0033] In this embodiment of the invention, pairwise correlation calculation is used. For the left and right feature maps... and Calculate the correlation values ​​of all possible disparities at each spatial location to form the correlation cost volume: , in, For line numbers, and These are the column numbers of the left and right feature maps, respectively. This is the feature channel index.

[0034] Next, one-dimensional average pooling (kernel size 2, stride 2) is applied to the relevant cost volumes to construct a 4-level correlation pyramid. By progressively downsampling the disparity dimension, multi-scale matching cost estimation is achieved, reducing computational complexity while covering a wide range of disparities.

[0035] The iterative optimization module is used to iteratively refine the disparity estimated by the network. In this embodiment of the invention, a multi-layer recurrent spiking neural network is used as the core component of this iterative module. By simulating the adaptive dynamic characteristics of neurons, iterative optimization from the current disparity to the refined disparity is achieved, such as... Figure 4 As shown in (b).

[0036] Specifically, the iterative optimization module adopts a three-layer recurrent spiking neural network structure, with each layer containing intra-layer recurrent connections. Interlayer feedforward connection , The current layer of the recurrent spiking neural network supports hierarchical temporal feature processing: 1 / 8 and 1 / 16 resolution layers are used to receive contextual features. Synaptic currents in the same and preceding layers under pulsed states are used to expand the receptive field and facilitate cross-scale feature propagation. A 1 / 4 resolution layer is used to additionally input the current disparity estimate and local correlation cost volume, which are then jointly optimized by the matching cost and depth prediction.

[0037] Each neuron in the recurrent spiking neural network in this module employs an adaptive leak integration-emission model, dynamically adjusting membrane potential, firing threshold, and reset mechanism as shown below: , , , , in, For the current moment, For the previous moment, This represents the membrane potential of the neuron at the current moment after it has been charged and before it releases its pulse. This represents the membrane potential at the current moment. This indicates the pulse state being released at the current moment. These are adaptive parameters for the current moment, controlling membrane potential retention, excitation threshold, and soft reset, respectively. Indicates the peak point. This represents the threshold potential. and These are the recurrent synaptic weights and the feedforward synaptic weights (implemented through convolution kernels). This is a step function that simulates the pulse firing behavior of a neuron.

[0038] In addition, dynamic parameter updates and adaptive parameters are used. Calculated through convolutional layers: , , , , in, For the current moment, This represents the pulse state at the previous moment. For the current input features, Embedded vectors for context, Conv It is a convolutional neural network. , , These are the weights corresponding to the adaptive parameters. For the Sigmoid function, ensure that the parameter value is within... Interval.

[0039] The upsampling module is used to restore the low-resolution disparity map to its original resolution. After iterative refinement by the recurrent spiking neural network, the model outputs a disparity map at 1 / 4 resolution, which is then upsampled to the original resolution disparity map using a convex combination strategy. , in, For upsampling function, Weights for each scale, It is a low-resolution disparity map, learned through training. The low-resolution neighborhood grid size corresponding to each high-resolution pixel ensures smoothness of parallax edges and preservation of details during upsampling.

[0040] The depth conversion module is used to convert the output disparity map at the original resolution into an estimated depth map at the original resolution. In this embodiment of the invention, the parameters of the left and right pulse cameras are the same, therefore the calculation formula is as follows: , in m is the baseline distance. For camera focal length, For parallax, This is the final estimated depth.

[0041] Step S4: Train a pulse-flow-based stereo depth estimation network.

[0042] In this embodiment of the invention, the pulse-flow-based stereo depth estimation network uses a backpropagation algorithm over time to calculate the gradient of the loss function with respect to each parameter in the network, and is trained on a synthetic training dataset. The training process employs a composite loss function design, including a three-term regularized composite loss function, to balance depth estimation accuracy, neuron dynamic stability, and temporal sparsity, as specifically expressed below: , Among them, 3D loss ( ) Use weighted Norm, assigning higher weight to recent forecasts (discount factor) The model is guided to prioritize and optimize the latest estimation results. , in, For the first The true parallax of the next iteration For the first Predicted disparity in the next iteration T This represents the total number of iterations for the neuron.

[0043] Excitation rate regularization ( ), constraining the average firing rate of neurons Approaching the target value To avoid over-excitation or under-excitation states and improve network stability: , in, N The number of neurons in the network. T This represents the total number of iterations of the neuron. For the first i One neuron in t The status of the release at any given moment.

[0044] Voltage regularization ( Punishment membrane potential The dramatic fluctuations smooth out neuronal dynamics and promote the continuity of time series: in, N The number of neurons in the network. T This represents the total number of iterations of the neuron. For the first i One neuron in t The membrane potential at time t. Minimize it iteratively during training. This allows us to obtain the optimal parameters for the stereo depth estimation network. and These are the weights corresponding to the loss function. In this embodiment of the invention, they are set as follows: , The model and algorithm are implemented using the PyTorch framework, and training is performed on an NVIDIA RTX 4090 GPU (24GB VRAM). CUDA is used to accelerate pulse flow computation and neural network inference. The AdamW optimizer is used, and the gradient clipping range is [not specified]. To suppress gradient explosion, a single-cycle learning rate is used, with an initial value of... The learning rate is periodically adjusted to improve convergence speed. The batch size is 8, the number of iterations is 300,000, and each sample undergoes 16 iterations of the recurrent spiking neural network to balance computational accuracy and efficiency.

[0045] Step S5 uses a trained pulse-flow-based stereo depth estimation network to estimate the depth image.

[0046] Through the training process in step S4, the optimal parameters of the stereo depth estimation network model can be determined. Based on the trained model, to verify the effectiveness of the invention, inference is performed using a test set from a synthetic dataset and a dataset from a real-world scene as test sets. The left and right pulse flow data from the test set are input into the stereo depth estimation network, and the network output is the estimated depth image.

[0047] In the experiments, training was performed on the aforementioned synthesized training set, and testing was conducted on a synthesized test set and a real dataset. These two datasets contain 38,250 training pulse streams and 10,200 test pulse streams, respectively. To objectively evaluate the estimation accuracy of different methods, this invention uses the average endpoint error (AvgErr) and pixel error rate (Baderror) as comparative metrics, both commonly used evaluation metrics for stereo depth estimation.

[0048] To demonstrate the advanced nature of the technical solution of this invention, actual performance tests were conducted on several different methods, and the accuracy results of depth estimation were compared.

[0049] As shown in Table 1, the proposed StereoSpikeNet based on pulse flow was compared with various deep learning stereo depth estimation methods on the same dataset. The comparison methods (models) included: CREStereo [1], Stereospike [2], RAFT-Stereo [3], GMStereo [4], IGEV [5], DLNR [6], ZEST [7], MoCha-Stereo [8], Selective-Stereo [9], and MonSter

[10] . "↓" indicates that the lower the value, the better. The StereoSpikeNet proposed in this invention achieved the best estimation accuracy on different evaluation metrics (bad1.0, bad2.0, bad3.0, AvgErr) on the test dataset.

[0050] Table 1 shows the comparison results of the depth estimation evaluation metrics of the pulse flow-based depth estimation method of the present invention with other algorithms on the test set: Table 1 Figure 5The paper showcases depth image estimation results on a test set using different methods: IGEV, DLNR, Selective-Stereo, and the proposed StereoSpikeNet method. It is evident that the proposed method significantly outperforms the others, producing more accurate and clearer depth results. Furthermore, the sharpness and density of the predicted depth maps demonstrate the effectiveness of the proposed method in handling peak-flow inputs, particularly in complex and cluttered scenes.

[0051] The embodiments of the present invention have been described above with reference to the accompanying drawings. These are merely specific implementations of the present invention, but the present invention is not limited to the specific implementations described above. The specific implementations described above are merely illustrative and not restrictive. Those skilled in the art can make many variations or substitutions under the guidance of the present invention without departing from the spirit and scope of the claims, and all such variations or substitutions are within the protection scope of the present invention.

[0052] [1] Li, Jiankun, et al. "Practical stereo matching viacascadedrecurrent network with adaptive correlation." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022. [2] Rançon, Ulysse, et al. "Stereospike: Depth learning with aspikingneural network." IEEE Access 10 (2022): 127428-127439. [3] Lipson, Lahav, Zachary Teed, and Jia Deng. "Raft-stereo: Multilevel recurrent field transforms for stereo matching." 2021International Conference on 3D Vision (3DV). IEEE, 2021. [4] Xu, Haofei, et al. "Unifying flow, stereo and depthestimation."IEEE Transactions on Pattern Analysis and Machine Intelligence 45.11 (2023):13941-13958. [5] Xu, Gangwei, et al. "Iterative geometry encoding volume forstereomatching." Proceedings of the IEEE / CVF conference on computer vision andpattern recognition. 2023. [6] Zhao, Haoliang, et al. "High-frequency stereo matchingnetwork."Proceedings of the IEEE / CVF conference on computer vision and patternrecognition. 2023. [7] Lou, Hanyue, et al. "Zero-shot event-intensity asymmetricstereovia visual prompting from image domain." Advances in Neural InformationProcessing Systems 37 (2024): 13274-13301. [8] Chen, Ziyang, et al. "Mocha-stereo: Motif channelattentionnetwork for stereo matching." Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition. 2024. [9] Wang, Xianqi, et al. "Selective-stereo: Adaptivefrequencyinformation selection for stereo matching." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024.

[10] Cheng, Junda, et al. "Monster: Marry monodepth tostereounleashes power." Proceedings of the Computer Vision and PatternRecognition Conference. 2025。

Claims

1. A method for stereo depth estimation based on pulse streams, characterized in that, The method comprises the following steps: Step S1: constructing a training and test dataset of a synthetic pulse stream: in order to train and test a stereo depth estimation neural network, a synthetic dataset is constructed, which is composed of a plurality of data pairs, each data pair being composed of a set of left and right perspective pulse stream data and a corresponding depth map and containing different scenes and objects; Step S2: constructing a pulse stream dataset of a real scene: in order to train and test a stereo depth estimation neural network, a dataset of a real scene is constructed, which is acquired by a pulse camera and a depth camera, each data pair being composed of a set of left and right perspective real pulse stream data and a corresponding real depth map and containing different objects and illumination; Step S3: constructing a stereo depth estimation network based on a pulse stream: the network model comprises a pulse acquisition module, a feature extraction module, a correlation volume construction module, an iterative optimization module, an up-sampling module and a depth conversion module, which are used to adaptively fuse visual information and correlation features contained in the pulse stream data to generate a high-quality parallax information image; Step S4: a stereo depth estimation network training process based on a pulse stream: based on the above synthetic training dataset, a loss function is given, and the parameters in the stereo depth estimation network are continuously optimized using backpropagation through time and gradient descent algorithm until the loss function value is stable; Step S5: predicting depth by using the stereo depth estimation network based on a pulse stream trained in step S4: selecting a test set in the synthetic dataset and a real scene dataset as a test set for testing, using the original pulse stream data in the test set as the input of the stereo depth estimation network, and the output of the network is the predicted depth image.

2. The method of claim 1, wherein, In the stereo depth estimation network based on a pulse stream constructed in step S3, the pulse acquisition module is used to read from a synthetic or real pulse stream data file, convert the information in the data file into a three-dimensional information tensor and use it as the input of the network; The feature extraction module is used to extract the context and multi-scale features of the original pulse stream data read to generate spatial-temporal feature maps of different resolutions, which are used as an input of the iterative optimization module; the correlation volume construction module is used to construct left and right feature maps of the left and right perspective pulse stream data to obtain a correlation cost volume and a multi-scale correlation pyramid, which are used to calculate the correlation features of the left and right perspectives and are used as another input of the iterative optimization module; the iterative optimization module uses a multi-layer multi-scale recurrent pulse neural network to simultaneously receive the context features and the correlation features from the aforementioned two modules, combines and iteratively optimizes the parallax estimation, captures spatial-temporal dynamics by dynamically and adaptively adjusting the neuron membrane potential, firing threshold and reset membrane potential, thereby generating a low-resolution and clear parallax image; The up-sampling module is used to receive the low-resolution parallax image output by the iterative optimization module and obtain a parallax image of the original resolution through interpolation and mapping up-sampling; and the depth conversion module is used to receive the parallax image of the original resolution, combine the intrinsic and extrinsic parameters of the camera and estimate a depth image of the original resolution through a stereo matching mathematical method.

3. The method of claim 2, wherein, In the multi-layer multi-scale recurrent pulse neural network used by the iterative optimization module, each layer of the network contains intra-layer recurrent connections and inter-layer feedforward connections, supports hierarchical temporal feature processing, low-resolution layers are used to receive contextual features, synaptic currents of pulse states of the same layer and previous layers, expand the receptive field and promote cross-scale feature propagation, high-resolution layers are used to additionally input current disparity estimation and local related cost volume, and the matching cost and depth prediction are combined for joint optimization, wherein the neurons of each recurrent pulse neural network adopt an adaptive leaky integration-firing model, dynamically adjust the membrane potential, firing threshold and reset mechanism, and use dynamic parameter updating, so that the related parameters can be adaptively adjusted.

4. The method of claim 1, wherein, In the pulse stream-based stereo depth estimation network of the step S4, a loss function is calculated with respect to the gradient of each parameter in the network using a backpropagation algorithm over time, and then the parameters of the entire network layer are optimized using a gradient descent algorithm based on the training data set until the loss function value is stable, and the optimal parameters of the stereo depth estimation network are obtained.

Images