Image and event fusion monocular depth estimation method based on structural state space

Abstract

The application discloses a kind of monocular depth estimation methods based on structure state space image and event fusion, belong to depth estimation technical field. Including: first, the original event stream data obtained in target time window in target scene is constructed as event voxel grid that can be aligned with image data, then the input level alignment of two kinds of modal data is completed by frequency domain bidirectional prior injection;Subsequently, the multi-level features between event modal data and image modal data are extracted using asymmetric modal perception encoder with shared weights, and a modal interaction local refinement mechanism is introduced in the fusion stage, realizing cross-modal state guidance and local geometric enhancement, to obtain fusion features between event modal data and image modal data;Finally, through the decoder based on visual state space model, the depth map under the target scene is recovered and output according to the fusion features, thereby improving the efficiency and precision of monocular depth estimation in complex scenes.

Description

Technical Field

[0001] This application belongs to the field of depth estimation technology, specifically relating to a monocular depth estimation method based on image and event fusion in structural state space. Background Technology

[0002] In recent years, existing monocular depth estimation techniques typically use traditional images as the primary input. While these methods can provide relatively rich texture and appearance information, they are prone to motion blur, overexposure, or underexposure in scenes with low light, rapid motion, and drastic dynamic range changes, leading to unstable depth prediction results, particularly difficulty in accurately recovering object boundaries and the geometric relationships of complex regions. Meanwhile, event cameras can asynchronously record brightness changes, offering high temporal resolution and high dynamic range, demonstrating significant advantages in fast motion and extreme lighting conditions. However, event data itself is characterized by sparsity, edge-driven nature, and incomplete structure; if used alone for depth estimation, it often fails to recover complete scene texture and global geometry. Existing event-image fusion schemes, although attempting to utilize both modalities simultaneously, still suffer from the following problems: The first problem is that image and event data are heterogeneous modalities, and both have limitations when input separately. Images can provide rich texture and appearance information, but they are prone to motion blur, overexposure, or underexposure in extreme scenes. Event data has high temporal resolution and high dynamic range, but it is characterized by sparsity, edge-driven nature, and incomplete structure, making it difficult to recover complete scene texture and global geometry. Direct fusion of events and images is prone to semantic shifts due to differences in source domain, data sparsity density, and semantic hierarchy, resulting in poor cross-modal alignment.

[0003] The second problem is that existing modeling networks struggle to balance accuracy and efficiency. Traditional convolutional networks excel at modeling local structures, but their receptive field is limited, making them unable to effectively characterize long-distance dependencies and cross-regional geometric relationships, and their global modeling capabilities are insufficient.

[0004] The third problem is that existing fusion schemes are insufficient in mining the multi-level semantic complementarity relationships between events and images, lack a unified fusion framework, and cannot simultaneously take into account the global consistency of depth estimation, the ability to restore boundary details, and the robustness to complex scenes. Summary of the Invention

[0005] The purpose of this application is to provide a monocular depth estimation method based on image and event fusion in a structured state space. This method constructs a unified fusion framework with a visual state space model as the core, achieves cross-modal explicit alignment at the input layer and feature layer, and completes global dependency modeling, local detail enhancement and multi-level semantic interaction by combining an efficient state space modeling mechanism. Under the premise of controlling computational complexity, it significantly improves the accuracy, robustness and generalization ability of monocular depth estimation in complex environments, thereby solving the above problems.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a monocular depth estimation method based on image and event fusion in a structured state space, the method comprising: Acquire image data and event stream data of the target scene synchronized within the target time window, and generate an event voxel mesh based on the event stream data; For the event voxel grid, event modal data with injected global structure priors are generated based on the image data; and for the image data, image modal data with injected motion-sensitive priors are generated based on the event voxel grid. Feature extraction is performed on the image modality data and the event modality data respectively to obtain image features and event features. The image features include at least shallow to mid-level features of the image modality data, and the event features include at least mid- to deep-level features of the event modality data. The image features and the event features are fused to obtain fused features, and a depth map of the target scene within the target time window is generated based on the fused features.

[0007] Optionally, generating an event voxel grid based on the event stream data includes: The event stream data within the target time window is discretized along the time axis into K time interval event stream data units; For the event stream data units of the K time channels, a voxel grid representation of size K×H×W is constructed by interpolation accumulation to obtain the event voxel grid, where K represents the K time channels corresponding to the K time intervals, and H and W represent the image size.

[0008] Optionally, generating event modality data for the event voxel grid based on the image data, which includes injecting global structure priors into the image, comprises: Two-dimensional discrete Fourier transforms are performed on the event voxel grid and the image data respectively to obtain the event amplitude and event phase of the event voxel grid and the image amplitude of the image data; The event amplitudes of the event voxel grid below the preset frequency threshold are replaced with the image amplitudes of the image data that are below the preset frequency threshold to obtain the updated event amplitudes. The updated event amplitude and event phase are subjected to inverse Fourier transform to obtain the event modal data of the global structure prior of the injected image.

[0009] Optionally, generating image modal data with injected motion-sensitive priors based on the event voxel grid for the image data includes: A two-dimensional discrete Fourier transform is performed on the event voxel grid to obtain the event amplitude of the event voxel grid; From the event voxel grid, obtain the event voxel grids corresponding to M time channels in K time channels whose event amplitude is greater than the global response amplitude threshold; The event voxel grids corresponding to the M time channels are stitched together with the image data to obtain image modality data with injected motion-sensitive priors; Where M is not greater than K.

[0010] Optionally, the step of extracting features from the image modality data and the event modality data to obtain image features and event features respectively includes: For the image modal data, with the aim of preserving shallow to mid-level features, the features containing enhanced texture, edge and spatial details contained in the image modal data are extracted to obtain the image features; For the event modal data, with the aim of preserving mid-to-deep features, the dynamic changes, structural outlines and abstract semantic features contained in the event modal data are extracted to obtain the event features.

[0011] Optionally, fusing the image features and the event features to obtain fused features includes: The image features and the event features are respectively subjected to linear mapping and depthwise convolution to obtain enhanced image features and enhanced event features; During the state evolution process, the enhanced image features and the enhanced event features exchange modulation information with each other to obtain cross-modal semantically guided image features and cross-modal semantically guided event features. The cross-modal semantic guidance image features and the cross-modal semantic guidance event features are multiplied element-wise and the residuals are added to obtain the initial cross-modal joint features; With the aim of enhancing the object boundary contour, depth abrupt regions, and local geometric details, the overlapping local windows of the cross-modal initial joint features are scanned in multiple directions to obtain the cross-modal intermediate joint features; The multiple feature channels contained in the cross-modal intermediate joint feature are scored to determine the key channels among the multiple feature channels, and the key channels are enhanced to obtain the fused feature.

[0012] Optionally, generating a depth map based on the fusion features includes: The fused features are then subjected to progressive upsampling and feature reconstruction to obtain intermediate decoding features; Channel descriptors for each feature channel contained in the intermediate decoding feature are generated by global pooling, and the weights of each feature channel contained in the intermediate decoding feature are determined based on the channel descriptors. According to the weights, the multiple feature channels contained in the intermediate decoding features are divided into effective channels, invalid channels, and noise channels. The effective channels are enhanced, and the invalid and noise channels are suppressed to obtain the depth map.

[0013] Secondly, embodiments of this application provide a monocular depth estimation system based on image and event fusion in a structured state space. The system is applied to a visual state space model, which includes: an event representation module, a frequency domain alignment module, a modal sensing coding module, a modal interaction module, and a decoding and output module. The event representation module is used to acquire image data and event stream data synchronized in the target scene within the target time window, and generate an event voxel grid based on the event stream data; The frequency domain alignment module is used to generate event modal data with injected global structure priors for the image data based on the event voxel grid, and to generate image modal data with injected motion-sensitive priors based on the event voxel grid for the image data. The modality-aware coding module is used to extract features from the image modality data and the event modality data respectively to obtain image features and event features. The image features include at least shallow to mid-level features of the image modality data, and the event features include at least mid- to deep-level features of the event modality data. The modal interaction module is used to fuse the image features and the event features to obtain fused features; The decoding and output module is used to generate a depth map of the target scene within the target time window based on the fusion features; The visual state space model is trained using training samples processed with normalized logarithmic depth during the training phase, and the loss value calculated from the true predicted depth and the true depth is used to optimize the visual state space model. The true predicted depth is the depth obtained by restoring the normalized predicted depth in the depth map output by the visual state space model.

[0014] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps in the method described in the first aspect.

[0015] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps in the method described in the first aspect.

[0016] This application proposes a monocular depth estimation method based on image and event fusion in a structured state space. The method includes: acquiring synchronized image data and event stream data of a target scene within a target time window, and generating an event voxel grid based on the event stream data; generating event modal data with injected global structural priors based on the image data for the event voxel grid, and generating image modal data with injected motion-sensitive priors based on the event voxel grid for the image data; extracting features from the image modal data and the event modal data respectively to obtain image features and event features, wherein the image features include at least shallow to mid-level features of the image modal data, and the event features include at least mid- to deep features of the event modal data; fusing the image features and the event features to obtain fused features, and generating a depth map of the target scene within the target time window based on the fused features.

[0017] The technical solution of this application constructs a dense event voxel grid that can be aligned with image data from the original event stream data. Then, frequency-domain bidirectional prior injection is performed on the event voxel grid and image data to achieve input-level alignment. This transforms the event voxel grid and image data into event modal data with injected global image structure priors and image modal data with injected motion-sensitive priors. Based on the representational characteristics of both, an asymmetric hierarchical selection method is used for feature extraction to obtain image features and event features that retain features from different layers. This improves the quality of cross-modal feature alignment and information utilization efficiency, and addresses the limitation of traditional convolutional networks being prone to localization issues. While structural modeling is used, the receptive field is limited, making it unable to effectively characterize long-distance dependencies and cross-regional geometric relationships. It also suffers from insufficient global modeling capabilities and semantic bias and poor cross-modal alignment caused by direct fusion of two modalities, which result in differences in source domain, data sparsity density, and semantic hierarchy. By fusing image features and event features to output a depth map, efficient depth reconstruction of the target scene is achieved. This unified framework simultaneously considers the global consistency of depth estimation, the ability to restore boundary details, and robustness to complex scenes, thereby improving the efficiency and accuracy of monocular depth estimation in complex scenes. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the framework of a visual state space model in a monocular depth estimation system based on structured state space for image and event fusion, provided in an embodiment of this application. Figure 2This is a flowchart illustrating a monocular depth estimation method based on image and event fusion in structural state space, provided in an embodiment of this application. Figure 3 This is a schematic diagram of the framework of a monocular depth estimation device based on structured state space image and event fusion according to an embodiment of this application; Figure 4 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] Monocular depth estimation is a fundamental problem in autonomous driving, robot navigation, augmented reality, and 3D perception. Compared to solutions such as binocular vision, structured light, or LiDAR, monocular solutions are less expensive and more flexible in deployment. However, recovering 3D depth from only 2D observations inherently suffers from insufficient information. To improve the reliability of estimation in complex scenes, researchers have recently begun to combine traditional image processing with event cameras. Images provide dense texture, structural, and semantic information, while events provide highly sensitive responses to changes in brightness; the two are clearly complementary in terms of information.

[0022] However, many problems still exist, making it difficult to meet the requirements of high-precision and high-efficiency monocular depth estimation in complex scenarios. Specifically, these problems include: First, convolutional network-based methods are limited by the receptive field of convolution, making them unable to effectively handle occlusion, large-scale depth changes, and cross-regional geometric relationships, resulting in insufficient global modeling capabilities. Secondly, methods based on Transformer or hybrid architectures have high computational overhead, and most of them only perform simple fusion operations such as feature concatenation and element-wise addition at the feature layer, without systematically processing the differences in input domain, sparse density, and semantic hierarchy between images and events, resulting in poor cross-modal alignment. Third, existing solutions lack a unified framework that can simultaneously cover bidirectional prior injection at the input level, asymmetric coding under a shared backbone, multi-level semantic interaction, and local refinement enhancement, and do not fully explore the complementary relationships between events and images at different semantic levels.

[0023] To address the aforementioned issues, this application proposes a monocular depth estimation method based on the fusion of asymmetric images and events. This method uses a visual state space model as its backbone and designs targeted cross-modal alignment and interaction mechanisms in the input layer, feature layer, and fusion layer of the model, thereby balancing the accuracy, efficiency, and stability of depth estimation in complex scenes.

[0024] Figure 1 This is a schematic diagram of the framework of a visual state space model in a monocular depth estimation system based on structured state space for image and event fusion, provided in an embodiment of this application.

[0025] The technical solution of this application aims to improve the efficiency and accuracy of monocular depth estimation in complex scenes, especially for challenging application conditions such as low light, fast motion, strong dynamic range changes and occlusion. The core idea of ​​this application is: first, to construct the original event stream data into a dense representation that can be aligned with the image data, and then to complete the input-level alignment of events and images through bidirectional prior injection in the frequency domain; then, to extract multi-level features between events and images after bidirectional injection using a shared-weight asymmetric modal perception encoder, and to introduce a modal interaction local refinement mechanism in the fusion stage to achieve cross-modal state guidance and local geometric enhancement; finally, to recover the output depth map through a decoder based on a state space model.

[0026] refer to Figure 1 The system for implementing the technical solution of this application is applied to a pre-trained visual state space model, which includes the following modules: an event representation module, a frequency domain alignment module, a modal awareness encoding module, a modal interaction module, a decoding and output module, and a supervised training module. The relationship between these modules is as follows: The event representation module is responsible for converting the raw event stream data into a structured input of event voxel grids; The frequency domain alignment module is responsible for bidirectional prior injection of image data and event voxel grids; The modality-aware coding module is responsible for extracting multi-level image features and event features between image modality data and event modality data; The modal interaction module is responsible for interacting and fusing selected hierarchical features between image features and event features to obtain fused features; The decoding and output module is responsible for progressively restoring the spatial resolution and outputting the depth map; The supervised training module is used to optimize the visual state space model based on the real depth map during the training phase.

[0027] Specifically, the system is applied to a visual state space model, which includes: an event representation module, a frequency domain alignment module, a modal perception coding module, a modal interaction module, and a decoding and output module. The event representation module is used to acquire image data and event stream data synchronized in the target scene within the target time window, and generate an event voxel grid based on the event stream data; The frequency domain alignment module is used to generate event modal data with injected global structure priors for the image data based on the event voxel grid, and to generate image modal data with injected motion-sensitive priors based on the event voxel grid for the image data. The modality-aware coding module is used to extract features from the image modality data and the event modality data respectively to obtain image features and event features. The image features include at least shallow to mid-level features of the image modality data, and the event features include at least mid- to deep-level features of the event modality data. The modal interaction module is used to fuse the image features and the event features to obtain fused features; The decoding and output module is used to generate a depth map of the target scene within the target time window based on the fusion features; The visual state space model is trained using training samples processed with normalized logarithmic depth during the training phase, and the loss value calculated from the true predicted depth and the true depth is used to optimize the visual state space model. The true predicted depth is the depth obtained by restoring the normalized predicted depth in the depth map output by the visual state space model.

[0028] During the training phase of the visual state space model, a targeted prediction strategy and loss function are designed through a supervised training module to achieve efficient optimization of network parameters, thereby improving model stability and estimation accuracy. This approach does not directly predict the original physical depth but prioritizes predicting the normalized logarithmic depth, effectively reducing training instability caused by large-scale depth value fluctuations. After obtaining the logarithmic depth prediction result, it is then restored to the metric depth, and error calculation is performed with the true depth map. This avoids situations where the distance between objects in the training samples varies too much during training (e.g., the depth of nearby object A is 1 meter, and the depth of distant object B is 100 meters, resulting in a large numerical range), which could lead to oscillations, non-convergence, and instability during model training. By using the normalized logarithmic depth, the original depth values ​​of each object in the training samples are compressed into a numerical range (e.g., compressed to between 0 and 1). The model's output is then restored and compared with the true value, making the model more stable during training.

[0029] Furthermore, the supervised loss employs a hybrid loss function that includes both absolute and squared error terms, jointly optimizing all valid pixels. This loss mechanism balances the structural accuracy and numerical stability of depth estimation, achieving simultaneous optimization and improvement of the overall scale and local boundaries of the depth map. Using the backpropagation algorithm, the parameters of all modules in the model are updated based on the calculation results of the hybrid loss, completing end-to-end training of the model.

[0030] The following description, in conjunction with the accompanying drawings, details a monocular depth estimation method based on structured state space for image and event fusion, provided by the embodiments of this application, through specific implementations and application scenarios.

[0031] Figure 2 This is a flowchart illustrating a monocular depth estimation method based on image and event fusion in structural state space, provided in an embodiment of this application.

[0032] refer to Figure 2 This application provides a monocular depth estimation method based on image and event fusion in structured state space, which includes steps S11 to S14: Step S11: Obtain the image data and event stream data synchronized within the target time window of the target scene, and generate an event voxel mesh based on the event stream data.

[0033] In this embodiment, the synchronous image data and event stream data of the target scene acquired by the image camera and the event camera within the target time window are first received. The target scene is the scene that needs to be depth estimated. In order to facilitate the spatial alignment of the event stream data and the image data and use it as input, the original event stream data needs to be constructed into a dense representation that can be aligned with the image data through the event representation module to obtain the event voxel grid.

[0034] Optionally, in one embodiment, the step S11 of "generating an event voxel mesh based on the event stream data" specifically includes steps S11-1 to S11-2: Step S11-1: Discretize the event stream data within the target time window into K time interval event stream data units along the time axis.

[0035] In this embodiment, each event stream data can be represented as a quadruple (x, y, t, p), where x and y represent pixel positions, t represents timestamps, and p represents event polarity. Multiple discrete event stream data obtained within a given target time window are discretized into K time intervals along the time axis, and the event stream data in each time interval is a single event stream data unit.

[0036] Step S11-2: For the event stream data units of the K time channels, a voxel grid representation of size K×H×W is constructed by interpolation accumulation to obtain the event voxel grid, where K represents the K time channels corresponding to the K time intervals, and H and W represent the image size.

[0037] In this embodiment, in order to facilitate spatial alignment with image data and input into the network, a voxel grid representation of size K×H×W needs to be constructed by interpolation accumulation. In this way, the original sparse event stream data can be converted into a structured event input suitable for subsequent frequency domain processing and feature encoding, i.e., an event voxel grid, while preserving the dynamic information in the event stream data. The image data is not processed and is directly used as a standardized structured image input in subsequent steps.

[0038] Specifically, the process of constructing a voxel grid representation using interpolation accumulation is as follows:

[0039] Step S12: For the event voxel grid, generate event modal data with injected global structure priors for the image based on the image data; and for the image data, generate image modal data with injected motion-sensitive priors based on the event voxel grid.

[0040] In this embodiment, considering the heterogeneous characteristics of dense information in image data and sparse information in event voxel grids, it is necessary to perform frequency domain transformation on the structured image data and event voxel grids obtained in step S11 through a frequency domain alignment module. Then, bidirectional prior injection and input-level alignment are performed on the image data and event voxel grids, so that the event voxel grids are converted into event modal data with injected global structural priors of the image, and the image data is converted into image modal data with injected motion-sensitive priors.

[0041] Optionally, in one embodiment, the step S12 of "generating event modality data for the event voxel grid based on the image data to inject the global structure prior of the image" specifically includes steps S12-1-1 to S12-1-3: Step S12-1-1: Perform two-dimensional discrete Fourier transform on the event voxel grid and the image data respectively to obtain the event amplitude and event phase of the event voxel grid and the image amplitude of the image data.

[0042] In this embodiment, by performing a two-dimensional discrete Fourier transform on the event voxel grid and the image data, the amplitude spectrum and phase spectrum of each are obtained, that is, the event amplitude and event phase of the event voxel grid, and the image amplitude and image phase of the image data are obtained.

[0043] Step S12-1-2: Replace the event amplitude of the event voxel grid with the image amplitude of the image data that is below the preset frequency threshold to obtain the updated event amplitude.

[0044] Specifically, the determination of frequencies below the preset frequency threshold follows the process for selecting the low-frequency square region:

[0045] In this embodiment, the corresponding part of the event amplitude is then replaced with the image amplitude in the low-frequency region (i.e. below a preset frequency threshold) representing the overall outline and overall scene layout structure of the target scene in the amplitude spectrum.

[0046] Step S12-1-3: Perform inverse Fourier transform on the updated event amplitude and the event phase to obtain the event modal data of the global structure prior of the injected image.

[0047] In this embodiment, the replaced event amplitude and event phase are then subjected to inverse Fourier transform to obtain the event modal data of the global structure prior of the injected image.

[0048] Through the above embodiments, it is possible to retain the high-frequency dynamic characteristics such as motion boundaries, motion features, and brightness changes contained in the phase spectrum of the event voxel grid, while supplementing the event voxel grid with features such as the overall framework, contour, general shape of objects, and spatial layout contained in the amplitude spectrum of the image data.

[0049] Optionally, in one embodiment, step S12, "generating image modal data with injected motion-sensitive priors based on the event voxel grid for the image data," specifically includes steps S12-2-1 to S12-2-3: Step S12-2-1: Perform a two-dimensional discrete Fourier transform on the event voxel grid to obtain the event amplitude of the event voxel grid.

[0050] In this embodiment, refer to step S12-1-1.

[0051] Step S12-2-2: Obtain the event voxel grids corresponding to the M time channels in the K time channels whose event amplitude is greater than the global response amplitude threshold from the event voxel grid.

[0052] In this embodiment, the event voxel grids corresponding to M time channels with event amplitudes greater than the global response amplitude threshold are selected from the event voxel grids, that is, several time channels with higher global response amplitudes are selected.

[0053] Step S12-2-3: The event voxel grids corresponding to the M time channels are stitched together with the image data to obtain image modal data with injected motion-sensitive priors; wherein, M is not greater than K.

[0054] In this embodiment, the event voxel grid corresponding to the selected time channel is stitched with the image data to obtain image modality data with injected motion-sensitive priors, thereby improving the ability of subsequent depth estimation to perceive dynamic and boundary regions in the target scene.

[0055] For example, an event voxel grid of size M×H×W can be stitched together with image data of size C×H×W to obtain image modal data of size (C+M)×H×W.

[0056] Through the above embodiments, explicit alignment of image data and event modality data at the input level can be achieved through the bidirectional prior injection operation, effectively reducing the source domain differences of heterogeneous modalities and laying the foundation for subsequent feature extraction and fusion.

[0057] Step S13: Perform feature extraction on the image modality data and the event modality data respectively to obtain image features and event features. The image features include at least shallow to mid-level features of the image modality data, and the event features include at least mid- to deep-level features of the event modality data.

[0058] In this embodiment, the process employs a shared-weight visual state space encoder as the backbone for feature extraction. This encoder comprises multiple cascaded encoding stages, each consisting of visual state space blocks. It progressively reduces spatial resolution and increases channel dimensions through downsampling operations. By inputting aligned image modal data and event modal data into a modality-aware encoding module centered on the visual state space model, multi-level feature extraction and selection are achieved.

[0059] Optionally, in one embodiment, step S13, "performing feature extraction on the image modality data and the event modality data respectively to obtain image features and event features," specifically includes steps S13-1 to S13-2: Step S13-1: For the image modal data, with the aim of preserving shallow to mid-level features, extract the features containing enhanced texture, edge and spatial details contained in the image modal data to obtain the image features.

[0060] In this embodiment, unlike the traditional symmetrical dual-branch structure, this scheme does not require image modal data and event modal data to adopt the exact same feature usage strategy at each layer of the model. Instead, it uses an asymmetric hierarchical selection method to extract features based on the representation characteristics of the two.

[0061] Specifically, for image modal data, shallow to mid-level features are retained first to enhance the expression of texture, edge and spatial details.

[0062] Step S13-2: For the event modality data, with the aim of preserving mid-to-deep features, extract the dynamic changes, structural outlines and abstract semantic features contained in the event modality data to obtain the event features.

[0063] In this embodiment, for event modality data, mid- to deep features are preferentially retained to highlight dynamic changes, structural outlines, and abstract semantics. This asymmetric design improves cross-modal feature alignment quality and information utilization efficiency while sharing backbone parameters.

[0064] Step S14: The image features and the event features are fused to obtain fused features, and a depth map of the target scene within the target time window is generated based on the fused features.

[0065] In this embodiment, after performing deep interaction, fusion, and local geometric detail enhancement on the image features and event features obtained in step S13 to obtain fused features, the depth map is reconstructed and output step by step based on the fused features.

[0066] Optionally, in one embodiment, step S14, "fusing the image features and the event features to obtain fused features," specifically includes steps S14-1-1 to S14-5: Step S14-1-1: Perform linear mapping and depthwise convolution on the image features and event features respectively to obtain enhanced image features and enhanced event features.

[0067] In this embodiment, linear mapping and depthwise convolution are first performed on the image features and event features respectively to enhance the local structural sensitivity of the image features and event features, resulting in enhanced image features and enhanced event features.

[0068] Step S14-1-2: For the enhanced image features and the enhanced event features, during the state evolution process, the modulation information between the enhanced image features and the enhanced event features is exchanged to obtain cross-modal semantic guidance image features and cross-modal semantic guidance event features.

[0069] In this embodiment, an interactive selection scanning mechanism is then used to enable the enhanced image features and enhanced event features of the two modalities to exchange modulation information during the state evolution process, thereby achieving cross-modal semantic guidance without destroying the internal state transition rules of each feature, and finally obtaining cross-modal semantic guided image features and cross-modal semantic guided event features.

[0070] Step S14-1-3: Perform element-wise multiplication and residual summation on the cross-modal semantic guidance image features and the cross-modal semantic guidance event features to obtain the initial cross-modal joint features.

[0071] In this embodiment, after the interaction of the above steps is completed, the two features, cross-modal semantic guidance image features and cross-modal semantic guidance event features, are fused by element-wise multiplication and residual addition to form cross-modal initial joint features.

[0072] Step S14-1-4, with the aim of enhancing the object boundary contour, depth abrupt region and local geometric details, performs multi-directional scanning on the overlapping local window of the cross-modal initial joint feature to obtain the cross-modal intermediate joint feature.

[0073] In this embodiment, the local spatial selection scanning mechanism is used to perform multi-directional scanning of the overlapping local windows to enhance the characterization of object boundaries, depth change regions and local geometric details, thereby obtaining cross-modal intermediate joint features.

[0074] Step S14-1-5: Score the multiple feature channels contained in the cross-modal intermediate joint feature, determine the key channel among the multiple feature channels, and enhance the key channel to obtain the fused feature.

[0075] In this embodiment, the response of key channels is enhanced by channel recalibration, and fusion features are output, thereby improving the recovery quality of local depth boundaries and complex regions while maintaining global consistency.

[0076] The technical solutions described above enable deep fusion of cross-modal features, significantly improving the recovery quality of local depth boundaries and complex regions while maintaining global consistency in depth estimation.

[0077] Optionally, in one embodiment, the step S14 of "generating a depth map based on the fusion features" specifically includes steps S14-2-1 to S14-2-3: Step S14-2-1: Perform step-by-step upsampling and feature reconstruction on the fused features to obtain intermediate decoding features.

[0078] In this embodiment, the decoder used to generate the depth map is a lightweight decoder based on channel-aware visual state space blocks. The core performs stepwise upsampling and feature reconstruction operations, and a channel attention mechanism is introduced during the decoding process.

[0079] Step S14-2-2: Generate a channel descriptor for each feature channel contained in the intermediate decoding feature through global pooling, and determine the weight of each feature channel contained in the intermediate decoding feature based on the channel descriptor.

[0080] In this embodiment, channel descriptors are generated by global pooling of fused features, and the weight values ​​of each feature channel are calculated.

[0081] Step S14-2-3: According to the weights, the multiple feature channels contained in the intermediate decoding features are divided into effective channels, invalid channels and noise channels. The effective channels are enhanced, and the invalid channels and noise channels are suppressed to obtain the depth map.

[0082] In this embodiment, by calculating the weight values ​​of each feature channel in the fusion feature, effective channels, invalid channels, and noise channels are divided according to the magnitude of the weight values. The response capability of effective fusion features in the effective channels is enhanced, while the interference of invalid features in the invalid and noise channels is suppressed. Finally, the fusion feature is processed by the convolutional output layer to generate a dense depth map with a spatial resolution that is completely consistent with the input image data.

[0083] The technical solution of this application constructs a dense event voxel grid that can be aligned with image data from the original event stream data. Then, frequency-domain bidirectional prior injection is performed on the event voxel grid and image data to achieve input-level alignment. This transforms the event voxel grid and image data into event modal data with injected global image structure priors and image modal data with injected motion-sensitive priors. Based on the representational characteristics of both, an asymmetric hierarchical selection method is used for feature extraction to obtain image features and event features that retain features from different layers. This improves the quality of cross-modal feature alignment and information utilization efficiency, and addresses the limitation of traditional convolutional networks being prone to localization issues. While structural modeling is used, the receptive field is limited, making it unable to effectively characterize long-distance dependencies and cross-regional geometric relationships. It also suffers from insufficient global modeling capabilities and semantic bias and poor cross-modal alignment caused by direct fusion of two modalities, which result in differences in source domain, data sparsity density, and semantic hierarchy. By fusing image features and event features to output a depth map, efficient depth reconstruction of the target scene is achieved. This unified framework simultaneously considers the global consistency of depth estimation, the ability to restore boundary details, and robustness to complex scenes, thereby improving the efficiency and accuracy of monocular depth estimation in complex scenes.

[0084] To verify the performance of the image event fusion monocular depth estimation method based on structured state space proposed in this application, comparative experiments were conducted using two public datasets, MVSEC (real-world scene) and DENSE (synthetic scene). The experiments evaluated the method based on three core dimensions: accuracy, robustness, and computational efficiency. Commonly used evaluation metrics in the field of monocular depth estimation were employed, including accuracy metrics. < The method uses the mean absolute error (Abs. Rel) and floating-point operations (FLOPs) as the evaluation index for computational efficiency. The experimental results are shown in Table 1. The proposed method demonstrates excellent accuracy and robustness in both real-world and synthetic scenarios. Compared to existing UniCTDepth methods, it achieves improved accuracy while maintaining lower computational complexity, meeting the deployment requirements for depth estimation in resource-constrained scenarios.

[0085] Table 1: Experimental Results

[0086] It should be noted that the monocular depth estimation method based on structure-state-space image event fusion provided in this application can be executed by a monocular depth estimation device based on structure-state-space image event fusion, or by a control module within that device for executing the monocular depth estimation method based on structure-state-space image event fusion. This application embodiment uses the execution of the monocular depth estimation method based on structure-state-space image event fusion by a device based on structure-state-space image event fusion as an example to illustrate the monocular depth estimation method based on structure-state-space image event fusion provided in this application embodiment.

[0087] Figure 3 This is a schematic diagram of the framework of a monocular depth estimation device based on structured state space image and event fusion according to an embodiment of this application. (Refer to...) Figure 3 .

[0088] One embodiment of this application provides a monocular depth estimation device based on structured state space-based image and event fusion, the device comprising: The data acquisition module 11, configured in the event representation module, is used to acquire image data and event stream data synchronized in the target scene within the target time window, and generate an event voxel grid based on the event stream data; The data alignment module 12 is configured with a frequency domain alignment module, used to generate event modal data with injected global structure priors for the image data based on the event voxel grid, and to generate image modal data with injected motion-sensitive priors based on the event voxel grid for the image data. The feature extraction module 13 is configured in the modality-aware coding module and is used to extract features from the image modality data and the event modality data respectively to obtain image features and event features. The image features include at least shallow to mid-level features of the image modality data, and the event features include at least mid- to deep-level features of the event modality data. The feature fusion module 14, configured in the modal interaction module, is used to fuse the image features and the event features to obtain fused features; The depth map acquisition module 15, configured and decoded in the output module, is used to generate a depth map of the target scene within the target time window based on the fusion features.

[0089] Optionally, the data acquisition module 11 includes: An event stream data unit acquisition unit is used to discretize the event stream data within the target time window into K time interval event stream data units along the time axis; The event voxel grid acquisition unit is used to construct a voxel grid representation of size K×H×W for the event stream data units of the K time channels by using interpolation accumulation, thereby obtaining the event voxel grid, where K represents the K time channels corresponding to the K time intervals, and H and W represent the image size.

[0090] Optionally, the data alignment module 12 includes: The first Fourier transform unit is used to perform two-dimensional discrete Fourier transform on the event voxel grid and the image data respectively, to obtain the event amplitude and event phase of the event voxel grid and the image amplitude of the image data; The update unit is used to replace the event amplitude of the event voxel grid within the preset frequency threshold with the image amplitude of the image data within the preset frequency threshold, so as to obtain the updated event amplitude; The event modal data acquisition unit is used to perform inverse Fourier transform on the updated event amplitude and the event phase to obtain the event modal data of the global structure prior of the injected image.

[0091] Optionally, the data alignment module 12 includes: The second Fourier transform unit is used to perform a two-dimensional discrete Fourier transform on the event voxel grid to obtain the event amplitude of the event voxel grid. The event voxel grid selection unit is used to obtain the event voxel grids corresponding to M time channels in K time channels whose event amplitude is greater than the global response amplitude threshold from the event voxel grid; The image modal data acquisition unit is used to stitch the event voxel grids corresponding to the M time channels with the image data to obtain image modal data with injected motion-sensitive priors, where M is no greater than K.

[0092] Optionally, the feature extraction module 13 includes: The image feature acquisition unit is used to extract the features of enhanced texture, edge and spatial details contained in the image modal data with the aim of preserving shallow to middle layer features, in order to obtain the image features; The event feature acquisition unit is used to extract the dynamic changes, structural contours and abstract semantic features contained in the event modality data with the aim of prioritizing the retention of mid-to-deep features, and obtain the event features.

[0093] Optionally, the feature fusion module 14 includes: The enhanced feature acquisition unit is used to perform linear mapping and depthwise convolution processing on the image features and the event features respectively to obtain enhanced image features and enhanced event features; A cross-modal semantic guidance feature acquisition unit is used to exchange modulation information between the enhanced image features and the enhanced event features during the state evolution process to obtain cross-modal semantic guidance image features and cross-modal semantic guidance event features; The cross-modal initial joint feature acquisition unit is used to perform element-wise multiplication and residual summation on the cross-modal semantic guidance image features and the cross-modal semantic guidance event features to obtain the cross-modal initial joint features; The cross-modal intermediate joint feature acquisition unit is used to perform multi-directional scanning on the overlapping local window of the cross-modal initial joint feature in order to enhance the object boundary contour, depth abrupt region and local geometric details, so as to obtain the cross-modal intermediate joint feature. The fusion feature acquisition unit is used to score multiple feature channels contained in the cross-modal intermediate joint feature, determine the key channel among the multiple feature channels, and enhance the key channel to obtain the fusion feature.

[0094] Optionally, the depth map acquisition module 15 includes: An intermediate decoding feature acquisition unit is used to perform step-by-step upsampling and feature reconstruction on the fused features to obtain intermediate decoding features; The weight acquisition unit is used to generate a channel descriptor for each feature channel contained in the intermediate decoding feature through global pooling, and to determine the weight of each feature channel contained in the intermediate decoding feature based on the channel descriptor. The depth map acquisition unit is used to divide the multiple feature channels contained in the intermediate decoded features into effective channels, invalid channels and noise channels according to the weights, and to enhance the effective channels and suppress the invalid channels and noise channels to obtain the depth map.

[0095] The monocular depth estimation device based on structured state space image and event fusion in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not specifically limit the specific implementation.

[0096] The monocular depth estimation device based on structured state space-based image and event fusion in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0097] The monocular depth estimation device based on structure state space-based image and event fusion provided in this application can achieve... Figure 2 The various processes implemented by the image and event fusion monocular depth estimation device based on the structural state space in the method embodiment will not be described again here to avoid repetition.

[0098] Optionally, the event representation module is used to acquire image data and event stream data synchronized within a target time window of the target scene, and generate an event voxel grid based on the event stream data, including: The event stream data within the target time window is discretized along the time axis into K time interval event stream data units; For the event stream data units of the K time channels, a voxel grid representation of size K×H×W is constructed by interpolation accumulation to obtain the event voxel grid, where K represents the K time channels corresponding to the K time intervals, and H and W represent the image size.

[0099] Optionally, the frequency domain alignment module is used to generate event modal data for injecting global structure priors of the image into the event voxel grid based on the image data, including: Two-dimensional discrete Fourier transforms are performed on the event voxel grid and the image data respectively to obtain the event amplitude and event phase of the event voxel grid and the image amplitude of the image data; The event amplitudes of the event voxel grid below the preset frequency threshold are replaced with the image amplitudes of the image data that are below the preset frequency threshold to obtain the updated event amplitudes. The updated event amplitude and event phase are subjected to inverse Fourier transform to obtain the event modal data of the global structure prior of the injected image.

[0100] Optionally, the frequency domain alignment module is used to generate image modal data with injected motion-sensitive priors based on the event voxel grid for the image data, including: A two-dimensional discrete Fourier transform is performed on the event voxel grid to obtain the event amplitude of the event voxel grid; From the event voxel grid, obtain the event voxel grids corresponding to M time channels in K time channels whose event amplitude is greater than the global response amplitude threshold; The event voxel grids corresponding to the M time channels are stitched together with the image data to obtain image modality data with injected motion-sensitive priors; Where M is not greater than K.

[0101] Optionally, the modality-aware coding module is used to extract features from the image modality data and the event modality data respectively to obtain image features and event features, including: For the image modal data, with the aim of preserving shallow to mid-level features, the features containing enhanced texture, edge and spatial details contained in the image modal data are extracted to obtain the image features; For the event modal data, with the aim of preserving mid-to-deep features, the dynamic changes, structural outlines and abstract semantic features contained in the event modal data are extracted to obtain the event features.

[0102] Optionally, the modal interaction module is used to fuse the image features and the event features to obtain fused features, including: The image features and the event features are respectively subjected to linear mapping and depthwise convolution to obtain enhanced image features and enhanced event features; During the state evolution process, the enhanced image features and the enhanced event features exchange modulation information with each other to obtain cross-modal semantically guided image features and cross-modal semantically guided event features. The cross-modal semantic guidance image features and the cross-modal semantic guidance event features are multiplied element-wise and the residuals are added to obtain the initial cross-modal joint features; With the aim of enhancing the object boundary contour, depth abrupt regions, and local geometric details, the overlapping local windows of the cross-modal initial joint features are scanned in multiple directions to obtain the cross-modal intermediate joint features; The multiple feature channels contained in the cross-modal intermediate joint feature are scored to determine the key channels among the multiple feature channels, and the key channels are enhanced to obtain the fused feature.

[0103] Optionally, the decoding and output module is used to generate a depth map based on the fused features, including: The fused features are then subjected to progressive upsampling and feature reconstruction to obtain intermediate decoding features; Channel descriptors for each feature channel contained in the intermediate decoding feature are generated by global pooling, and the weights of each feature channel contained in the intermediate decoding feature are determined based on the channel descriptors. According to the weights, the multiple feature channels contained in the intermediate decoding features are divided into effective channels, invalid channels, and noise channels. The effective channels are enhanced, and the invalid and noise channels are suppressed to obtain the depth map.

[0104] Optionally, Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. This application also provides an electronic device; it should be noted that the electronic device in this application includes the mobile electronic device and non-mobile electronic device described above.

[0105] The electronic device includes, but is not limited to, components such as: radio frequency unit, network module, audio output unit, input unit, sensor, display unit, user input unit, interface unit, memory, and processor.

[0106] Those skilled in the art will understand that electronic devices may also include power supplies (such as batteries) that supply power to various components. The power supply may be connected to the processor logic through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0107] As an example, such as Figure 4As shown, the electronic device 600 includes a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus for communication. The memory 610 stores a computer program that can run on the processor 620 to implement the steps in the image and event fusion monocular depth estimation method based on structured state space disclosed in the above embodiments of this application.

[0108] As the apparatus is basically similar to the method embodiment, it is described in a relatively simple way. For relevant details, please refer to the description of the method embodiment.

[0109] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0110] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects.

[0111] Furthermore, this application embodiment also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image and event fusion monocular depth estimation method embodiment based on structural state space, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0112] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0113] This application also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiments of the image and event fusion monocular depth estimation method based on structure state space, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0114] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0115] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0117] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A monocular depth estimation method based on image and event fusion in structured state space, characterized in that, The method includes: Acquire image data and event stream data of the target scene synchronized within the target time window, and generate an event voxel mesh based on the event stream data; For the event voxel grid, event modal data with injected global structure priors are generated based on the image data; and for the image data, image modal data with injected motion-sensitive priors are generated based on the event voxel grid. Feature extraction is performed on the image modality data and the event modality data respectively to obtain image features and event features. The image features include at least shallow to mid-level features of the image modality data, and the event features include at least mid- to deep-level features of the event modality data. The image features and the event features are fused to obtain fused features, and a depth map of the target scene within the target time window is generated based on the fused features.

2. The monocular depth estimation method based on structured state space image and event fusion according to claim 1, characterized in that, The generation of the event voxel grid based on the event stream data includes: The event stream data within the target time window is discretized along the time axis into K time interval event stream data units; For the event stream data units of the K time channels, a voxel grid representation of size K×H×W is constructed by interpolation accumulation to obtain the event voxel grid, where K represents the K time channels corresponding to the K time intervals, and H and W represent the image size.

3. The monocular depth estimation method based on image and event fusion in structured state space according to claim 1, characterized in that, The step of generating event modality data for the event voxel grid, based on the image data, to inject global structure priors into the image includes: Two-dimensional discrete Fourier transforms are performed on the event voxel grid and the image data respectively to obtain the event amplitude and event phase of the event voxel grid and the image amplitude of the image data; The event amplitudes of the event voxel grid below the preset frequency threshold are replaced with the image amplitudes of the image data that are below the preset frequency threshold to obtain the updated event amplitudes. The updated event amplitude and event phase are subjected to inverse Fourier transform to obtain the event modal data of the global structure prior of the injected image.

4. The monocular depth estimation method based on image and event fusion in structured state space according to claim 3, characterized in that, The step of generating motion-sensitive prior image modality data based on the event voxel grid for the image data includes: A two-dimensional discrete Fourier transform is performed on the event voxel grid to obtain the event amplitude of the event voxel grid; From the event voxel grid, obtain the event voxel grids corresponding to M time channels in K time channels whose event amplitude is greater than the global response amplitude threshold; The event voxel grids corresponding to the M time channels are stitched together with the image data to obtain image modality data with injected motion-sensitive priors; Where M is no greater than K.

5. The monocular depth estimation method based on structured state space image and event fusion according to claim 1, characterized in that, The step of extracting features from the image modality data and the event modality data to obtain image features and event features includes: For the image modal data, with the aim of preserving shallow to mid-level features, the features containing enhanced texture, edge and spatial details contained in the image modal data are extracted to obtain the image features; For the event modal data, with the aim of preserving mid-to-deep features, the dynamic changes, structural outlines and abstract semantic features contained in the event modal data are extracted to obtain the event features.

6. The monocular depth estimation method based on image and event fusion in structured state space according to claim 1, characterized in that, The process of fusing the image features and the event features to obtain fused features includes: The image features and the event features are respectively subjected to linear mapping and depthwise convolution to obtain enhanced image features and enhanced event features; During the state evolution process, the enhanced image features and the enhanced event features exchange modulation information with each other to obtain cross-modal semantically guided image features and cross-modal semantically guided event features. The cross-modal semantic guidance image features and the cross-modal semantic guidance event features are multiplied element-wise and the residuals are added to obtain the initial cross-modal joint features; With the aim of enhancing the object boundary contour, depth abrupt regions, and local geometric details, the overlapping local windows of the cross-modal initial joint features are scanned in multiple directions to obtain the cross-modal intermediate joint features; The multiple feature channels contained in the cross-modal intermediate joint feature are scored to determine the key channels among the multiple feature channels, and the key channels are enhanced to obtain the fused feature.

7. The monocular depth estimation method based on structured state space image and event fusion according to claim 1, characterized in that, The step of generating a depth map based on the fusion features includes: The fused features are then subjected to progressive upsampling and feature reconstruction to obtain intermediate decoding features; Channel descriptors for each feature channel contained in the intermediate decoding feature are generated by global pooling, and the weights of each feature channel contained in the intermediate decoding feature are determined based on the channel descriptors. According to the weights, the multiple feature channels contained in the intermediate decoding features are divided into effective channels, invalid channels, and noise channels. The effective channels are enhanced, and the invalid and noise channels are suppressed to obtain the depth map.

8. A monocular depth estimation system based on image and event fusion in structured state space, characterized in that, The system is applied to a visual state space model, which includes: an event representation module, a frequency domain alignment module, a modal perception encoding module, a modal interaction module, and a decoding and output module. The event representation module is used to acquire image data and event stream data synchronized in the target scene within the target time window, and generate an event voxel grid based on the event stream data; The frequency domain alignment module is used to generate event modal data with injected global structure priors for the image data based on the event voxel grid, and to generate image modal data with injected motion-sensitive priors based on the event voxel grid for the image data. The modality-aware coding module is used to extract features from the image modality data and the event modality data respectively to obtain image features and event features. The image features include at least shallow to mid-level features of the image modality data, and the event features include at least mid- to deep-level features of the event modality data. The modal interaction module is used to fuse the image features and the event features to obtain fused features; The decoding and output module is used to generate a depth map of the target scene within the target time window based on the fusion features; The visual state space model is trained using training samples processed with normalized logarithmic depth during the training phase, and the loss value calculated from the true predicted depth and the true depth is used to optimize the visual state space model. The true predicted depth is the depth obtained by restoring the normalized predicted depth in the depth map output by the visual state space model.

9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-7.

Images