Monocular depth estimation method and product based on convolution compensation double-channel self-attention

Abstract

The application provides a monocular depth estimation method and product based on convolution compensation double-channel self-attention, and relates to the field of computer vision. The method comprises the following steps: converting an event stream of a target scene into a three-dimensional tensor representation; obtaining event image fusion multi-scale spatial features based on an image of the target scene and the three-dimensional tensor representation; modeling spatial context correlation of the event image fusion multi-scale spatial features in a spatial dimension by a context modeling self-attention branch to obtain a context modeling self-attention result; modeling modality correlation of the event image fusion multi-scale spatial features in a channel dimension by a modality fusion self-attention branch to obtain a modality fusion self-attention result; and performing pixel-level depth prediction by using the context modeling self-attention result and the modality fusion self-attention result to obtain a depth map, so as to fully mine the complementary characteristics between events and images, realize fine-grained depth fusion expression, and effectively improve the depth estimation precision and generalization ability.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a monocular depth estimation method and product based on convolution-compensated dual-channel self-attention. Background Technology

[0002] Among related technologies, monocular depth estimation has attracted widespread attention in fields such as 3D reconstruction and autonomous driving due to its low hardware cost and flexible deployment. In recent years, depth estimation methods that fuse event data and images have become an effective means to improve estimation accuracy in complex scenes due to the complementary characteristics of their respective modalities. Among them, image data can provide rich texture and semantic information, while event data has high dynamic range and microsecond-level temporal resolution, showing significant advantages in fast motion or low-light conditions. However, monocular depth estimation methods based on events and images often face the following significant problems in complex scenes:

[0003] The first problem is that both image and event data have their limitations when used as input signals alone. Images may lose key scene information in low light or fast motion, while event data, although having high dynamic range and temporal resolution, is difficult to capture complete texture information due to its sparsity, leading to inaccurate prediction results.

[0004] The second problem is that most existing fusion methods use convolutional or traditional Transformer structures. Convolutional structures have a limited receptive field and are prone to failure in scenes with occlusion or large-scale depth differences. While Transformer structures have the ability to model global dependencies, related methods have design deficiencies in modal interaction, often simply concatenating events with image features, resulting in low depth estimation accuracy. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a monocular depth estimation method and product based on convolution-compensated dual-channel self-attention, aiming to overcome or at least partially solve the aforementioned problems.

[0006] The first aspect of this invention provides a monocular depth estimation method based on convolution-compensated dual-channel self-attention, the method comprising:

[0007] The event representation module transforms the event stream of the target scene into a three-dimensional tensor representation. This three-dimensional tensor representation retains the temporal dynamic information of the events and also has a spatial structure aligned with the image modality.

[0008] The event and image fusion encoding module extracts features from the image of the target scene and the three-dimensional tensor representation to obtain multi-scale spatial features of event image fusion.

[0009] By using the context modeling self-attention branch in the event and image fusion coding module, the event image fusion multi-scale spatial features are utilized to model spatial context correlation in the spatial dimension, thereby obtaining the context modeling self-attention result.

[0010] By using the modal fusion self-attention branch in the event and image fusion coding module, the modal correlation between the image and the event is modeled in the channel dimension using the multi-scale spatial features of the event image fusion, and the modal fusion self-attention result is obtained.

[0011] The deconvolution decoding module uses the context modeling self-attention result and the modality fusion self-attention result to perform pixel-level depth prediction to obtain a depth map. The resolution of the depth map is the same as the resolution of the image of the target scene.

[0012] A second aspect of the present invention provides a monocular depth estimation system based on convolution-compensated dual-channel self-attention, the system comprising:

[0013] The event processing unit is used to convert the event stream of the target scene into a three-dimensional tensor representation through the event representation module. The three-dimensional tensor representation retains the temporal dynamic information of the event and also has a spatial structure aligned with the image modality.

[0014] The feature extraction unit is used to extract features from the image of the target scene and the three-dimensional tensor representation through the event and image fusion encoding module to obtain multi-scale spatial features of event image fusion.

[0015] The spatial modeling unit is used to model spatial context correlation in the spatial dimension by using the context modeling self-attention branch in the event and image fusion coding module, and to obtain the context modeling self-attention result by using the multi-scale spatial features of the event image fusion.

[0016] The channel modeling unit is used to model the modal correlation between the image and the event in the channel dimension by using the modal fusion self-attention branch in the event and image fusion coding module and the multi-scale spatial features of the event image fusion, thereby obtaining the modal fusion self-attention result.

[0017] The depth prediction unit is used to perform pixel-level depth prediction using the context modeling self-attention result and the modality fusion self-attention result through the deconvolution decoding module to obtain a depth map, the resolution of which is the same as the resolution of the image of the target scene.

[0018] A third aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the monocular depth estimation method based on convolution-compensated dual-channel self-attention as described in the first aspect of the present invention.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the monocular depth estimation method based on convolution-compensated dual-channel self-attention as described in the first aspect of the present invention.

[0020] In the monocular depth estimation method based on convolutional compensation dual-channel self-attention proposed in this invention, a unified convolutional-self-attention architecture is adopted. After encoding the event data (i.e., event stream) into a three-dimensional tensor representation, it is combined with image data for unified feature modeling. Feature extraction is performed on the image and three-dimensional tensor representation of the target scene to obtain multi-scale spatial features of event image fusion. Then, by introducing an event and image fusion encoding module that includes a context modeling self-attention mechanism and a modality fusion self-attention mechanism, spatial context and cross-modal channel correlation are modeled respectively. The complementary characteristics between events and images are fully explored to achieve fine-grained depth fusion expression, which effectively improves the depth estimation accuracy and generalization ability in complex, occluded, and multi-scale scenes. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the steps of a monocular depth estimation method based on convolution-compensated dual-channel self-attention, according to an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram illustrating a framework for monocular depth estimation by fusing events and images, according to an embodiment of the present invention.

[0024] Figure 3 This is a flowchart illustrating a monocular depth estimation process that fuses events and images, according to an embodiment of the present invention.

[0025] Figure 4 This is a block diagram of a monocular depth estimation system based on convolution-compensated dual-channel self-attention, provided in an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Currently, for monocular depth estimation of events and images, CNN architectures are widely used in related fusion methods due to their efficient local modeling capabilities. Some studies have also attempted to introduce Transformers to enhance global modeling capabilities. SRFNet, for example, designs an interaction module at the spatial attention level to capture consistent regions between images and events, thereby improving fusion results. Transformer-based methods, on the other hand, uniformly use a single Transformer architecture to encode both modalities and leverage ConvLSTM to process temporal information. However, these methods still suffer from the following problems, limiting their application in real-world complex scenes: First, CNN-based methods are limited by the convolutional receptive field, making it difficult to model long-distance cross-regional depth correlations, resulting in poor performance in occluded or large-scale depth variation scenes. Second, existing Transformer fusion schemes generally employ a modal feature concatenation strategy, ignoring fine-grained information interaction between modalities at the channel dimension, easily introducing redundancy or noise, resulting in coarse fusion results, especially noticeable in high-noise or asymmetric modal information scenes. Furthermore, schemes employing temporal modeling techniques such as ConvLSTM suffer from the vanishing gradient problem in long-term dependency modeling, further impacting model stability and performance. In other words, related event-image fusion monocular depth estimation methods still face challenges in terms of coordinated local and global modeling, channel-level modality fusion, and model robustness.

[0029] Based on this, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, this invention proposes a monocular depth estimation method based on convolutionally compensated dual-channel self-attention. This method proposes a unified depth estimation framework that integrates local convolution and global Transformer structures, combining parallel spatial context modeling attention and channel modality fusion mechanisms. This effectively improves the depth estimation accuracy and generalization ability in complex, occluded, and multi-scale scenes. In other words, this invention aims to improve the accuracy of monocular depth estimation in complex scenes, especially under challenging conditions such as low illumination, fast motion, and occlusion, by fully leveraging the complementarity between image and event data. The core idea is to encode event data into a dense representation and then perform unified feature modeling with image data; using a convolutionally compensated dual-channel self-attention mechanism as the core unit of the encoder, a fusion depth estimation network architecture is constructed to achieve more robust and accurate depth estimation.

[0030] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a monocular depth estimation method based on convolution-compensated dual-channel self-attention, as shown in an embodiment of the present invention. Figure 1 As shown, the monocular depth estimation method based on convolution-compensated dual-channel self-attention provided in this embodiment includes at least the following steps:

[0031] Step S11: Transform the event flow of the target scene into a three-dimensional tensor representation through the event representation module.

[0032] In this embodiment, for the target scene to be depth estimated, the event stream and image of the target scene can be acquired. The event stream of the target scene can be encoded into a structured input using an event representation module, constructing a dense spatiotemporal event tensor, which facilitates fusion with image modalities. That is, the event representation module converts the event stream of the target scene into a three-dimensional tensor representation. This three-dimensional tensor representation retains the temporal dynamics of the events and also possesses a spatial structure aligned with the image modalities, providing high-quality event input for subsequent depth estimation tasks.

[0033] Step S12: Through the event and image fusion encoding module, feature extraction is performed on the image of the target scene and the three-dimensional tensor representation to obtain multi-scale spatial features of event image fusion.

[0034] In this embodiment, joint modeling and deep fusion of image modalities and event modalities can be achieved through an event and image fusion coding module. This module includes at least downsampling convolution and a convolution-compensated dual-channel self-attention branch. By extracting and fusing modal data through this module, which uses the convolution-compensated dual-channel self-attention branch as its core, multi-scale semantic features are extracted while allowing for full interaction between different modalities.

[0035] Specifically, feature extraction can be performed on the image of the target scene and its corresponding 3D tensor representation through downsampling convolution to obtain multi-scale spatial features of the event image fusion, thereby preserving key structural information while reducing computational cost. For example, the image of the target scene and its corresponding 3D tensor representation can be input into a shallow convolutional network in downsampling convolution, and 3×3 convolution can be used to achieve feature extraction and spatial downsampling, resulting in multi-scale spatial features of the event image fusion.

[0036] Step S13: By using the context modeling self-attention branch in the event and image fusion coding module, the event image fusion multi-scale spatial features are used to model spatial context correlation in the spatial dimension, and the context modeling self-attention result is obtained.

[0037] In this embodiment, a dual-channel self-attention branch is obtained by inputting multi-scale spatial features of event image fusion into a convolutionally compensated branch. This convolutionally compensated dual-channel self-attention branch includes a parallel context modeling self-attention branch and a modality fusion self-attention branch. Specifically, the context modeling self-attention branch in the event and image fusion coding module can utilize the multi-scale spatial features of event image fusion to model spatial context correlation in the spatial dimension, thereby obtaining the context modeling self-attention result and enhancing the understanding of occluded regions and large-scale structures.

[0038] Step S14: Using the modal fusion self-attention branch in the event and image fusion coding module, the modal correlation between the image and the event is modeled in the channel dimension by utilizing the multi-scale spatial features of the event image fusion to obtain the modal fusion self-attention result.

[0039] In this embodiment, the modal fusion self-attention branch in the event and image fusion coding module can be used to leverage the multi-scale spatial features of event-image fusion to model the modal correlation between images and events in the channel dimension, thereby obtaining the modal fusion self-attention result and achieving fine-grained fusion representation. This embodiment, through this spatial-channel dual-dimensional collaborative modeling, can effectively improve the expressive power of multimodal fusion.

[0040] Step S15: Using the deconvolution decoding module, pixel-level depth prediction is performed using the context modeling self-attention result and the modality fusion self-attention result to obtain a depth map.

[0041] In this embodiment, after obtaining the context modeling self-attention results and the modality fusion self-attention results, the deconvolution decoding module can be used to perform step-by-step upsampling and reconstruction of the encoded features, restore the image spatial structure, and output a depth prediction map with the same resolution as the original image. This embodiment can use the deconvolution decoding module to perform pixel-level depth prediction based on the context modeling self-attention results and the modality fusion self-attention results to obtain a depth map of the target scene. The resolution of this depth map is the same as the resolution of the target scene image, thus achieving high-quality pixel-level depth prediction.

[0042] This embodiment proposes a unified convolution-compensated dual-attention depth estimation framework that fuses image and event data to address the insufficient estimation accuracy of traditional single-modality or simple fusion methods in dynamic environments, occluded scenes, and low-light conditions. First, the sparse asynchronous event stream is transformed into a dense and structured 3D tensor representation, thereby preserving the spatiotemporal dynamic information of the event data and achieving spatial alignment with the image modality, thus enhancing the basic expressive power of multimodal fusion. Second, through convolution-compensated dual-channel self-attention branches, contextual relationships are modeled in the spatial dimension and modal correlations are modeled in the channel dimension, fully exploring the complementary characteristics between events and images to achieve fine-grained depth fusion representation. Thus, the unified fusion structure of this embodiment effectively improves depth estimation performance in dynamic, complex, occluded, multi-scale scenes, and multi-light-varying environments, while balancing real-time performance and accuracy, and possesses good engineering application potential.

[0043] In conjunction with the above embodiments, in one implementation, the present invention also provides a monocular depth estimation method based on convolution-compensated dual-channel self-attention. In this method, in addition to the steps described above, steps S21 to S22 may be included, and step S15 may specifically include step S23:

[0044] Step S21: Through the detail compensation convolution module in the event and image fusion coding module, global average pooling and max pooling are performed on the multi-scale spatial features of the event image, and a spatial attention map is generated using convolution as the detail compensation convolution result.

[0045] In this embodiment, to compensate for the shortcomings of the Transformer structure in texture and edge detail modeling, a detail-compensating convolutional module is introduced into the event and image fusion coding module. That is, the event and image fusion coding module also includes a detail-compensating convolutional module. This detail-compensating convolutional module employs a spatial attention mechanism to enhance local structural features, compensating for the shortcomings in detail recovery of the self-attention mechanisms used in the context modeling self-attention branch and the modality fusion self-attention branch.

[0046] Specifically, the event image can be fused with multi-scale spatial features and input into the detail compensation convolution module. The detail compensation convolution module processes the fused multi-scale spatial features of the event image through global average pooling and max pooling, extracts spatially significant regions, and uses convolution to generate a spatial attention map. This spatial attention map is then used as the detail compensation convolution result output by the detail compensation convolution module to emphasize edge and detail features, effectively improving the accuracy and clarity of the final depth map.

[0047] In an optional example, the detail-compensated convolution module is used to enhance local structure modeling capabilities, compensating for the limitations of self-attention in representing textures and edges. The detail-compensated convolution module fuses multi-scale spatial features for event images, obtains a saliency response map through global average pooling and max pooling, and processes the saliency response map using convolution and activation functions to generate a spatial attention map.

[0048] Step S22: Using the context modeling self-attention result as the first weight and the modality fusion self-attention result as the second weight, the detail compensation convolution result is weighted and enhanced using the first weight and the second weight to obtain the final event image fusion result.

[0049] In this embodiment, after obtaining the detail-compensated convolution result, the input features of the deconvolution decoding module are weighted and enhanced based on the detail-compensated convolution result, thereby improving the depth estimation performance in edge and detail regions. Specifically, the context modeling self-attention result can be used as the first weight, and the modality fusion self-attention result can be used as the second weight. The detail-compensated convolution result is then weighted and enhanced using the first and second weights to obtain the final event image fusion result.

[0050] In an optional example, the final event image fusion result = context modeling self-attention result * detail compensation convolution result + modality fusion self-attention result * detail compensation convolution result.

[0051] Step S23: Using the deconvolution decoding module, pixel-level depth prediction processing is performed using the final event image fusion result to obtain the depth map.

[0052] In this embodiment, after obtaining the final event image fusion result, the final event image fusion result is input to the deconvolution decoding module. The deconvolution decoding module performs upsampling and reconstruction of the final event image fusion result step by step to perform pixel-level depth prediction and obtain the depth map of the target scene.

[0053] In this embodiment, to improve the modeling accuracy of texture and edge regions, a detail-compensated convolutional module is introduced during the encoding stage. This module enhances local structural features through a spatial attention mechanism, compensating for the shortcomings of autonomous attention structures in detail recovery. This embodiment strengthens the representation of local texture features and edge information through the detail-compensated convolutional module, thereby improving the depth estimation accuracy in occluded areas and complex scenes.

[0054] In conjunction with any of the above embodiments, the present invention also provides a monocular depth estimation method based on convolution-compensated dual-channel self-attention, in which step S11 may specifically include steps S31 to S32:

[0055] Step S31: Construct a voxel mesh in the form of a three-dimensional tensor, and divide the event flow into a fixed number of time intervals along the time axis.

[0056] Since the event camera outputs a sparse asynchronous event stream of the target scene, directly using it for network training would lead to problems of sparse representation and incomplete structure. Therefore, this embodiment introduces a voxelization encoding mechanism to transform the event stream of the target scene into a structured 3D tensor representation. Specifically, the event representation module mainly performs structuring processing on the sparse asynchronous event stream of the target scene output by the event camera to construct dense input data aligned with the image modality.

[0057] Specifically, the event stream in this embodiment includes multiple quadruples. The four elements of a quadruple (x, y, t, p) represent: the location coordinates (x, y) of the event occurrence, the time t, and the polarity p, respectively. To fully utilize the temporal information in the event data and achieve spatial alignment, this embodiment employs a voxelization encoding strategy to construct a three-dimensional tensor voxel mesh. The event stream is divided into a fixed number of time segments along the time axis. Here, H and W represent the spatial resolution of the target scene image, and B represents the number of time segments (bins).

[0058] Step S32: Based on the time period, use time normalization to map each event in the event stream to the corresponding voxel grid.

[0059] In this embodiment, based on a time period, a time normalization method can be used to map each event in the event stream of the target scene to a corresponding voxel grid. Then, the grid values ​​are accumulated using bilinear interpolation to obtain a 3D tensor representation of the event stream of the target scene. This process not only fully preserves the temporal dynamics and spatiotemporal characteristics of the event stream (event data), but also gives the event data a spatial structure that can be aligned with the image, providing high-quality modal input for subsequent depth estimation tasks.

[0060] In a specific example, the event stream can be mapped to the corresponding voxel grid according to the time normalization formula, and the grid values ​​can be accumulated using bilinear interpolation to obtain a three-dimensional tensor representation, which can be formalized as follows:

[0061] ;

[0062] The normalization formula is: ( );

[0063] It is a three-dimensional tensor representation; for the th One event , Its timestamp, ( ) represents its pixel position. Its polarity; For cumulative time windows, The timestamp of the first event that occurred within the window. This is the normalized timestamp.

[0064] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a monocular depth estimation method based on convolution-compensated dual-channel self-attention. In this method, and specifically, step S13 above may include steps S41 to S43:

[0065] Step S41: Based on the query, key, and value of the j-th attention head, obtain the self-attention result of the j-th attention head.

[0066] In this embodiment, the context modeling self-attention branch employs a local window partitioning strategy. It partitions the event image into local windows based on multi-scale spatial features in the spatial dimension, capturing regional contextual dependencies to enhance feature representation capabilities in occluded scenes or scenes with large-scale depth variations. Specifically, within the i-th local window, the query, key, and value of the j-th attention head are respectively... ; i and j are both integers greater than 0, j∈[1, ..., j] ].

[0067] In this embodiment, in the context modeling self-attention branch, for the i-th local window obtained by partitioning, the self-attention result of the j-th attention head in the i-th local window can be obtained according to the query, key, and value of the j-th attention head.

[0068] For example, the self-attention result of the j-th attention head in the i-th local window can be represented by the following formula:

[0069] .

[0070] Step S42: According to the first to the last The self-attention results of each attention head are used to obtain the self-attention result of the i-th local window.

[0071] In this embodiment, each local window has Each attention point. For the i-th local window, we can refer to the first to the last attention points. The self-attention results of the first to the last attention heads are used to obtain the self-attention results of the i-th local window. For example, the self-attention results of the first to the last attention heads are used to obtain the self-attention results of the i-th local window. The self-attention results of each attention head are concatenated to obtain the self-attention result of the i-th local window, which can be expressed by the following formula:

[0072] .

[0073] Step S43: Obtain the context modeling self-attention result based on the self-attention results of each of the divided local windows.

[0074] In this embodiment, the context modeling self-attention branch can obtain the context modeling self-attention result based on the self-attention results of each of the divided local windows. For example, a total of [number] local windows are divided. A local window can be used to represent the context-modeling self-attention result using the following formula:

[0075] .

[0076] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a monocular depth estimation method based on convolution-compensated dual-channel self-attention. In this method, S14 specifically includes steps S51 to S52:

[0077] Step S51: Obtain the self-attention result of the i-th channel group based on the image-level query, image-level key, and image-level value of the i-th channel group.

[0078] In this embodiment, the modality fusion self-attention branch employs a channel grouping mechanism. This mechanism groups the multi-scale spatial features of the event-image fusion along the channel dimension and models the interaction between event modalities and image modalities along the channel dimension. This improves the efficiency of cross-modal dependency modeling and reduces computational overhead through the channel grouping mechanism. Specifically, in the i-th channel group, the image-level query, image-level key, and image-level value of the i-th channel group are respectively... , , ; i is an integer, and i∈[1, ..., ... ].

[0079] In this embodiment, in the modality fusion self-attention branch, for the i-th channel group obtained by partitioning, an image-level query can be performed based on the i-th channel group. Image-level keys and image level values We obtain the self-attention result for the i-th channel group.

[0080] For example, the self-attention result of the i-th channel group can be expressed by the following formula:

[0081] .

[0082] Step S52: Based on the self-attention results of each of the divided channel groups, obtain the modality fusion self-attention result.

[0083] In this embodiment, the modality fusion self-attention branch can obtain the modality fusion self-attention result by grouping the self-attention results of each of the divided channels. For example, a total of [number] channels are divided. The modality fusion self-attention result can be represented by the following formula: (The formula is missing from the provided text.)

[0084] .

[0085] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a monocular depth estimation method based on convolution-compensated dual-channel self-attention. In this method, in addition to the steps described above, steps S61 to S62 may also be included:

[0086] Step S61: Input the sample image of the sample scene carrying the depth map label and the sample event stream of the sample scene into the convolution-compensated dual-channel self-attention model to be trained to obtain the sample predicted depth map.

[0087] In this embodiment, the monocular depth estimation method based on convolutionally compensated dual-channel self-attention can be implemented based on a trained convolutionally compensated dual-channel self-attention model. The trained convolutionally compensated dual-channel self-attention model is obtained through supervised training of the model to be trained, thereby improving prediction accuracy and ultimately outputting a high-resolution depth map.

[0088] In this embodiment, sample images and sample event streams of a sample scene can be input into the convolutionally compensated dual-channel self-attention model to be trained, resulting in a sample predicted depth map output by the model. The sample images of the sample scene carry depth map labels. The sample scene can be understood as the target scene during model training, the sample images of the sample scene can be understood as images of the target scene during model training, the sample event stream of the sample scene can be understood as the event stream of the target scene during model training, and the sample predicted depth map can be understood as the depth map during model training.

[0089] In this embodiment, the specific method by which the convolution-compensated dual-channel self-attention model to be trained obtains the sample predicted depth map based on the sample image and sample event stream of the sample scene is the same as or similar to the method of obtaining the depth map based on the event stream and image of the target scene in any of the aforementioned embodiments of the monocular depth estimation method based on convolution-compensated dual-channel self-attention, and will not be repeated here.

[0090] Step S62: Based on the difference between the sample predicted depth map and the depth map label, update the network parameters of the training-based convolution compensated dual-channel self-attention model to obtain the trained convolution compensated dual-channel self-attention model.

[0091] In this embodiment, the difference between the predicted depth map and the depth map label can be determined based on the sample predicted depth map and its corresponding depth map label obtained during training. Then, based on the difference between the sample predicted depth map and the depth map label, the network parameters of the training-based convolutionally compensated dual-channel self-attention model are updated using a loss function, resulting in the trained convolutionally compensated dual-channel self-attention model. The trained convolutionally compensated dual-channel self-attention model includes at least: an event representation module, an event and image fusion encoding module, and a deconvolution decoding module.

[0092] In one alternative example, the training model based on convolutionally compensated dual-channel self-attention may include: an event representation module (pre-trained and with fixed parameters), an event and image fusion encoding module (pre-trained), and a deconvolution decoding module (pre-trained). Based on the difference between the sample-predicted depth map and the depth map label, the network parameters of the event and image fusion encoding module and the deconvolution decoding module are updated using a loss function, thus obtaining the trained convolutionally compensated dual-channel self-attention model. Alternatively, the training model may include: an event representation module (pre-trained and with fixed parameters), an event and image fusion encoding module (pre-trained), and a deconvolution decoding module (pre-trained and with fixed parameters), thus updating the network parameters of the event and image fusion encoding module and the deconvolution decoding module based on the difference between the sample-predicted depth map and the depth map label, thereby obtaining the trained convolutionally compensated dual-channel self-attention model. The difference between the predicted depth map and the depth map label is used to update the network parameters of the event and image fusion encoding module to be trained through a loss function, thereby obtaining a trained convolution-compensated dual-channel self-attention model. Alternatively, the model to be trained may include an event representation module, an event and image fusion encoding module, and a deconvolution decoding module. Based on the difference between the sample predicted depth map and the depth map label, the network parameters of the event representation module, the event and image fusion encoding module, and the deconvolution decoding module to be trained are updated through a loss function, thereby obtaining a trained convolution-compensated dual-channel self-attention model. This embodiment does not limit the scope of the model.

[0093] In an optional example, the loss function of this embodiment is defined as the joint loss of L1 and L2, which can be expressed as:

[0094] ;

[0095] in, That is, the error between the predicted depth and the actual depth. These are the actual depth and the predicted depth, respectively. That is, the difference between the sample predicted depth map and the depth map label, where n is the number of effective pixels.

[0096] In this embodiment, a weighted combination of L1 and L2 loss is used in the loss function, where the L1 term promotes structural accuracy and the L2 term enhances numerical stability. This loss considers both point-to-point numerical errors and emphasizes the matching of the overall structure, thereby improving local prediction accuracy while ensuring the rationality of the global structure.

[0097] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a monocular depth estimation method based on convolution-compensated dual-channel self-attention. In this method, step S12 specifically includes steps S71 to S72, and step S23 specifically includes step S73:

[0098] Step S71: Extract features from the image of the target scene and the three-dimensional tensor representation using the multi-level convolutional unit, and then concatenate the extracted features to obtain concatenated features.

[0099] In this embodiment, the event and image fusion encoding module includes multi-level convolutional units, which are included in downsampling convolutions. This embodiment allows the target scene image and its 3D tensor representation to be input into the multi-level convolutional units respectively. The multi-level convolutional units then extract features from both the target scene image and its 3D tensor representation, and concatenate the extracted features to obtain a concatenated feature set.

[0100] Step S72: Downsample the stitched features to obtain the multi-scale spatial features of the event image fusion.

[0101] In this embodiment, after the multi-level convolutional units obtain the spliced ​​features, the spliced ​​features can be downsampled to obtain the fused features, that is, the multi-scale spatial features of the event image are fused.

[0102] Step S73: Using the multi-level deconvolution unit, pixel-level depth prediction processing is performed using the final event image fusion result to obtain the depth map.

[0103] In this embodiment, the deconvolution decoding module can adopt a symmetrical U-Net-style progressive upsampling decoding structure design to restore spatial resolution. This deconvolution decoding module includes multiple levels of deconvolution units (e.g., up to 5 levels, with no specific limit on the number). Each level of deconvolution unit restores spatial resolution through transpose convolution. The multiple levels of deconvolution units are connected to each other using a skip connection mechanism (i.e., each level of deconvolution unit corresponds to and is connected to the convolution units within the same level of the multi-level deconvolution unit). The shallow features output by each level of the convolution unit are concatenated and fused with the deep semantic features output by the same level of the deconvolution unit in the multi-level deconvolution unit along the channel dimension, thereby enhancing the ability to restore edge and structural information. The skip connection is used to fuse the shallow high-resolution information from the encoding stage with the deep semantic features from the decoding stage, effectively enhancing detail recovery capabilities and supplementing detail information and spatial boundaries.

[0104] In this embodiment, after the final event image fusion result is input into the deconvolution decoding module, the final event image fusion result can be upsampled and reconstructed step by step through multi-level deconvolution units to perform pixel-level depth prediction and obtain the depth map of the target scene.

[0105] To evaluate the performance of this invention in monocular depth estimation by fusing events and images in complex dynamic scenes, one embodiment uses the MVSEC and DENSE public datasets for experimental verification to demonstrate its superior accuracy and robustness on both real and simulated datasets. This embodiment employs commonly used performance evaluation metrics for depth estimation, including mean error (Avg. Error), mean absolute error (Abs.Rel), and logarithmic mean squared error (RMSE log). The test environment for this embodiment is an Ubuntu 20.04 operating system equipped with two NVIDIA RTX 3090 graphics processors, each with 24GB of video memory and a core frequency of 1695MHz. Quantitative comparative tests were conducted on the aforementioned datasets, and the quantitative results are shown in Table 1. This embodiment outperforms the relevant advanced SRFNet methods on both the MVSEC and DENSE datasets, exhibiting lower error metrics and demonstrating excellent robustness. Furthermore, this embodiment can achieve a rendering speed of 40ms on an NVIDIA RTX 3090 graphics processor, meeting real-time requirements.

[0106] Table 1. Performance Evaluation Comparison of the Method in This Embodiment with Related Methods

[0107]

[0108] In one embodiment, such as Figure 2 As shown, Figure 2This is a schematic diagram illustrating a framework for monocular depth estimation by fusing events and images, as shown in an embodiment of the present invention. The framework for monocular depth estimation by fusing events and images includes at least: an event representation module, which uses voxelization encoding to transform the event stream into a structured 3D tensor representation, providing high-quality event input for subsequent depth estimation tasks; an event and image fusion encoding module, responsible for deep fusion modeling of image and event features. Specifically, it includes three parts: downsampling convolution, a convolution-compensated dual-channel self-attention branch, and detail-compensated convolution. The downsampling convolution module extracts multi-scale spatial features, reducing computation while retaining key structural information. Secondly, a convolution-compensated dual-channel self-attention branch module combines two attention mechanisms for parallel modeling: a context-modeling self-attention branch models spatial contextual relevance, enhancing the understanding of occluded regions and large-scale structures; and a modality fusion self-attention branch models the modal relevance of images and events in the channel dimension, achieving fine-grained fusion expression. The detail-compensated convolution module enhances local structure modeling capabilities, compensating for the deficiencies of self-attention in expressing textures and edges. The deconvolutional decoding module employs a U-Net-like progressive upsampling decoding structure to restore spatial resolution. It uses skip connections to effectively supplement detail information and spatial boundaries. The final output depth map maintains the same resolution as the input image, achieving high-quality pixel-level depth prediction. Additionally, a supervised training module optimizes the network during training, improving prediction accuracy and ultimately outputting a high-resolution depth map. In this embodiment, the decoding stage utilizes a multi-scale skip connection upsampling structure combined with an end-to-end supervised training mechanism to output a high-precision pixel-level depth map.

[0109] In one embodiment, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating a monocular depth estimation process for fusing events and images, as shown in an embodiment of the present invention. The workflow for monocular depth estimation by fusing events and images is as follows:

[0110] Step 1: Construct input representations for multimodal data to facilitate modality fusion: For event data, the original event stream is encoded into structured inputs to construct dense spatiotemporal event tensors, which facilitates modality fusion with images.

[0111] Step 2: Extract and fuse multimodal features, and encode and fuse them through convolutional downsampling and convolutional compensated dual attention branches: The encoder module with convolutional compensated dual attention branches (convolutional compensated dual-channel self-attention branches) as the core extracts and fuses modal data, and performs full interaction between different modalities while extracting multi-scale semantic features.

[0112] Step 3: Upsample using deconvolution, decode features and restore image size, output predicted depth value: Upsample using a deconvolution-based decoder module to restore image spatial structure.

[0113] Step 4: Train the network in a supervised manner to optimize prediction accuracy: During the network training process, a supervised mechanism is used to optimize the network, improve prediction accuracy, and finally output a high-resolution depth map.

[0114] In summary, this invention achieves deep fusion modeling of images and events through dense event representation, a unified fusion encoder structure, an efficient decoding structure, and a stable training mechanism, significantly improving the prediction accuracy and generalization ability of monocular depth estimation in dynamic, low-light, and complex structural scenes. Specifically, this invention first constructs a unified voxelized event input and image input structure to fully preserve the dynamic spatiotemporal characteristics of event data, while utilizing a convolution-compensated dual-channel self-attention mechanism to achieve deep fusion of multimodal information. Second, this invention introduces a detail-compensated convolution module, effectively compensating for the shortcomings of the self-attention mechanism in local structure modeling and improving the recovery ability of occluded and complex texture regions. Furthermore, through a U-Net-style decoder and supervised optimization strategy, this invention achieves high-precision and robust monocular depth estimation in various practical complex scenes. Experimental results fully demonstrate that this invention exhibits excellent performance in dynamic environments, occluded scenes, and low-light conditions, possessing good practical application potential and promotional value.

[0115] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0116] Based on the same inventive concept, one embodiment of the present invention provides a monocular depth estimation system based on convolution-compensated dual-channel self-attention. (Reference) Figure 4 , Figure 4 This is a block diagram of a monocular depth estimation system based on convolution-compensated dual-channel self-attention, provided in an embodiment of the present invention. Figure 4 As shown, the system includes:

[0117] The event processing unit is used to convert the event stream of the target scene into a three-dimensional tensor representation through the event representation module. The three-dimensional tensor representation retains the temporal dynamic information of the event and also has a spatial structure aligned with the image modality.

[0118] The feature extraction unit is used to extract features from the image of the target scene and the three-dimensional tensor representation through the event and image fusion encoding module to obtain multi-scale spatial features of event image fusion.

[0119] The spatial modeling unit is used to model spatial context correlation in the spatial dimension by using the context modeling self-attention branch in the event and image fusion coding module, and to obtain the context modeling self-attention result by using the multi-scale spatial features of the event image fusion.

[0120] The channel modeling unit is used to model the modal correlation between the image and the event in the channel dimension by using the modal fusion self-attention branch in the event and image fusion coding module and the multi-scale spatial features of the event image fusion, thereby obtaining the modal fusion self-attention result.

[0121] The depth prediction unit is used to perform pixel-level depth prediction using the context modeling self-attention result and the modality fusion self-attention result through the deconvolution decoding module to obtain a depth map, the resolution of which is the same as the resolution of the image of the target scene.

[0122] Optionally, the system further includes:

[0123] The compensation convolution unit is used to perform global average pooling and max pooling on the multi-scale spatial features of the event image fusion through the detail compensation convolution module in the event and image fusion coding module, and to generate a spatial attention map using convolution as the detail compensation convolution result; the detail compensation convolution module adopts a spatial attention mechanism to enhance local structural features and make up for the deficiencies in detail recovery of the self-attention mechanisms adopted by the context modeling self-attention branch and the modality fusion self-attention branch respectively.

[0124] The fusion processing unit is used to use the context modeling self-attention result as the first weight and the modality fusion self-attention result as the second weight, and to use the first weight and the second weight to perform weighted enhancement on the detail compensation convolution result to obtain the final event image fusion result.

[0125] The depth prediction unit includes:

[0126] The first prediction unit is used to perform pixel-level depth prediction processing using the final event image fusion result through the deconvolution decoding module to obtain the depth map.

[0127] Optionally, the event stream includes multiple quadruples, where the four elements of a quadruple represent the location coordinates, time, and polarity of the event occurrence; the event processing unit includes:

[0128] A construction unit is used to construct a voxel mesh in the form of a three-dimensional tensor, and to divide the event stream into a fixed number of time periods along the time axis. The three dimensions of the three-dimensional tensor are H, W, and B, where H and W represent the resolution of the image of the target scene, and B represents the number of time periods.

[0129] The normalization unit is used to map each event in the event stream to a corresponding voxel grid based on the time period using a time normalization method.

[0130] Optionally, the context modeling self-attention branch employs a local window partitioning strategy, dividing the event image into local windows based on the fusion of multi-scale spatial features in the spatial dimension, and capturing regional contextual dependencies in the spatial dimension to enhance feature representation capabilities in occluded scenes or scenes with large-scale depth changes; in the i-th local window, the query, key, and value of the j-th attention head are respectively ;

[0131] Spatial modeling unit, including:

[0132] The first processing unit is used to obtain the self-attention result of the j-th attention head based on the query, key, and value of the j-th attention head;

[0133] The second processing unit is used to process the data from the first to the second... The self-attention results of each attention head are used to obtain the self-attention result of the i-th local window;

[0134] The third processing unit is used to obtain the context modeling self-attention result based on the self-attention results of each of the divided local windows.

[0135] Optionally, the modality fusion self-attention branch employs a channel grouping mechanism, grouping the multi-scale spatial features of the event image fusion along the channel dimension, and performing interactive modeling of event modality and image modality; in the i-th channel group, the image-level query, image-level key, and image-level value of the i-th channel group are respectively , , ;

[0136] Channel modeling unit, including:

[0137] The fourth processing unit is used to obtain the self-attention result of the i-th channel group based on the image-level query, image-level key, and image-level value of the i-th channel group;

[0138] The fifth processing unit is used to obtain the modality fusion self-attention result based on the self-attention results of each of the divided channel groups.

[0139] Optionally, the system further includes:

[0140] The second prediction unit is used to input the sample image of the sample scene carrying depth map labels and the sample event stream of the sample scene into the convolution-compensated dual-channel self-attention model to be trained, so as to obtain the sample predicted depth map.

[0141] The training unit is used to update the network parameters of the convolution-compensated dual-channel self-attention model to be trained based on the difference between the sample predicted depth map and the depth map label, so as to obtain the trained convolution-compensated dual-channel self-attention model. The trained convolution-compensated dual-channel self-attention model includes at least the event representation module, the event and image fusion encoding module, and the deconvolution decoding module.

[0142] Optionally, the event and image fusion coding module includes multi-level convolutional units;

[0143] The feature extraction unit includes:

[0144] An extraction and stitching unit is used to extract features from the image of the target scene and the three-dimensional tensor representation through the multi-level convolutional unit, and to stitch the extracted features to obtain stitched features;

[0145] The downsampling unit is used to downsample the stitched features to obtain the multi-scale spatial features fused from the event image;

[0146] The deconvolution decoding module includes multiple levels of deconvolution units. Each level of deconvolution unit restores spatial resolution through transpose convolution operation. The multiple levels of deconvolution units and the multiple levels of convolution units adopt a skip connection mechanism to fuse the shallow features output by each level of convolution unit with the deep semantic features output by the same level of deconvolution unit in the channel dimension.

[0147] The first prediction unit includes:

[0148] The second prediction unit is used to perform pixel-level depth prediction processing using the final event image fusion result through the multi-level deconvolution unit to obtain the depth map.

[0149] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the monocular depth estimation method based on convolution-compensated dual-channel self-attention as described in any of the above embodiments of the present invention.

[0150] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps in the monocular depth estimation method based on convolution-compensated dual-channel self-attention described in any of the above embodiments of the present invention.

[0151] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0152] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0153] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0158] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0159] The above provides a detailed description of the monocular depth estimation method and product based on convolutional compensated dual-channel self-attention provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A monocular depth estimation method based on convolutional compensation double-channel self-attention, characterized in that, The method comprises: transforming, by an event representation module, an event stream of a target scene into a three-dimensional tensor representation that not only retains temporal dynamic information of the event but also has a spatial structure aligned with an image modality; extracting, by an event and image fusion encoding module, features from the three-dimensional tensor representation and an image of the target scene to obtain event and image fusion multi-scale spatial features; modeling, by a context modeling self-attention branch in the event and image fusion encoding module, spatial context correlation in a spatial dimension using the event and image fusion multi-scale spatial features to obtain a context modeling self-attention result; modeling, by a modality fusion self-attention branch in the event and image fusion encoding module, modality correlation between the image and the event in a channel dimension using the event and image fusion multi-scale spatial features to obtain a modality fusion self-attention result; performing, by a deconvolution decoding module, pixel-level depth prediction using the context modeling self-attention result and the modality fusion self-attention result to obtain a depth map, the resolution of the depth map being the same as that of the image of the target scene.

2. The monocular depth estimation method based on convolution compensation double-channel self-attention of claim 1, characterized in that, The method further comprises: performing, by a detail compensation convolution module in the event and image fusion encoding module, global average pooling and maximum pooling on the event and image fusion multi-scale spatial features, and generating a spatial attention map using convolution as a detail compensation convolution result; the detail compensation convolution module adopts a spatial attention mechanism to enhance local structural features and make up for the deficiencies of the self-attention mechanisms adopted by the context modeling self-attention branch and the modality fusion self-attention branch in detail recovery; weighting and enhancing the detail compensation convolution result using a first weight of the context modeling self-attention result and a second weight of the modality fusion self-attention result to obtain a final event and image fusion result; performing, by a deconvolution decoding module, pixel-level depth prediction using the context modeling self-attention result and the modality fusion self-attention result to obtain a depth map, comprising: performing, by the deconvolution decoding module, pixel-level depth prediction processing using the final event and image fusion result to obtain the depth map.

3. The monocular depth estimation method based on convolution compensation double-channel self-attention of claim 1, characterized in that, The event stream comprises a plurality of four-tuples, and the four elements in one four-tuple represent the position coordinates, time and polarity of an event; transforming, by an event representation module, an event stream of a target scene into a three-dimensional tensor representation, comprising: constructing a voxel grid in the form of a three-dimensional tensor, and dividing the event stream into a fixed number of time periods along a time axis, the three dimensions of the three-dimensional tensor being H, W and B, wherein H and W represent the resolution of the image of the target scene, and B represents the number of the time periods; based on the time periods, mapping each event in the event stream to a corresponding voxel grid in a time normalization manner. The event stream comprises a plurality of four-tuples, and the four elements in one four-tuple represent the position coordinates, time and polarity of an event; transforming, by an event representation module, an event stream of a target scene into a three-dimensional tensor representation, comprising: constructing a voxel grid in the form of a three-dimensional tensor, and dividing the event stream into a fixed number of time periods along a time axis, the three dimensions of the three-dimensional tensor being H, W and B, wherein H and W represent the resolution of the image of the target scene, and B represents the number of the time periods; based on the time periods, mapping each event in the event stream to a corresponding voxel grid in a time normalization manner.

4. The monocular depth estimation method based on convolution compensation double-channel self-attention of claim 1, characterized in that, The context modeling self-attention branch adopts a local window division strategy, performs local window division on the event image fusion multi-scale spatial features in the spatial dimension, and captures regional context dependence in the spatial dimension to enhance the feature representation capability in the occlusion scene or large-scale depth change scene; in the i th local window divided, the query, key and value of the j th attention head are respectively ; The event image fusion multi-scale spatial feature is utilized by the context modeling self-attention branch in the event and image fusion encoding module to model spatial context correlation in a spatial dimension to obtain a context modeling self-attention result, including: The self-attention result of the jth attention head is obtained according to the query, key and value of the jth attention head; According to the self-attention results of the first to the i-th local window, the self-attention results of the i-th local window are obtained. The context modeling self-attention result is obtained according to the self-attention results of all the divided local windows.

5. The monocular depth estimation method based on convolution compensation double-channel self-attention of claim 1, characterized in that, The modal fusion self-attention branch adopts a channel grouping mechanism to group the event image fusion multi-scale spatial feature in a channel dimension and model interaction between event and image modalities; In the divided i-th channel group, the image-level query, the image-level key and the image-level value of the i-th channel group are respectively , , ; The modal fusion self-attention result is obtained by the modal fusion self-attention branch in the event and image fusion encoding module, which utilizes the event image fusion multi-scale spatial feature to model the modal correlation between image and event in a channel dimension, including: The self-attention result of the ith channel group is obtained according to the image-level query, image-level key and image-level value of the ith channel group. The modal fusion self-attention result is obtained according to the self-attention results of all the divided channel groups.

6. The monocular depth estimation method based on convolution compensation double-channel self-attention of claim 1, characterized in that, The method further includes: The sample image of a sample scene carrying a depth map label and the sample event stream of the sample scene are input into the convolution compensation double-channel self-attention model to be trained to obtain a sample predicted depth map; The network parameters of the convolution compensation double-channel self-attention model to be trained are updated according to the difference between the sample predicted depth map and the depth map label to obtain a trained convolution compensation double-channel self-attention model, which at least includes the event representation module, the event and image fusion encoding module and the deconvolution decoding module.

7. The monocular depth estimation method based on convolution compensation double-channel self-attention of claim 2, characterized in that, The event and image fusion encoding module includes a multi-level convolution unit; The image of a target scene and the three-dimensional tensor representation are subjected to feature extraction by the event and image fusion encoding module to obtain event image fusion multi-scale spatial features, including: The image of the target scene and the three-dimensional tensor representation are subjected to feature extraction by the multi-level convolution unit, and the extracted features are spliced to obtain spliced features; The spliced features are down-sampled to obtain the event image fusion multi-scale spatial features; The deconvolution decoding module includes a multi-level deconvolution unit, each level of which restores spatial resolution by a transpose convolution operation, and the multi-level deconvolution unit and the multi-level convolution unit adopt a skip connection mechanism to fuse the shallow features output by each level of the convolution unit with the deep semantic features output by the deconvolution unit of the same level in a channel dimension. The depth map is obtained by the deconvolution decoding module by utilizing the final event image fusion result for pixel-level depth prediction processing, including: The depth map is obtained by the multi-level deconvolution unit by utilizing the final event image fusion result for pixel-level depth prediction processing.

8. A monocular depth estimation system based on convolutional compensated bi-channel self-attention, characterized in that, The system includes: An event processing unit is configured to convert an event stream of a target scene into a three-dimensional tensor representation by an event representation module, which not only retains temporal dynamic information of the event but also has a spatial structure aligned with an image modality; A feature extraction unit is configured to extract features of an image of the target scene and the three-dimensional tensor representation by an event and image fusion encoding module to obtain event-image fusion multi-scale spatial features; A spatial modeling unit is configured to model spatial context correlation in a spatial dimension by a context modeling self-attention branch in the event and image fusion encoding module using the event-image fusion multi-scale spatial features to obtain a context modeling self-attention result; A channel modeling unit is configured to model modality correlation between the image and the event in a channel dimension by a modality fusion self-attention branch in the event and image fusion encoding module using the event-image fusion multi-scale spatial features to obtain a modality fusion self-attention result; A depth prediction unit is configured to perform pixel-level depth prediction by a deconvolution decoding module using the context modeling self-attention result and the modality fusion self-attention result to obtain a depth map, a resolution of the depth map being the same as a resolution of the image of the target scene.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the monocular depth estimation method based on convolution compensation double-channel self-attention according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the monocular depth estimation method based on convolution compensation double-channel self-attention according to any one of claims 1-7.

Images