Lightweight self-supervised monocular depth estimation method and device based on cross-sequence interaction

Abstract

The present application relates to the technical field of computer vision, and provide a kind of based on the lightweight self-supervised monocular depth estimation method and device of cross-sequence interaction, method includes: by continuous spatial pooling mechanism module from initial feature map capture global long-range dependence relationship and obtain sequence reorganization feature;By high-dimensional sequence feature interaction module, sequence reorganization feature is carried out fine-grained extraction and decoding to generate multi-scale depth map;The relative attitude change information of target frame and adjacent frame is acquired;Based on multi-scale depth map and relative attitude change information, reconstruct the image and construct image reconstruction loss, and the network is self-supervised training in combination with image reconstruction loss.The present application captures global dependence by continuous spatial pooling mechanism, alleviates the problem of receptive field limitation;Local fine-grained features are extracted using high-dimensional sequence feature interaction, edge sharpness is ensured, and the problem of balancing accuracy and efficiency and long-distance dependence modeling is solved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a lightweight self-supervised monocular depth estimation method and apparatus based on cross-sequence interaction. Background Technology

[0002] Monocular depth estimation is a key technology in applications such as autonomous driving and robot navigation. Since supervised learning relies on expensive real depth data, self-supervised methods achieved through image reconstruction are gradually becoming a research hotspot.

[0003] Existing self-supervised methods are mostly based on convolutional neural networks (CNNs) or transformers. CNN-based methods are limited by local receptive fields and have difficulty modeling long-range dependencies; while transformer-based methods have global perception capabilities, they have high computational complexity and lack spatial information extraction capabilities, making them difficult to deploy on resource-constrained platforms. Summary of the Invention

[0004] This invention provides a lightweight self-supervised monocular depth estimation method and apparatus based on cross-sequence interaction to address the deficiencies in the prior art.

[0005] This invention provides a lightweight self-supervised monocular depth estimation method based on cross-sequence interaction, comprising the following steps:

[0006] Construct a cross-sequence interactive deep network that includes a depth estimation network and a pose estimation network. The depth estimation network integrates a continuous spatial pooling mechanism module and a high-dimensional sequence feature interaction module.

[0007] The initial feature map of the target frame image is extracted, and the global long-range dependencies are captured from the initial feature map through the continuous spatial pooling mechanism module to obtain sequence recombination features;

[0008] The high-dimensional sequence feature interaction module is used to extract the sequence recombination features in a fine-grained manner, and the extracted fine-grained features are decoded to generate a multi-scale depth map.

[0009] The pose estimation network is used to obtain the relative pose change information between the target frame image and adjacent frame images;

[0010] The image is reconstructed based on the multi-scale depth map and the relative pose change information, and an image reconstruction loss is constructed based on the reconstructed image and the target frame image. After self-supervised training of the cross-sequence interactive depth network in combination with the image reconstruction loss, the depth estimation result of the target frame image is output using the trained depth estimation network.

[0011] According to the present invention, a lightweight self-supervised monocular depth estimation method based on cross-sequence interactions is provided, wherein the step of capturing global long-range dependencies from the initial feature map through the continuous spatial pooling mechanism module to obtain sequence recombination features specifically includes:

[0012] Perform a spatial-to-channel dimension transformation operation on the initial feature map, rearrange adjacent spatial pixel blocks to the channel dimension, and generate an intermediate feature map with channel dimension expansion;

[0013] The intermediate feature map is scanned from multiple directions to obtain a serialized representation, and linear computation is performed on the serialized representation to extract sequence features with global long-range dependencies.

[0014] The sequence features are restored to a two-dimensional spatial structure and fused with the features processed by convolution to obtain the sequence recombination features.

[0015] According to the present invention, a lightweight self-supervised monocular depth estimation method based on cross-sequence interaction is provided, wherein performing a spatial-to-channel dimensionality transformation operation on the initial feature map, rearranging adjacent spatial pixel blocks to the channel dimension, and generating an intermediate feature map with channel dimension expansion includes:

[0016] The initial feature map is divided into non-overlapping local block regions;

[0017] The spatial feature vectors within each local block region are concatenated along the channel dimension to obtain the concatenated feature map;

[0018] The concatenated feature map is subjected to a linear transformation along the channel dimension to obtain the intermediate feature map.

[0019] According to the present invention, a lightweight self-supervised monocular depth estimation method based on cross-sequence interaction is provided, wherein scanning the intermediate feature map from multiple directions to obtain a serialized representation, and performing linear computation on the serialized representation to extract sequence features with global long-range dependencies, includes:

[0020] The intermediate feature map is scanned in four directions: top to bottom, bottom to top, left to right, and right to left, to generate feature sequences in four directions.

[0021] The structured state-space dual model is applied to the feature sequences in the four directions respectively, and the processed feature sequences in the four directions are cross-merged to obtain the sequence features.

[0022] According to the present invention, a lightweight self-supervised monocular depth estimation method based on cross-sequence interaction is provided, wherein the high-dimensional sequence feature interaction module performs fine-grained extraction of the sequence recombination features, including:

[0023] Multiple parallel branches are constructed, and each branch uses a convolutional kernel with a different dilation rate to process the sequence recombination features, capturing spatial context information at the corresponding scale;

[0024] The spatial context information output by the multiple parallel branches is concatenated along the channel dimension to obtain multi-scale fusion features, and the multi-scale fusion features are compressed through a convolutional layer to obtain compressed fusion features.

[0025] The compressed fused features are mapped to a high-dimensional space to obtain high-dimensional features, and the high-dimensional features are input into the attention mechanism module to calculate the correlation weights of pixel-level features to obtain attention-weighted features.

[0026] The attention-weighted features are residually connected with the compressed fused features to obtain features containing local fine-grained information.

[0027] According to the present invention, a lightweight self-supervised monocular depth estimation method based on cross-sequence interaction is provided, wherein the multiple parallel branches include a first branch, a second branch, and a third branch;

[0028] The first branch uses a convolution kernel with a dilation rate of one to convolve the sequence recombination features and extract spatial features within the local neighborhood.

[0029] The second branch uses a convolution kernel with a dilation rate of 2 to convolve the sequence recombination features and extracts context features of a first range, which is larger than the local neighborhood.

[0030] The third branch uses a convolution kernel with a dilation rate of 3 to convolve the sequence recombination features and extracts context features of a second range, which is larger than the first range.

[0031] According to the present invention, a lightweight self-supervised monocular depth estimation method based on cross-sequence interaction is provided, comprising: reconstructing an image based on the multi-scale depth map and the relative pose change information; constructing an image reconstruction loss based on the reconstructed image and the target frame image; and performing self-supervised training of the cross-sequence interaction depth network in conjunction with the image reconstruction loss, including:

[0032] Calculate the structural similarity index and the first norm distance between the reconstructed image and the target frame image, and then perform a weighted summation of the structural similarity index and the first norm distance to obtain the image reconstruction loss;

[0033] Based on the gradient of the multi-scale depth map and the gradient of the target frame image, the edge-aware smoothing loss is calculated;

[0034] The image reconstruction loss and the edge-aware smoothing loss are weighted and summed to construct a total loss, and the parameters of the cross-sequence interaction deep network are updated based on the total loss.

[0035] According to the present invention, a lightweight self-supervised monocular depth estimation method based on cross-sequence interaction further includes, before constructing the image reconstruction loss based on the reconstructed image and the target frame image:

[0036] Detect stationary pixels and occluded areas in the target frame image;

[0037] A mask matrix is ​​generated based on the detection results, and the pixel differences between the reconstructed image and the target frame image are filtered using the mask matrix.

[0038] According to a lightweight self-supervised monocular depth estimation method based on cross-sequence interaction provided by the present invention, the extraction of the initial feature map of the target frame image includes:

[0039] The target frame image is input into a dual-path convolution backbone block. The first path of the dual-path convolution backbone block performs downsampling processing on the target frame image through a convolutional layer with a stride of two. The second path of the dual-path convolution backbone block performs nonlinear feature transformation on the target frame image by sequentially passing through a convolutional layer with a stride of two, an activation function layer, and a convolutional layer with a stride of one.

[0040] The features output by the first path are added element by element to the features output by the second path to obtain the initial feature map.

[0041] The present invention also provides a lightweight self-supervised monocular depth estimation device based on cross-sequence interaction, comprising the following modules:

[0042] The construction module is used to construct a cross-sequence interactive deep network that includes a depth estimation network and a pose estimation network. The depth estimation network integrates a continuous spatial pooling mechanism module and a high-dimensional sequence feature interaction module.

[0043] The capture module is used to extract the initial feature map of the target frame image and capture the global long-range dependencies from the initial feature map through the continuous spatial pooling mechanism module to obtain sequence recombination features;

[0044] The extraction module is used to perform fine-grained extraction of the sequence recombination features through the high-dimensional sequence feature interaction module, and to decode the extracted fine-grained features to generate a multi-scale depth map.

[0045] The acquisition module is used to acquire relative pose change information between the target frame image and adjacent frame images using the pose estimation network;

[0046] The estimation module is used to reconstruct the image based on the multi-scale depth map and the relative pose change information, construct an image reconstruction loss based on the reconstructed image and the target frame image, and perform self-supervised training on the cross-sequence interactive depth network in combination with the image reconstruction loss, and output the depth estimation result of the target frame image using the trained depth estimation network.

[0047] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the lightweight self-supervised monocular depth estimation method based on cross-sequence interaction as described above.

[0048] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the lightweight self-supervised monocular depth estimation method based on cross-sequence interaction as described above.

[0049] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the lightweight self-supervised monocular depth estimation method based on cross-sequence interaction as described above.

[0050] This invention provides a lightweight self-supervised monocular depth estimation method and apparatus based on cross-sequence interaction. By constructing a depth estimation network integrating a continuous spatial pooling mechanism module and a high-dimensional sequence feature interaction module, it can fully integrate the global long-range modeling capability of the state-space model, the fine-grained feature capture capability of the transformer, and the local perception advantage of the convolutional neural network. Specifically, the continuous spatial pooling mechanism module effectively captures the global long-range dependencies of the image, alleviating the problem of limited receptive field in traditional convolution; the high-dimensional sequence feature interaction module extracts local features in a high-dimensional space, ensuring the edge clarity of the depth map. Simultaneously, by combining a pose estimation network and a self-supervised training strategy, efficient learning on unlabeled data is achieved. Due to the lightweight network architecture design, the computational complexity is significantly reduced, making this method applicable to resource-constrained edge computing platforms such as autonomous driving and robot navigation, avoiding the shortcomings of traditional methods that struggle to balance accuracy and efficiency and model long-range dependencies. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating the lightweight self-supervised monocular depth estimation method based on cross-sequence interaction provided by the present invention.

[0053] Figure 2 This is a schematic diagram of the lightweight self-supervised monocular depth estimation device based on cross-sequence interaction provided by the present invention.

[0054] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] Monocular depth estimation, a key technology in applications such as autonomous driving and robot navigation, has always attracted much attention. Traditional supervised learning methods often rely on expensive equipment such as LiDAR to obtain real depth data, which is not only costly to annotate but also difficult to scale. In contrast, self-supervised methods achieve label-free training through image reconstruction and are gradually becoming a research hotspot. However, existing self-supervised methods still have significant limitations in their architectural design. Specifically, CNN-based methods are limited by their inherent local receptive field and struggle to effectively model long-distance dependencies; while Transformer-based methods have strong global perception capabilities, their computational complexity is too high and they lack spatial information extraction capabilities, making them difficult to deploy on resource-constrained platforms. Furthermore, related technologies generally suffer from insufficient information coordination in multi-scale feature fusion, often leading to blurred boundaries and structural distortions in depth prediction results in complex scenes.

[0057] Furthermore, State Space Models (SSMs) offer novel approaches to visual tasks due to their ability to model long sequences with linear complexity. However, directly applying such models to image sequence modeling still faces challenges such as insufficient representation of two-dimensional spatial relationships and loss of local information. Therefore, there is an urgent need in the industry for a lightweight deep estimation architecture that can organically integrate the long-range modeling capabilities of Mamba, the feature capture capabilities of Transformers, and the local perception advantages of CNNs, and is suitable for self-supervised training.

[0058] To address this, this invention provides a lightweight, self-supervised monocular depth estimation method based on cross-sequence interaction. The method aims to construct a cross-sequence interaction deep network comprising a depth estimation network and a pose estimation network. It utilizes a continuous spatial pooling mechanism integrated within the depth estimation network to capture global long-range dependencies from the initial feature map, and employs a high-dimensional sequence feature interaction module to perform fine-grained extraction and decoding of sequence reconstruction features. Simultaneously, it combines relative pose change information obtained from the pose estimation network for image reconstruction and self-supervised training. This effectively integrates the advantages of state-space models, transformers, and convolutional neural networks, solving the problems of insufficient two-dimensional spatial relationship representation and local information loss, while achieving high-precision depth estimation in complex scenes with a lightweight architecture.

[0059] in, Figure 1 This is a flowchart illustrating the lightweight self-supervised monocular depth estimation method based on cross-sequence interaction provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130, 140 and 150.

[0060] Step 110: Construct a cross-sequence interactive deep network that includes a depth estimation network and a pose estimation network. The depth estimation network integrates a continuous spatial pooling mechanism module and a high-dimensional sequence feature interaction module.

[0061] Here, cross-sequence interaction deep networks can be understood as neural network architectures that can utilize the interaction information between sequences to optimize depth estimation performance. This network architecture mainly consists of two core sub-networks: a depth estimation network (DepthNet) and a pose estimation network (PoseNet).

[0062] A depth estimation network is used to predict the depth information of each pixel in a single image. In this embodiment, the depth estimation network can adopt an encoder-decoder structure. The encoder part not only includes conventional convolutional layers, but also integrates a Continuous Spatial Pooling Mechanism (CSPM) module and a High-dimensional Sequential Feature Interaction (HSFI) module.

[0063] Since conventional convolutional layers excel at extracting local features, demonstrating the local perception capability of CNNs, and ensuring the capture of basic image texture and shape, the integration of the CSPM module introduces the long sequence modeling capability of state-space models (such as Mamba), enabling the network to transcend local windows and understand global long-distance dependencies in the image, such as road extensions and perspective between objects, thus compensating for the limitation of CNNs' receptive field. The integration of the HSFI module borrows the feature capture capability of Transformers, enabling precise focusing on key fine-grained features, such as object edges, in a high-dimensional feature space, further improving the precision of depth estimation. By organically combining these three elements, a lightweight depth estimation model is constructed that possesses a global perspective while preserving rich details, and boasts a small number of parameters and high computational efficiency.

[0064] Pose estimation networks are used to estimate the relative motion of the camera between two adjacent frames, i.e., the change in camera pose. They provide the necessary constraints for self-supervised learning, allowing the network to be trained solely on video sequences without real depth labels.

[0065] Step 120: Extract the initial feature map of the target frame image, and capture the global long-range dependencies from the initial feature map through the continuous spatial pooling mechanism module to obtain the sequence recombination features.

[0066] Specifically, the target frame image can be understood as the image frame at the current moment for which depth estimation is required, such as real-time road condition images captured by cameras in autonomous vehicles. Extracting the initial feature maps of the target frame image involves inputting the target frame image into the encoder front end of the depth estimation network. Through a series of convolutions, activations, and other operations, the original RGB image data is converted into a feature representation with a certain number of channels and spatial resolution. These feature maps carry preliminary visual information about the image, such as color, edges, and texture.

[0067] The continuous spatial pooling mechanism is a key component in depth estimation networks. While traditional pooling operations can reduce feature map size, they often lead to the permanent loss of local spatial information and have a fixed receptive field, making it difficult to capture long-range pixel dependencies. In contrast, the continuous spatial pooling mechanism in this embodiment aims to collaboratively model global long-range dependencies and mitigate spatial information degradation.

[0068] Specifically, the continuous spatial pooling mechanism module transforms two-dimensional image features into a sequence form through spatial-to-channel conversion and a multi-directional scanning strategy, and models the sequence using a structured state-space dual model (SS2D). This approach allows the network to process image pixel sequences like text sequences, effectively capturing global long-range dependencies between distant pixels in the image, such as the direction of road extension and the perspective relationship between distant and near objects. The reconstructed sequence features obtained after processing by this module not only retain rich local details but also incorporate global contextual information.

[0069] Step 130: Through the high-dimensional sequence feature interaction module, fine-grained extraction of sequence recombination features is performed, and the extracted fine-grained features are decoded to generate a multi-scale depth map.

[0070] Specifically, the high-dimensional sequence feature interaction module is another core component in the deep estimation network. Its design purpose is to further extract and enhance the local fine-grained features of the image in the high-dimensional feature space.

[0071] After processing by the continuous spatial pooling mechanism, the feature map, while possessing global dependencies, may lack refinement in certain local details. The high-dimensional sequence feature interaction module expands the receptive field and enhances the spatial diversity of features without reducing resolution by employing multi-scale dilated convolution. Simultaneously, it utilizes a self-attention mechanism to calculate the correlation between pixels in the high-dimensional context, thereby accurately extracting fine-grained features crucial for depth estimation, such as the sharpness of object edges and the contours of small objects.

[0072] Decoding extracted fine-grained features to generate multi-scale depth maps refers to feeding these features into a decoder to generate multi-scale depth maps. The decoder typically employs upsampling to progressively restore the spatial resolution of the feature map and outputs depth prediction results at different scales. The generated multi-scale depth maps mean that the network not only outputs a final depth map with the same resolution as the original image but also a series of intermediate depth maps with lower resolutions (such as 1 / 2, 1 / 4, 1 / 8, etc. of the original image size). This multi-scale output strategy helps the network learn the deep structure from coarse to fine, improving the overall robustness of predictions.

[0073] Step 140: Use a pose estimation network to obtain the relative pose change information between the target frame image and adjacent frame images.

[0074] Specifically, adjacent frame images refer to the frames that are temporally adjacent to the target frame image in a video sequence. The pose estimation network receives the target frame image and adjacent frame images as input. For example, it can use the two frames as input after concatenating them along the channel dimension. After feature extraction and regression prediction, it outputs the relative pose change information of the camera between the two frames.

[0075] Relative pose change information typically includes six degrees of freedom (6-DoF): three translation parameters (displacement along the x, y, and z axes) and three rotation parameters (rotation angles about the x, y, and z axes, such as Euler angles or quaternions). This information describes the spatial motion trajectory of the camera from capturing adjacent frames to capturing the target frame. In this embodiment, the pose estimation network can be built based on pre-trained models such as ResNet18, which can quickly and accurately infer the camera's motion parameters.

[0076] Step 150: Reconstruct the image based on the multi-scale depth map and relative pose change information, and construct the image reconstruction loss based on the reconstructed image and the target frame image. After self-supervised training of the cross-sequence interactive depth network in combination with the image reconstruction loss, the depth estimation result of the target frame image is output using the trained depth estimation network.

[0077] Specifically, image reconstruction refers to the synthesis of an image generated using geometric projection relationships. Specifically, based on the multi-scale depth map of the target frame output by the depth estimation network, the relative pose change information output by the pose estimation network, and the known camera intrinsic matrix, pixels in adjacent frames can be projected and transformed to the viewpoint of the target frame image, thereby synthesizing a reconstructed target frame image. The reconstruction formula is as follows:

[0078]

[0079]

[0080] in, Let these be the homogeneous coordinates of the pixels in the target frame image. To reconstruct a 2D image, For camera parameters, The pose matrix includes translation and rotation matrices. This is the depth map of the target frame image. This is the RGB image of the target frame. To reconstruct the RGB representation of the image.

[0081] The purpose of constructing an image reconstruction loss is to measure the difference between the reconstructed image and the ground truth target frame image. The smaller the difference, the more accurate the predicted depth map and pose information. Image reconstruction loss typically includes photometric error, such as a weighted loss combining structural similarity (SSIM) and L1 norm.

[0082] Self-supervised training of cross-sequence interactive deep networks using image reconstruction loss involves calculating gradients through backpropagation using the aforementioned loss function and updating the parameters of the depth estimation and pose estimation networks. This process requires no manually labeled depth information; learning is accomplished solely through geometric consistency constraints between video frames.

[0083] After training, the trained depth estimation network can be used to infer the depth of any given target frame image and directly output its corresponding depth estimation result. This result can be a grayscale image or a pseudo-color image reflecting the scene depth, with pixel values ​​representing the distance from the camera.

[0084] This embodiment presents a lightweight, self-supervised monocular depth estimation method based on cross-sequence interactions. By constructing a depth estimation network integrating a continuous spatial pooling mechanism module and a high-dimensional sequence feature interaction module, it fully integrates the global long-range modeling capability of the state-space model, the fine-grained feature capture capability of the transformer, and the local perception advantage of the convolutional neural network. Specifically, the continuous spatial pooling mechanism module effectively captures the global long-range dependencies of the image, alleviating the problem of limited receptive field in traditional convolution; the high-dimensional sequence feature interaction module extracts local features in a high-dimensional space, ensuring the edge clarity of the depth map. Simultaneously, by combining a pose estimation network and a self-supervised training strategy, efficient learning on unlabeled data is achieved. Due to the lightweight network architecture design, computational complexity is significantly reduced, making this method applicable to resource-constrained edge computing platforms such as autonomous driving and robot navigation, avoiding the shortcomings of traditional methods that struggle to balance accuracy and efficiency and model long-range dependencies.

[0085] As an optional embodiment, the cross-sequence interactive deep network constructed in this embodiment mainly includes two parallel sub-networks, namely the depth estimation network and the pose estimation network.

[0086] The depth estimation network employs an encoder-decoder structure, fusing features from different levels through skip connections. The encoder is responsible for extracting multi-scale feature representations from the input image. In this embodiment, the encoder can be a specially designed lightweight architecture, whose core components include a dual-path convolutional stem block, a continuous spatial pooling (CSPM) module, and a high-dimensional sequence feature interaction (HSFI) module.

[0087] The dual-path convolutional backbone is located at the very beginning of the encoder and is used for initial downsampling and feature extraction of the input image. It consists of two parallel paths: one path performs direct downsampling through a convolutional layer with a stride of 2 to quickly reduce resolution while preserving key structural information; the other path performs nonlinear transformations through a series of convolutions and activation functions to extract richer semantic features. The outputs of the two paths are added element-wise to obtain an initial feature map containing rich information.

[0088] The CSPM module is embedded in each feature extraction stage of the encoder, replacing or assisting traditional pooling layers. It utilizes the Structured State Space Dual Model (SS2D) and a multi-directional scanning strategy to convert two-dimensional feature maps into sequences for processing. This reduces the resolution of the feature maps while capturing global long-range dependencies and preserving important spatial information, thus avoiding information loss caused by traditional pooling.

[0089] The HSFI module is used after or alternately with the CSPM module. It utilizes multi-scale dilated convolution and self-attention mechanisms to refine features in a high-dimensional feature space, enhancing the network's ability to perceive local fine-grained features (such as object edges and texture details).

[0090] The encoder generates feature maps at different scales (such as 1 / 2, 1 / 4, 1 / 8, 1 / 16 resolutions) by stacking multiple CSPM and HSFI modules in series, forming a feature pyramid.

[0091] Furthermore, the decoder is responsible for progressively restoring the deep abstract features extracted by the encoder to the resolution of the original image and outputting a depth map. The decoder typically consists of multiple upsampling blocks, each containing an upsampling layer and a convolutional layer. At each scale, the decoder receives feature maps from the encoder at the corresponding scale via skip connections and concatenates or fuses the upsampled features with the encoder features to supplement lost spatial details. Finally, the decoder outputs multi-scale depth prediction results, i.e., a multi-scale depth map.

[0092] Pose estimation networks aim to regress the relative pose of the camera from a sequence of adjacent frames. This network consists of an encoder and a decoder. The encoder can use a pre-trained lightweight convolutional neural network as its backbone to extract motion features from the image sequence. The decoder performs global average pooling or further processes the features extracted by the encoder through several convolutional / fully connected layers, ultimately outputting a 6-dimensional vector. This vector contains three translation parameters and three rotation parameters, representing the camera's displacement and rotation angles along the X, Y, and Z axes, respectively, i.e., relative pose change information (6-DoF Pose).

[0093] Through the above architecture design, the cross-sequence interactive deep network can synergistically utilize the feature extraction capability of the depth estimation network and the motion perception capability of the pose estimation network to achieve high-precision monocular depth estimation under a self-supervised training framework.

[0094] Traditional pooling operations often calculate based directly on statistical values ​​within a local window when downsampling feature maps. While this reduces computation, it inevitably loses spatial structural information within the window. Furthermore, the fixed receptive field limits the network's ability to capture relationships between distant pixels in the image, which is detrimental to depth estimation tasks that require accurate understanding of scene geometry. To address this issue, the continuous spatial pooling mechanism module in this embodiment employs a more refined processing flow.

[0095] Specifically, the sequential recombination features are obtained by capturing global long-range dependencies from the initial feature map through a continuous spatial pooling mechanism module, including:

[0096] Perform a spatial-to-channel dimensionality transformation operation on the initial feature map, rearrange adjacent spatial pixel blocks to the channel dimension, and generate an intermediate feature map with channel dimension expansion;

[0097] The intermediate feature map is scanned from multiple directions to obtain a serialized representation, and linear computation is performed on the serialized representation to extract sequence features with global long-range dependencies.

[0098] The sequence features are restored to a two-dimensional spatial structure and fused with the features processed by convolution to obtain the sequence recombination features.

[0099] Considering that directly downsampling the initial feature map would result in the loss of spatial details, this embodiment employs a space-to-channel dimensionality transformation as a lossless information recombination method. In this way, while reducing the spatial resolution of the feature map, the information originally scattered in spatial locations is completely preserved and transferred to the channel dimension, thus achieving a channel-for-space tradeoff and avoiding permanent information loss.

[0100] Here, the spatial-to-channel dimensionality transformation operation can be understood as rearranging and combining pixels within a local neighborhood on the feature map according to certain rules. Adjacent spatial pixel blocks refer to small, non-overlapping regions on the initial feature map. The intermediate feature map for channel dimensionality expansion refers to the feature map whose spatial size decreases but the number of channels increases accordingly after the transformation.

[0101] As an optional implementation, the spatial-to-channel dimensionality transformation can be achieved as follows: Assuming the initial feature map has dimensions H (height) × W (width) × C (number of channels), it is divided into multiple non-overlapping 2×2 local block regions. For each 2×2 local block region, the feature vectors of its four contained pixel locations (each vector of length C) are concatenated along the channel dimension to form a new feature vector of length 4C. After performing this operation on all local block regions, an intermediate feature map of size (H / 2) × (W / 2) × 4C is obtained.

[0102] Furthermore, while state-space models (such as Mamba) possess excellent linear complexity for modeling long sequences, they are essentially designed for one-dimensional sequence data (such as text and audio), making it difficult to directly understand the complex spatial neighborhood relationships in two-dimensional images, such as vertical and horizontal adjacency. Simply flattening an image into a one-dimensional sequence may destroy the original two-dimensional structural information, causing the model to fail to effectively capture the spatial dependencies between pixels. Therefore, this embodiment introduces a multi-directional scanning strategy to transform the two-dimensional intermediate feature map into a serialized representation containing rich spatial context information, thereby leveraging the efficient computational power of the state-space model to extract global long-range dependencies.

[0103] Here, scanning the intermediate feature map from multiple directions refers to traversing all pixels in the intermediate feature map along multiple preset paths, thereby generating multiple corresponding one-dimensional feature sequences. The serialized representation is the set of these one-dimensional feature sequences. Linear computation typically refers to linear operations such as matrix multiplication and convolution performed based on the Structured State-Space Dual Model (SS2D). Sequence features with global long-range dependencies refer to feature representations where, after model processing, the features at each position incorporate information from other positions in the sequence (including distant positions).

[0104] As an optional implementation, a four-directional scanning strategy can be used. Specifically, the intermediate feature map is scanned in four directions: top to bottom, bottom to top, left to right, and right to left, generating four independent feature sequences. For each feature sequence, a structured state-space dual model is used for modeling. This model transforms the continuous system into a global convolution operation through discretized state-space equations, enabling the output at any position in the sequence to be aware of the input information of the entire sequence while maintaining linear computational complexity.

[0105] Specifically, for each direction of the sequence, the model recursively or via global convolution to update its internal state using learned state transition parameters, thereby capturing long-range dependencies in the sequence. Finally, the processed feature sequences from the four directions are fused to obtain sequence features that contain both global information and retain the two-dimensional spatial structure.

[0106] While single sequence features capture global long-range dependencies, they may not be as intuitive and refined in representing local details as traditional convolutional features. To construct a feature representation that combines global perspective and local detail awareness, this embodiment restores the sequence features to the two-dimensional spatial structure of the image and fuses them with the convolutional processing results that preserve local spatial features. This fusion mechanism creates complementary advantages, enabling the final reconstructed sequence features to understand both the overall layout of the scene and clearly depict the local contours of objects.

[0107] Here, restoring to a two-dimensional spatial structure refers to reversing the one-dimensional sequence features according to the previous scanning path, restoring them to the same spatial size as the intermediate feature map. Features processed by convolution refer to the features obtained after performing a regular convolution operation on the initial feature map or intermediate feature map, which mainly contains texture and edge information within the local neighborhood.

[0108] As an optional implementation, fusion can be achieved through residual connections. Specifically, the restored two-dimensional sequence features are added element-wise with the features processed by convolution. The fused features are then further integrated and enhanced through fully connected layers or convolutional layers, ultimately outputting sequence reconstruction features. These reconstructed features will serve as input to a subsequent high-dimensional sequence feature interaction module for further fine-grained feature extraction.

[0109] This embodiment employs a continuous spatial pooling mechanism to capture global long-range dependencies from the initial feature map. The spatial-to-channel dimensionality transformation effectively avoids information loss during downsampling. Multi-directional scanning and linear computation cleverly address the limitations of state-space models in processing two-dimensional images, achieving efficient global long-range dependency capture. By fusing sequential and convolutional features, the integrity and richness of feature representation at both global and local levels are ensured. Due to these mechanisms, the depth estimation network's ability to understand complex scene structures is significantly improved, avoiding the inaccurate depth prediction or geometric distortion problems caused by limited receptive fields in traditional methods.

[0110] In deep learning networks, as the number of layers increases, the spatial resolution of feature maps typically needs to be reduced to expand the receptive field and reduce computational cost. However, traditional downsampling methods often discard some pixels or retain only local statistical information, inevitably leading to the permanent loss of original image information. This information loss is particularly problematic in depth estimation tasks that demand high spatial detail, severely impacting the final prediction accuracy. To address this issue, this embodiment employs a meticulous spatial-to-channel dimension transformation strategy to achieve lossless downsampling, i.e., reducing the spatial dimension while completely preserving all original information in the channel dimension.

[0111] Specifically, a spatial-to-channel dimensionality transformation operation is performed on the initial feature map, rearranging adjacent spatial pixel blocks to the channel dimension to generate an intermediate feature map with expanded channel dimensions, including:

[0112] The initial feature map is divided into non-overlapping local block regions;

[0113] The spatial feature vectors within each local block region are concatenated along the channel dimension to obtain the concatenated feature map;

[0114] A linear transformation along the channel dimension is performed on the stitched feature map to obtain an intermediate feature map.

[0115] To reorganize feature information in an orderly manner, the entire feature map needs to be decomposed into regular small units. By dividing the initial feature map into non-overlapping local block regions, it can be ensured that each pixel in the image can be assigned to one and only one specific processing unit, avoiding duplicate calculations or omissions of information, and laying the structural foundation for subsequent feature rearrangement.

[0116] Here, the initial feature map refers to the feature tensor input to the current module, which typically has dimensions of H×W×C. Non-overlapping local block regions refer to rectangular regions divided into fixed sizes on the spatial plane of the feature map.

[0117] As an optional implementation, the size of the local block region can be set to 2×2. This means that the initial feature map is divided into several 2×2 pixel squares every two pixels in both the height and width directions. Each square contains four spatially adjacent feature vectors.

[0118] Considering that physically adjacent pixels often have strong correlations, and in order to preserve the information of these pixels while reducing spatial resolution, this embodiment adopts a strategy of shifting the spatial dimension to the channel dimension. By concatenating the spatial feature vectors within each local block region along the channel dimension, the four feature vectors originally distributed in a 2×2 spatial location can be integrated into a single feature vector located in a single spatial location but with double the number of channels. This achieves both a reduction in spatial size and ensures that all feature information within the local region is not lost, only its form of existence is changed.

[0119] Here, a spatial feature vector refers to a sequence of feature values ​​of length C corresponding to a single pixel position in the feature map. Concatenating along the channel dimension means joining multiple feature vectors end-to-end to form a longer vector. The concatenated feature map is the new feature tensor generated after the above operations.

[0120] As an optional embodiment, for a local block region of size 2×2 and number of channels C, it contains feature vectors at four positions: top left, top right, bottom left, and bottom right. , , , Concatenate these four vectors into a new vector of length 4C. Perform this operation on all local block regions in the initial feature map to obtain the stitched feature map, whose size becomes (H / 2)×(W / 2)×4C.

[0121] Considering that the number of channels in the feature map increases significantly after the concatenation operation (e.g., quadrupling the original number), this could lead to a sharp increase in subsequent computational costs, and the concatenated channels may lack effective information interaction and feature fusion. To control model complexity and promote information integration between channels, this embodiment performs a linear transformation on the channel dimension of the concatenated feature map. This step allows for weighted combination of the expanded channel features, extracting a more compact and representative feature representation. Simultaneously, the number of channels can be adjusted to the desired dimension, such as restoring it to the original number C or adjusting it to other values, thereby obtaining the final intermediate feature map.

[0122] In this context, linear transformation of the channel dimension typically refers to linearly mapping the feature vector at each pixel location of the feature map using point convolutions or fully connected layers. The intermediate feature map is the feature map output after linear transformation and dimensionality adjustment.

[0123] As an alternative implementation, a convolutional layer with a kernel size of 1×1 can be used to process the concatenated feature map. Specifically, this transformation process can be represented as:

[0124]

[0125] in, Represents the initial feature map. This represents the operation of dividing the initial feature map into non-overlapping local block regions and concatenating the spatial feature vectors along the channel dimension. This represents a convolution operation with a kernel size of 1×1. This represents the intermediate feature map.

[0126] This embodiment cleverly converts spatial information into channel information by dividing local block regions and stitching along channels, achieving lossless feature map downsampling. Through linear transformation of the channel dimension, it effectively integrates the expanded channel features and controls the model size. By employing this strategy of rearranging before transformation, it can preserve rich original image information while providing a compact and information-density intermediate feature map for subsequent global dependency modeling, avoiding the information loss problem caused by traditional downsampling methods.

[0127] While the intermediate feature maps retain their original information through dimensionality transformation, they are still essentially two-dimensional mesh structures. Although the structured state-space dual model introduced in this embodiment has the advantage of linear complexity in capturing long-sequence dependencies, its original design is for one-dimensional temporal data and struggles to directly handle the complex spatial context relationships in two-dimensional images. Using only a single-direction scan would cause the model to capture dependencies only in that specific direction, ignoring geometric information in other directions. To fully utilize SS2D's efficient modeling capabilities and overcome its limitations with two-dimensional data, this embodiment employs a multi-directional scanning strategy.

[0128] Specifically, the intermediate feature map is scanned from multiple directions to obtain a serialized representation, and linear computation is performed on the serialized representation to extract sequence features with global long-range dependencies, including:

[0129] The middle feature map is scanned in four directions: top to bottom, bottom to top, left to right, and right to left, generating feature sequences in four directions.

[0130] The structured state-space dual model is applied to the feature sequences in the four directions respectively, and the processed feature sequences in the four directions are cross-merged to obtain the sequence features.

[0131] Considering that objects in an image may extend in any direction, serialization in a single direction can disrupt the spatial adjacency of the image. For example, scanning only from left to right may sever continuous features in the vertical direction. To comprehensively capture all spatial relationships of image pixels in a two-dimensional plane, this embodiment scans the intermediate feature map in four directions: top to bottom, bottom to top, left to right, and right to left. This omnidirectional scanning strategy generates four feature sequences containing different spatial order information, ensuring that any two pixels in the image can establish a direct or indirect contextual relationship in at least one sequence, thereby eliminating perceptual blind spots caused by causal directions.

[0132] Here, the intermediate feature map refers to the feature tensor with a two-dimensional spatial structure generated in the preceding steps. Scanning from top to bottom and other directions refers to flattening the two-dimensional feature map into a one-dimensional sequence according to a specific pixel traversal order. The feature sequences in the four directions refer to the sets of one-dimensional vector sequences generated corresponding to the four scanning paths, respectively.

[0133] As an optional embodiment, assume the intermediate feature map size is H′×W′. For top-to-bottom scanning, feature vectors are read sequentially from row 1 to row H′, following row-major order. For bottom-to-top scanning, feature vectors are read sequentially from row H′ to row 1, following the reverse of row-major order. For left-to-right scanning, feature vectors are read sequentially from column 1 to column W′, following column-major order. For right-to-left scanning, feature vectors are read sequentially from column W′ to column 1, following the reverse of column-major order.

[0134] Considering that after generating feature sequences in four directions, a model with long memory is needed to extract the dependencies, this embodiment applies a structured state-space dual model to each of the four feature sequences. This model can effectively remember distant past information in the sequence, i.e., pixel features that are far apart in the image, and update the current state through linear computation, thereby establishing global context awareness in each direction. After processing, since these four feature sequences represent different spatial perspectives, in order to obtain a unified and complete global feature representation, the processed feature sequences in the four directions need to be cross-merged and reintegrated into their corresponding spatial locations to obtain the final sequence features.

[0135] Among them, the Structured State-Space Dual Model (SS2D) is a deep learning model based on continuous-time state-space equations that have been discretized. Its core dynamic equations can be described as follows:

[0136]

[0137]

[0138] Where A is the state transition matrix constructed based on the High-order Polynomial Projection Operators (HiPPO) framework, ensuring the model's ability to model long sequences, and B is the input projection matrix. D is the output projection matrix, and D is the feedforward matrix. This represents the system state vector at time t. Indicates the input signal. Indicates the output signal. express The derivative of . Furthermore, cross-merging refers to reversing the four processed sequences back to a two-dimensional grid along their original scan paths and fusing the features at corresponding positions.

[0139] Furthermore, the Structured State-Space Dual Model (SS2D) is a deep learning model based on continuous-time state-space equations that have been discretized. To achieve efficient parallel computation, this embodiment employs the zero-order preservation method to discretize the continuous-time system, transforming the computation of the entire sequence into a global convolution operation. Specifically, the discretized convolution kernel... The calculation formula is as follows:

[0140]

[0141] in, It is the discretized form of the continuous state matrix A. It is the discretized form of the input matrix B. It is the output projection matrix. It is the sequence length.

[0142] Based on the calculated global convolution kernel For the input sequence Model output It is calculated using the following formula: , This indicates a convolution operation.

[0143] Considering that after global long-range dependency modeling through the continuous spatial pooling mechanism module, the feature map, while possessing macroscopic scene understanding capabilities, may become blurred in microscopic local details due to multiple transformations, this embodiment designs a high-dimensional sequence feature interaction module that combines multi-scale dilated convolution with a self-attention mechanism. This module aims to enhance the network's ability to perceive local fine-grained features without significantly increasing computational cost and to address the issue of a fixed receptive field for a single convolutional kernel.

[0144] Specifically, the high-dimensional sequence feature interaction module performs fine-grained extraction of sequence recombination features, including:

[0145] Multiple parallel branches are constructed, and each branch uses convolutional kernels with different dilation rates to process the sequence recombination features and capture spatial context information at the corresponding scale.

[0146] The spatial context information output by multiple parallel branches is concatenated along the channel dimension to obtain multi-scale fusion features, and then the multi-scale fusion features are compressed through a convolutional layer to obtain compressed fusion features.

[0147] The compressed fused features are mapped to a high-dimensional space to obtain high-dimensional features, and the high-dimensional features are input into the self-attention mechanism module to calculate the correlation weights of pixel-level features, thus obtaining attention-weighted features;

[0148] By performing a residual connection between the attention-weighted features and the compressed fused features, features containing local fine-grained information are obtained.

[0149] Considering the significant scale differences between objects in an image—for example, a car in the foreground is large while a pedestrian in the distance is small—a single-scale convolutional kernel struggles to capture the key features of all objects simultaneously. To enable the network to perceive multiple scales while avoiding resolution loss due to downsampling, this embodiment constructs multiple parallel branches, each employing a convolutional kernel with a different dilation rate. By adjusting the dilation rate, the receptive field size of the convolutional kernel can be flexibly controlled while maintaining the feature map resolution; a smaller dilation rate focuses on immediate local details, while a larger dilation rate focuses on the broader surrounding environment. In this way, different branches can capture spatial context information at different scales, thereby enriching the expressive power of the features.

[0150] Here, parallel branches refer to data processing paths that exist simultaneously in the network structure without interfering with each other. The dilation rate refers to the number of zero values ​​inserted between convolutional kernel elements to expand the coverage of the convolutional kernel. Sequence recombination features refer to the feature maps output by the continuous spatial pooling mechanism module. Spatial context information at the corresponding scale refers to pixel-related features extracted within a specific receptive field.

[0151] As an alternative implementation, three parallel branches can be constructed: the first branch uses a 3×3 convolution with a dilation rate r=1, resulting in a small receptive field, focusing on extracting minute textures and edges. The second branch uses a 3×3 convolution with a dilation rate r=2, resulting in a medium receptive field, used to capture the structure of medium-sized objects. The third branch uses a 3×3 convolution with a dilation rate r=3, resulting in a larger receptive field, used to perceive a wider range of background information. Each branch independently performs convolution operations on the input features, outputting feature maps at their respective scales.

[0152] Considering that the features extracted by each branch are distributed across different channels, in order to integrate this multi-scale information into a unified feature space, it is necessary to concatenate the spatial context information output by multiple parallel branches along the channel dimension. However, the concatenation operation will multiply the number of channels, which will place a huge burden on subsequent computations. Therefore, it is necessary to compress the multi-scale fused features through convolutional layers, reducing the number of channels to the original level or a preset value, thereby controlling the number of model parameters while fusing multi-scale information, and obtaining compact and information-rich compressed fused features.

[0153] Here, concatenation along the channel dimension refers to stacking the feature maps output from different branches in the depth direction. Multi-scale fusion features refer to feature maps with increased channel counts generated after concatenation. Convolutional layers typically refer to 1×1 point convolutional layers, used for channel fusion and dimensionality reduction.

[0154] As an optional implementation, assuming that the number of channels in the output feature map of each branch is C, the three branches are concatenated to obtain a multi-scale fusion feature F with 3C channels. multi Next, it is processed using a 1×1 convolutional layer with C kernels, outputting a compressed fusion feature F with C channels. compress This process can also be accompanied by batch normalization and activation function operations to enhance the nonlinear representation of features.

[0155] Considering that convolution operations are inherently local, although dilated convolution expands the receptive field, it is still difficult to establish flexible global relationships between pixels. To further extract the fine-grained features most valuable for depth estimation, this embodiment introduces the self-attention mechanism from the Transformer. First, the compressed fused features are mapped to a high-dimensional space to separate different feature components in a higher-dimensional semantic space. Then, the high-dimensional features are input into the self-attention mechanism module, which can calculate the correlation weight between any two pixels in the image and perform weighted aggregation of features accordingly. In this way, the network can automatically focus on key regions such as object edges and texture abrupt changes, while suppressing irrelevant information such as smooth backgrounds, thereby obtaining attention-weighted features. Here, mapping to the high-dimensional space can be achieved through 1×1 convolutions or fully connected layers. The self-attention mechanism module typically includes query, key, and value calculations. The correlation weights of pixel-level features constitute the attention map.

[0156] As an optional embodiment, after mapping the compressed fused features to a high-dimensional space, the resulting features are input into the attention mechanism module. First, a query (Q) matrix, a key (K) matrix, and a value (V) matrix are generated through linear projection. Among these, the attention-weighted features... The calculation formula is as follows:

[0157]

[0158] in, Indicates will and The correlation scores between them are normalized into probability weights.

[0159] Furthermore, considering that while the self-attention mechanism is powerful, it may lose some original low-frequency information during processing due to the redistribution of weights. To ensure the integrity of the information flow and prevent the gradient vanishing problem in deep networks, this embodiment employs a residual connection structure. Specifically, the attention-weighted features processed by the attention mechanism are added element-wise to the compressed fusion features that have not undergone this processing. This design allows the network to retain the original multi-scale features while adding the gains brought by the attention mechanism, thus ultimately obtaining features that contain both rich basic information and fine-grained local information. Here, residual connection refers to a structure that directly adds the input of a module to its output.

[0160] As an optional embodiment, features including local fine-grained information It can be calculated based on the following formula:

[0161]

[0162]

[0163] in, It is the feature obtained by mapping the compressed fused features to a high-dimensional space. Indicates to Perform attention calculations. Indicates to The result obtained after performing attention calculation.

[0164] This implementation effectively captures multi-scale spatial contextual information by constructing multiple parallel branches and employing convolutional kernels with different dilation rates; it efficiently fuses multi-scale features through channel concatenation and compression; it achieves pixel-level feature refinement and enhancement in high-dimensional space by introducing a self-attention mechanism; and it ensures the stability and integrity of feature propagation through residual connections. By employing these comprehensive methods, the network's ability to resolve fine-grained local image features is significantly improved, avoiding the common problems of blurred object boundaries and loss of minute structures in depth estimation.

[0165] Given the complexity of natural scene images, objects within them often exhibit significant scale variations. Using only a single, fixed-size convolutional kernel makes it difficult to simultaneously extract features from both large, nearby objects and small, distant objects. To enable the network to adapt to objects of different scales, thereby improving the robustness and accuracy of depth estimation, this embodiment designs three parallel branches with different receptive field levels within the HSFI module.

[0166] Specifically, the multiple parallel branches include a first branch, a second branch, and a third branch;

[0167] The first branch uses a convolution kernel with a dilation rate of one to convolve the sequence recombination features and extract spatial features within the local neighborhood.

[0168] The second branch uses a convolution kernel with a dilation rate of 2 to convolve the sequence recombination features and extracts contextual features in the first range, which is larger than the local neighborhood.

[0169] The third branch uses a convolution kernel with a dilation rate of 3 to convolve the sequence recombination features and extracts contextual features in the second range, which is larger than the first range.

[0170] Considering that much key information in an image is contained within the immediate vicinity of pixels, it is essential to retain the ability to keenly capture minute details. Therefore, this embodiment sets up a first branch, employing a convolutional kernel with a dilation rate of one to process the sequence recombination features. This configuration allows the convolutional kernel to focus only on the densest neighborhood around a pixel, thereby accurately extracting high-frequency spatial features within the local neighborhood and preventing the neglect of fine structures. Here, a dilation rate of one means there are no gaps between the convolutional kernel elements, resulting in the smallest receptive field. Spatial features within the local neighborhood refer to the microstructural features formed by directly adjacent pixels surrounding a pixel.

[0171] As an optional implementation, the first branch uses a 3×3 convolutional layer with a stride of 2 and a dilatation rate of 1. This convolutional layer directly acts on the input sequence recombination features and outputs the first set of feature maps. Its receptive field size is 3×3, which can effectively capture detailed information such as road surface cracks and distant vehicle outlines.

[0172] Considering that focusing solely on microscopic details is insufficient, it's also necessary to understand the relationship between the object itself and its surrounding environment (e.g., the relationship between car headlights and the car body). To expand the perceptual range without increasing the number of parameters and computational load, this embodiment introduces a second branch using a convolutional kernel with a dilation rate of 2. By introducing a dilation, this branch can read pixels in a skipping manner, thus covering a larger area than the first branch. The contextual features extracted in this first range help the network understand the structural information of medium-scale objects, filling the cognitive gap between microscopic details and the macroscopic background. Here, a dilation rate of 2 means inserting a zero value between adjacent weights of the convolutional kernel, expanding the effective coverage of the 3×3 convolutional kernel to 5×5. The first range refers to this expanded receptive field region.

[0173] As an alternative implementation, the second branch also uses a 3×3 convolutional layer with a stride of 2, but sets its porosity parameter to 2. This operation expands the receptive field to a 7×7 region, enabling the capture of medium-scale features such as complete human silhouettes and traffic signs.

[0174] Considering that a larger field of view is needed to determine the overall depth trend of large objects or background areas, this embodiment sets up a third branch with a convolutional kernel of dilation rate of 3 to further expand the network's perceptual boundaries. This allows the convolutional kernel to extend its coverage further, enabling the aggregation of information from a wider region to extract contextual features of a second range. These features are crucial for inferring the consistent depth of large areas. Here, a dilation rate of 3 means inserting two zero values ​​between the convolutional kernel weights. The second range refers to a receptive field region that is wider than the first range.

[0175] As an optional implementation, the third branch continues to use 3×3 convolutional layers with a stride of 2 and a void ratio parameter set to 3. This operation further expands the effective receptive field to an 11×11 region. This allows the network to perceive macroscopic features such as the direction of an entire lane and the overall facade of a building.

[0176] This embodiment constructs a multi-level receptive field system by setting up three parallel branches (first, second, and third) and using convolutional kernels with dilation rates of one, two, and three, respectively. Through differentiated dilation rate settings, it captures comprehensive features from microscopic details to medium-scale structures and macroscopic backgrounds simultaneously. This multi-scale parallel strategy allows the network to flexibly handle depth estimation needs for objects of various scales in complex scenes, avoiding the loss of details or inconsistent depths of large objects caused by a single receptive field.

[0177] The core of self-supervised depth estimation lies in using geometric consistency in video sequences as a supervision signal, assuming that the appearance of a static scene should remain consistent across different viewpoints. However, single pixel-level differences are often affected by factors such as illumination variations and automatic exposure adjustments, and relying solely on photometric errors can easily lead to discontinuities or artifacts in the depth map in areas with weak texture. To construct a robust supervision system that accurately reflects the scene structure while ensuring the quality of the depth map, this embodiment designs a composite loss function that includes image reconstruction loss and edge-aware smoothing loss.

[0178] Specifically, the method involves reconstructing images based on multi-scale depth maps and relative pose change information, constructing an image reconstruction loss based on the reconstructed image and the target frame image, and combining the image reconstruction loss with self-supervised training of the cross-sequence interaction deep network, including:

[0179] Calculate the structural similarity index and first norm distance between the reconstructed image and the target frame image, and then sum the structural similarity index and first norm distance by weight to obtain the image reconstruction loss;

[0180] The edge-aware smoothing loss is calculated based on the gradient of the multi-scale depth map and the gradient of the target frame image.

[0181] The image reconstruction loss and edge-aware smoothing loss are weighted and summed to construct the total loss, and the parameters of the cross-sequence interaction deep network are updated based on the total loss.

[0182] While simply comparing absolute pixel differences is computationally simple, it is highly sensitive to changes in illumination and struggles to capture local structural information. Structural similarity indices, on the other hand, measure image similarity across three dimensions: brightness, contrast, and structure, exhibiting strong robustness to illumination variations. To balance global luminance consistency and local structural consistency, this embodiment combines both approaches. Through weighted summation, pixel-level reconstruction accuracy is maintained while enhancing the network's perception of object contours and texture structures, resulting in a more reliable image reconstruction loss.

[0183] Here, the reconstructed image refers to the image synthesized using the outputs of the depth estimation network and the pose estimation network through projection transformation. The target frame image refers to the original ground truth image used as a supervisory benchmark. The structural similarity index is a metric for measuring the similarity between two images. The first norm distance, often called the L1 distance or Manhattan distance, represents the sum of the absolute values ​​of the pixel differences.

[0184] As an optional embodiment, image reconstruction loss It can be calculated using the following formula:

[0185]

[0186] in, Indicates the reconstructed image. Represents the target frame image. It is a hyperparameter used to balance the weights between the SSIM loss and the first norm distance. The value range is usually between 0 and 1, such as It can be 0.85. express and The SSIM (Structural Similarity Index) is used to assess the structural similarity between two images. A SSIM value closer to 1 indicates greater similarity between the two images, while a value closer to 0 indicates greater dissimilarity. This represents the first norm distance, also known as the Manhattan distance or minimum absolute error, used to calculate the sum of the absolute values ​​of the pixel differences between the reconstructed image and the target frame image.

[0187] The structural similarity index can be calculated using the following formula:

[0188]

[0189] in, express and Structural similarity index between them These represent two image windows (usually image blocks or the entire image). and They represent The mean, and They represent variance express covariance, and These are two constants used to avoid zero denominators and improve numerical stability.

[0190] Considering that self-supervised learning is prone to noise or depth discontinuities in areas lacking texture (such as walls and roads), smoothing constraints are needed. However, simple smoothing operations can blur object edges. To address this contradiction, this embodiment employs an edge-aware smoothing loss. This loss uses the gradient of the target frame image as weights. In areas with large image gradients (i.e., obvious edges), the weight of the smoothing constraint is reduced, allowing abrupt changes in depth; in areas with small image gradients (i.e., flat regions), the weight of the smoothing constraint is increased, forcing the depth to remain smooth.

[0191] In this context, the gradient of the multi-scale depth map refers to the rate of change of depth values ​​in space. The gradient of the target frame image refers to the rate of change of pixel brightness in space. The edge-aware smoothing loss is a regularization term for adaptive weights.

[0192] As an optional implementation, the edge-aware smoothing loss can be calculated using the following formula:

[0193]

[0194] in, This represents edge-aware smoothing loss. These represent the width and height of the target frame image, respectively. Represents the set of all pixels in the target frame image. Represents the displacement field exist The absolute value of the gradient in the direction, and Indicates the target frame image in The exponential decay term in the direction.

[0195] To achieve end-to-end training of the network, the different constraints mentioned above need to be integrated into a unified optimization objective. This embodiment uses a weighted sum of the image reconstruction loss and the edge-aware smoothing loss to construct the total loss. In this way, the network is simultaneously constrained by both reconstruction accuracy and depth map smoothness during training. Finally, using optimization algorithms such as gradient descent, the gradient is calculated based on the total loss and backpropagated to update the parameters in the cross-sequence interactive deep network until the network converges.

[0196] As an optional embodiment, total loss It can be represented as:

[0197]

[0198] in, and These are the weighting coefficients.

[0199] Self-supervised monocular depth estimation methods typically rely on a strong assumption: the scene is static and the camera is moving. However, in real-world applications, situations often arise where the camera is stationary or there are objects in the scene that are relatively stationary. In these cases, there is no parallax between adjacent frames, and the network is prone to incorrectly predicting these regions as infinitely far. Furthermore, due to changes in viewpoint, some pixels visible in the target frame may be occluded in adjacent frames. Directly calculating the loss for these regions introduces erroneous supervision signals, severely impacting the accuracy of depth estimation. To eliminate these interfering factors, this embodiment introduces an automatic masking and occlusion handling mechanism.

[0200] Specifically, before constructing the image reconstruction loss based on the reconstructed image and the target frame image, the following steps are also included:

[0201] Detect stationary pixels and occluded areas in the target frame image;

[0202] A mask matrix is ​​generated based on the detection results, and the pixel differences between the reconstructed image and the target frame image are filtered using the mask matrix.

[0203] To identify which pixels violate the geometric assumptions of a moving camera, it is necessary to compare photometric errors under different conditions. Detecting stationary pixels and occluded regions in the target frame image aims to identify pixels that generate misleading information during reconstruction.

[0204] Here, a stationary pixel refers to a pixel that has not shifted relative to the camera coordinate system between consecutive frames, or whose visual appearance changes in accordance with the camera's motion. An occluded region refers to an area that is visible in the current viewpoint but is occluded by other objects in the adjacent viewpoints used for reconstruction.

[0205] As an optional implementation, for the detection of stationary pixels, the "luminance error between the reconstructed image and the target frame image" and the "luminance error between the original adjacent frame image without projection transformation and the target frame image" can be calculated. If the error of a pixel without transformation is less than its error after depth projection transformation, then the pixel is very likely to be a stationary pixel or an object moving at the same speed. For occluded regions, the minimum error of the reconstruction results of multiple adjacent frames (such as the previous frame and the next frame) is usually calculated. That is, for each pixel, only the one with the smallest reconstruction error among all adjacent source frames is taken as the effective supervision, thereby automatically avoiding the high error caused by occluded frames.

[0206] To prevent these invalid pixels from interfering with the network's gradient updates, a mathematical method is needed to remove them from the loss calculation. This embodiment generates a mask matrix based on the detection results and uses this mask matrix to filter pixel differences between the reconstructed image and the target frame image. This binarized filtering mechanism ensures that the final loss function is contributed only by valid pixels that satisfy geometric consistency, thereby guiding the network to learn the correct deep structure.

[0207] Here, the mask matrix is ​​a binary matrix with the same size as the target frame image, where a value of 1 indicates that the location is a valid pixel, and a value of 0 indicates that the location is a stationary pixel or an occluded area that needs to be ignored. Pixel difference refers to the pixel-by-pixel error value generated when calculating the image reconstruction loss, such as photometric error.

[0208] This embodiment identifies stationary objects and objects moving at the same speed in the scene by comparing the reconstruction error with the original error; and achieves hard rejection of invalid supervision signals by generating a mask matrix. This automatic filtering mechanism effectively prevents the network from making erroneous predictions of infinite depth in static or occluded areas, significantly improving the model's robustness and generalization ability in complex dynamic scenes, and avoiding depth map artifacts and structural distortion caused by incorrect supervision.

[0209] While traditional single-path downsampling is fast, it easily loses high-frequency details in the image; and while complex deep feature extraction is highly expressive, it significantly increases the computational burden. To ensure computational efficiency while preserving as much structural information as possible from the original image and enhancing the nonlinear expressive power of the features, this embodiment designs a lightweight and efficient dual-path convolution backbone block.

[0210] Specifically, the initial feature map of the target frame image is extracted, including:

[0211] The target frame image is input into the dual-path convolution backbone block. The first path of the dual-path convolution backbone block performs downsampling processing on the target frame image through a convolutional layer with a stride of two. The second path of the dual-path convolution backbone block performs nonlinear feature transformation on the target frame image by sequentially passing through a convolutional layer with a stride of two, an activation function layer, and a convolutional layer with a stride of one.

[0212] The features output by the first path are added element by element to the features output by the second path to obtain the initial feature map.

[0213] Considering the need to balance the efficiency of direct dimensionality reduction with the richness of feature transformation, this embodiment adopts a dual-path parallel processing strategy. The first path focuses on fast and direct spatial dimensionality reduction, preserving the basic structure through large-stride convolutions; the second path introduces non-linear activations and additional convolutional layers, focusing on extracting deeper semantic features to compensate for the information loss caused by direct downsampling.

[0214] Here, a dual-path convolutional backbone refers to a network structural unit containing two parallel data stream processing branches. The first path and the second path represent two independent processing branches. A convolutional layer with a stride of 2 means that the convolutional kernel moves with a stride of 2 pixels, typically used to halve the width and height of the feature map. Nonlinear feature transformation refers to introducing a nonlinear activation function to enable the network to learn complex nonlinear relationships in the input data.

[0215] As an optional embodiment, assume the input target frame image is The first path uses a 3×3 convolution kernel with a stride of ( Set the value to 2 for direct downsampling. This process can be accompanied by batch normalization. The output of the first path... It can be represented as:

[0216]

[0217] in, Indicates to Normalization is performed. This indicates the use of a 3×3 convolution kernel with a stride of 2. Perform downsampling.

[0218] The second path first uses a 3×3 convolutional kernel with a stride of 2 for initial feature extraction and downsampling. Then, it passes through an activation function layer (GELU), and finally, it uses another 3×3 convolutional kernel with a stride of 1 for feature refinement and enhancement. This process can also be accompanied by batch normalization. The output of the second path... It can be represented as:

[0219]

[0220] in, This indicates the use of a 3×3 convolution kernel with a stride of 2. Perform preliminary feature extraction and downsampling. Indicates to After preliminary feature extraction and downsampling, the resulting features are batch normalized. This indicates that the batch-normalized features are input into the activation function layer. This indicates that the output of the activation function layer is modified and enhanced using a 3×3 convolutional kernel with a stride of 1. This indicates that batch normalization is performed on the modified and enhanced features.

[0221] Considering the complementary nature of the features extracted by the two paths—the first path providing a robust structural foundation and the second path offering rich semantic details—this embodiment combines these advantages to form a feature map with high information density and strong expressive power as input to the subsequent network. The features output from the first path and the second path are added element-wise. This fusion method is similar to residual connections, facilitating smooth gradient propagation while avoiding an increase in feature dimensionality.

[0222] Here, element-wise addition refers to adding the values ​​at corresponding positions of two feature tensors of the same size. The initial feature map is the fused output, typically with a resolution of half that of the original image.

[0223] As an optional embodiment, the final initial feature map The calculation formula is:

[0224]

[0225] This embodiment utilizes a first path to ensure downsampling efficiency and gradient propagation stability, while a second path enhances the non-linear representation of features. Through element-wise addition, it achieves effective fusion of structural and semantic information. This dual-path convolutional backbone design enables the network to capture stronger features from the initial input stage, avoiding the premature loss of feature information caused by traditional simple downsampling.

[0226] To verify the technical effectiveness of the lightweight self-supervised monocular depth estimation method based on cross-sequence interaction provided in this invention, simulation experiments were conducted on the PyTorch deep learning framework platform. In the experiments, the constructed cross-sequence interaction deep network was trained using the Adaptive Moment Estimation (Adam) optimizer. Quantitative evaluation results on benchmark datasets demonstrate that the method of this invention has significant advantages in terms of lightweight design and high accuracy.

[0227] Specifically, this invention achieves superior performance compared to current state-of-the-art unsupervised monocular depth estimation methods within a limited parameter budget (less than 10 million parameters). This demonstrates that by integrating a continuous spatial pooling mechanism module with a high-dimensional sequence feature interaction module, this invention successfully addresses the challenge of traditional lightweight models simultaneously capturing long-range dependencies and preserving local details.

[0228] Under test conditions of 192×640 resolution, compared with the baseline method, the method of this application achieved significant reductions in several key error indicators, including: a 6.09% reduction in absolute relative error, a 17.83% reduction in square relative error, a 7.18% reduction in root mean square error, and a 6.22% reduction in root mean square logarithmic error.

[0229] The above data fully demonstrates that the present invention can achieve efficient and high-precision depth estimation with extremely low parameter count, and is particularly suitable for resource-constrained computing platforms.

[0230] The lightweight self-supervised monocular depth estimation device based on cross-sequence interaction provided by the present invention will be described below. The lightweight self-supervised monocular depth estimation device based on cross-sequence interaction described below can be referred to in correspondence with the lightweight self-supervised monocular depth estimation method based on cross-sequence interaction described above.

[0231] Based on any of the above embodiments Figure 2 This is a schematic diagram of the lightweight self-supervised monocular depth estimation device based on cross-sequence interaction provided by the present invention, as shown below. Figure 2 As shown, the device includes:

[0232] Module 210 is used to construct a cross-sequence interactive deep network that includes a depth estimation network and a pose estimation network. The depth estimation network integrates a continuous spatial pooling mechanism module and a high-dimensional sequence feature interaction module.

[0233] The capture module 220 is used to extract the initial feature map of the target frame image and capture the global long-range dependencies from the initial feature map through the continuous spatial pooling mechanism module to obtain the sequence recombination features;

[0234] The extraction module 230 is used to extract the sequence recombination features in a fine-grained manner through the high-dimensional sequence feature interaction module, and decode the extracted fine-grained features to generate a multi-scale depth map.

[0235] The acquisition module 240 is used to acquire relative pose change information between the target frame image and adjacent frame images using a pose estimation network;

[0236] The estimation module 250 is used to reconstruct the image based on the multi-scale depth map and relative pose change information, and to construct the image reconstruction loss based on the reconstructed image and the target frame image. After self-supervised training of the cross-sequence interactive depth network in combination with the image reconstruction loss, the depth estimation result of the target frame image is output using the trained depth estimation network.

[0237] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions from the memory 330 to execute a lightweight self-supervised monocular depth estimation method based on cross-sequence interactions.

[0238] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0239] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the lightweight self-supervised monocular depth estimation method based on cross-sequence interaction provided by the above methods.

[0240] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the lightweight self-supervised monocular depth estimation method based on cross-sequence interaction provided by the methods described above.

[0241] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0242] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight self-supervised monocular depth estimation method based on cross-sequence interactions, characterized in that, include: Construct a cross-sequence interactive deep network that includes a depth estimation network and a pose estimation network. The depth estimation network integrates a continuous spatial pooling mechanism module and a high-dimensional sequence feature interaction module. The initial feature map of the target frame image is extracted, and the global long-range dependencies are captured from the initial feature map through the continuous spatial pooling mechanism module to obtain sequence recombination features; The high-dimensional sequence feature interaction module is used to extract the sequence recombination features in a fine-grained manner, and the extracted fine-grained features are decoded to generate a multi-scale depth map. The pose estimation network is used to obtain the relative pose change information between the target frame image and adjacent frame images; The image is reconstructed based on the multi-scale depth map and the relative pose change information, and an image reconstruction loss is constructed based on the reconstructed image and the target frame image. After the cross-sequence interaction deep network is trained in conjunction with the image reconstruction loss, the depth estimation result of the target frame image is output using the trained depth estimation network. The step of capturing global long-range dependencies from the initial feature map through the continuous spatial pooling mechanism module to obtain sequence recombination features specifically includes: Perform a spatial-to-channel dimension transformation operation on the initial feature map, rearrange adjacent spatial pixel blocks to the channel dimension, and generate an intermediate feature map with channel dimension expansion; The intermediate feature map is scanned from multiple directions to obtain a serialized representation, and linear computation is performed on the serialized representation to extract sequence features with global long-range dependencies. The sequence features are restored to a two-dimensional spatial structure, and the restored two-dimensional sequence features are added element by element to the features processed by convolution. The fused features are then processed by a fully connected layer or a convolutional layer for feature integration and enhancement, and the sequence recombination features are output. The step of fine-grained extraction of the sequence recombination features through the high-dimensional sequence feature interaction module includes: Multiple parallel branches are constructed, and each branch uses a convolutional kernel with a different dilation rate to process the sequence recombination features, capturing spatial context information at the corresponding scale; The spatial context information output by the multiple parallel branches is concatenated along the channel dimension to obtain multi-scale fusion features, and the multi-scale fusion features are compressed through a convolutional layer to obtain compressed fusion features. The compressed fused features are mapped to a high-dimensional space to obtain high-dimensional features, and the high-dimensional features are input into the attention mechanism module to calculate the correlation weights of pixel-level features to obtain attention-weighted features. The attention-weighted features are residually connected with the compressed fused features to obtain features containing local fine-grained information.

2. The lightweight self-supervised monocular depth estimation method based on cross-sequence interaction as described in claim 1, characterized in that, The step of performing a spatial-to-channel dimensionality transformation operation on the initial feature map, rearranging adjacent spatial pixel blocks to the channel dimension, and generating an intermediate feature map with expanded channel dimensions includes: The initial feature map is divided into non-overlapping local block regions; The spatial feature vectors within each local block region are concatenated along the channel dimension to obtain the concatenated feature map; The concatenated feature map is subjected to a linear transformation along the channel dimension to obtain the intermediate feature map.

3. The lightweight self-supervised monocular depth estimation method based on cross-sequence interaction according to claim 1, characterized in that, The process of scanning the intermediate feature map from multiple directions to obtain a serialized representation, and then performing linear computation on the serialized representation to extract sequence features with global long-range dependencies includes: The intermediate feature map is scanned in four directions: top to bottom, bottom to top, left to right, and right to left, to generate feature sequences in four directions. The structured state-space dual model is applied to the feature sequences in the four directions respectively, and the processed feature sequences in the four directions are cross-merged to obtain the sequence features.

4. The lightweight self-supervised monocular depth estimation method based on cross-sequence interaction according to claim 1, characterized in that, The multiple parallel branches include a first branch, a second branch, and a third branch; The first branch uses a convolution kernel with a dilation rate of one to convolve the sequence recombination features and extract spatial features within the local neighborhood. The second branch uses a convolution kernel with a dilation rate of 2 to convolve the sequence recombination features and extracts context features of a first range, which is larger than the local neighborhood. The third branch uses a convolution kernel with a dilation rate of 3 to convolve the sequence recombination features and extracts context features of a second range, which is larger than the first range.

5. The lightweight self-supervised monocular depth estimation method based on cross-sequence interaction according to claim 1, characterized in that, The process of reconstructing an image based on the multi-scale depth map and the relative pose change information, constructing an image reconstruction loss based on the reconstructed image and the target frame image, and performing self-supervised training of the cross-sequence interaction deep network using the image reconstruction loss includes: Calculate the structural similarity index and the first norm distance between the reconstructed image and the target frame image, and then perform a weighted summation of the structural similarity index and the first norm distance to obtain the image reconstruction loss; Based on the gradient of the multi-scale depth map and the gradient of the target frame image, the edge-aware smoothing loss is calculated; The image reconstruction loss and the edge-aware smoothing loss are weighted and summed to construct a total loss, and the parameters of the cross-sequence interaction deep network are updated based on the total loss.

6. The lightweight self-supervised monocular depth estimation method based on cross-sequence interaction according to claim 5, characterized in that, Before constructing the image reconstruction loss based on the reconstructed image and the target frame image, the method further includes: Detect stationary pixels and occluded areas in the target frame image; A mask matrix is ​​generated based on the detection results, and the pixel differences between the reconstructed image and the target frame image are filtered using the mask matrix.

7. The lightweight self-supervised monocular depth estimation method based on cross-sequence interaction according to claim 1, characterized in that, The extraction of the initial feature map of the target frame image includes: The target frame image is input into a dual-path convolution backbone block. The first path of the dual-path convolution backbone block performs downsampling processing on the target frame image through a convolutional layer with a stride of two. The second path of the dual-path convolution backbone block performs nonlinear feature transformation on the target frame image by sequentially passing through a convolutional layer with a stride of two, an activation function layer, and a convolutional layer with a stride of one. The features output by the first path are added element by element to the features output by the second path to obtain the initial feature map.

8. A lightweight self-supervised monocular depth estimation device based on cross-sequence interaction, characterized in that, include: The construction module is used to construct a cross-sequence interactive deep network that includes a depth estimation network and a pose estimation network. The depth estimation network integrates a continuous spatial pooling mechanism module and a high-dimensional sequence feature interaction module. The capture module is used to extract the initial feature map of the target frame image and capture the global long-range dependencies from the initial feature map through the continuous spatial pooling mechanism module to obtain sequence recombination features; The extraction module is used to perform fine-grained extraction of the sequence recombination features through the high-dimensional sequence feature interaction module, and to decode the extracted fine-grained features to generate a multi-scale depth map. The acquisition module is used to acquire relative pose change information between the target frame image and adjacent frame images using the pose estimation network; The estimation module is used to reconstruct the image based on the multi-scale depth map and the relative pose change information, construct an image reconstruction loss based on the reconstructed image and the target frame image, and perform self-supervised training on the cross-sequence interaction deep network in combination with the image reconstruction loss, and output the depth estimation result of the target frame image using the trained depth estimation network. The step of capturing global long-range dependencies from the initial feature map through the continuous spatial pooling mechanism module to obtain sequence recombination features specifically includes: Perform a spatial-to-channel dimension transformation operation on the initial feature map, rearrange adjacent spatial pixel blocks to the channel dimension, and generate an intermediate feature map with channel dimension expansion; The intermediate feature map is scanned from multiple directions to obtain a serialized representation, and linear computation is performed on the serialized representation to extract sequence features with global long-range dependencies. The sequence features are restored to a two-dimensional spatial structure, and the restored two-dimensional sequence features are added element by element to the features processed by convolution. The fused features are then processed by a fully connected layer or a convolutional layer for feature integration and enhancement, and the sequence recombination features are output. The step of fine-grained extraction of the sequence recombination features through the high-dimensional sequence feature interaction module includes: Multiple parallel branches are constructed, and each branch uses a convolutional kernel with a different dilation rate to process the sequence recombination features, capturing spatial context information at the corresponding scale; The spatial context information output by the multiple parallel branches is concatenated along the channel dimension to obtain multi-scale fusion features, and the multi-scale fusion features are compressed through a convolutional layer to obtain compressed fusion features. The compressed fused features are mapped to a high-dimensional space to obtain high-dimensional features, and the high-dimensional features are input into the attention mechanism module to calculate the correlation weights of pixel-level features to obtain attention-weighted features. The attention-weighted features are residually connected with the compressed fused features to obtain features containing local fine-grained information.

Images