Self-supervised monocular depth estimation method based on enhanced multi-scale pose network
By employing a self-supervised method based on enhanced multi-scale pose networks, the problems of simple pose network structure and insufficient temporal modeling in self-supervised monocular depth estimation are addressed, achieving high-precision and robust depth estimation applicable to scenarios such as autonomous driving, robot perception, and augmented reality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHANGCHUN UNIV OF SCI & TECH
- Filing Date
- 2025-11-26
- Publication Date
- 2026-07-07
AI Technical Summary
Existing self-supervised monocular depth estimation methods have simple pose network structures and insufficient temporal modeling capabilities, resulting in discontinuous depth predictions, inconsistent scales, and decreased estimation accuracy. These methods are difficult to meet the efficiency and robustness requirements of autonomous driving, robot perception, and augmented reality.
We employ a self-supervised method based on an enhanced multi-scale pose network. By combining a layer-wise feature fusion encoder and a temporal attention decoder with self-supervised pose consistency constraints, we achieve synergistic optimization of depth and pose, thereby improving the robustness and global consistency of the model.
It significantly improves the accuracy and stability of depth estimation in the absence of real depth-annotated data, especially maintaining high performance in dynamic scenes and low-texture areas, making it suitable for applications such as autonomous driving, robot perception, and augmented reality.
Smart Images
Figure CN121661118B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, specifically to a self-supervised monocular depth estimation method based on an enhanced multi-scale pose network. Background Technology
[0002] Depth estimation is a crucial research area in computer vision, aiming to recover the 3D structural information of a scene from monocular or multi-view images. Traditional depth estimation methods primarily rely on principles such as stereo matching, structured light, or time-of-flight (ToF) to achieve depth recovery through multi-view geometric relationships or active optical ranging. However, these methods are limited by hardware, lighting conditions, and parallax calibration in practical applications, and are prone to measurement errors and occlusion problems in complex environments. Furthermore, traditional methods typically require computationally intensive pixel-level matching and optimization processes, resulting in low computational efficiency and insufficient real-time performance, making it difficult to meet the high efficiency and robustness requirements of scenarios such as autonomous driving, robot perception, and augmented reality.
[0003] In recent years, with the rapid development of deep learning technology, depth estimation methods based on convolutional neural networks (CNNs) and Transformer architectures have gradually become a research hotspot. Among them, self-supervised monocular depth estimation methods have become an important development direction in this field because they are free from the dependence on expensive real depth labels. These methods typically achieve joint prediction of depth and camera pose from monocular videos through image reconstruction constraints, using photometric consistency loss as the main supervision signal, thus enabling end-to-end training without additional annotations. Representative works such as SFM-Learner and Monodepth2 have achieved significant progress on standard datasets. However, due to limitations such as illumination changes, dynamic target motion, and occlusion regions, existing self-supervised methods still suffer from problems such as depth discontinuity, scale inconsistency, and reconstruction error accumulation. At the same time, the instability of pose network prediction and insufficient temporal modeling can also lead to camera trajectory discontinuity, further affecting the depth estimation accuracy and training convergence.
[0004] To address the aforementioned issues, existing technologies are beginning to improve model performance from two aspects: network structure design and geometric constraint optimization. Traditional self-supervised frameworks typically employ shallow convolutional encoders or U-Net-like structures, which struggle to fully extract multi-scale spatial features and temporal contextual information, resulting in limited modeling capabilities for complex scenes. Summary of the Invention
[0005] This invention addresses the problems of simple pose network structure, insufficient temporal modeling capability, and lack of geometric constraints in existing self-supervised monocular depth estimation methods, thereby overcoming the defects caused by discontinuous depth prediction, scale inconsistency, and decreased estimation accuracy. To achieve high-precision and robust monocular depth estimation under the condition of no real depth annotation data, this invention proposes a self-supervised monocular depth estimation method based on an enhanced multi-scale pose network.
[0006] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:
[0007] Option 1: This invention proposes a self-supervised monocular depth estimation method based on an enhanced multi-scale pose network. The method includes the following steps:
[0008] Step 1: Obtain consecutive raw monocular video sequence image frame groups from the dataset. and the original monocular video sequence image frame groups Standardization is performed to obtain image frames. For image frames Preprocessing yields the enhanced input sequence F s = ,in, For the target frame, and Each is the target frame Adjacent preceding and following reference frames;
[0009] Step 2: Enhance the target frame from Step 1. F t The target frame is extracted by using the alternating stacked structure of convolutional layers and multi-head self-attention layers in the DepthNet encoder. F t Local details F l and global context features F g ; local detailed features F l and global context features F g The data is fed into the DepthNet decoder, which employs a multi-scale upsampling structure combined with a skip connection structure to recover the spatial resolution, resulting in a dense depth map. ;
[0010] Step 3: Set the target frame F t and its adjacent frames F t-1and F t+1 In the common input layer-by-layer feature fusion encoder, the corresponding multi-scale feature set is extracted: The layer-by-layer feature fusion encoder internally achieves adaptive fusion of shallow structural features and deep semantic features through a channel-spatial dual attention fusion module. Indicates the first Feature map of the layer ;
[0011] Step 4: Fuse the feature maps of the three frames The input is fed into a temporal attention-based context fusion decoder to generate a temporal feature sequence. F T The data is then fed into the temporal attention module; the output is an enhanced temporal feature sequence. ;
[0012] Step 5: Enhance the temporal feature sequence The process is divided into two parallel paths for processing. The output features of the two paths are then fed into the Cross-Gate Fusion module. This module uses a gating mechanism to adaptively weight and fuse the features from the dynamic path and the context path. The fused features are denoted as... F fused ;
[0013] Step 6: Merge the features F fused The input is fed into a lightweight fully connected regression head to calculate the relative camera transformation matrix between the target frame and the reference frame. , s∈{t 1,t+1};
[0014] Step 7: Output the relative camera pose transformation matrix The geometric constraints are optimized to assist in depth prediction in the depth estimation network fed back to the network.
[0015] The depth maps obtained in steps 8 and 2 EQ1 is used to calculate the edge-aware smoothing constraint term, which constrains the magnitude of depth map changes at the gradient level, and the edge-aware smoothing loss is calculated. ;
[0016] Step 9: Given the camera intrinsic parameter matrix K, extract the target frame pixel p=(x,y) and its corresponding depth value. The EQ2 backprojection is used to obtain a 3D point P in 3D space; then, the 3D point P is transformed to the reference frame coordinate system using expression EQ3, and the transformed point is reprojected onto the reference frame image plane to obtain the reconstructed pixel P′; the bilinear interpolation method is used to reconstruct the reference frame image. and Perform sampling to generate the target frame. Reconstructed image from reference viewpoint And calculate the image reconstruction loss according to expression EQ4. ,in These are the weights of the L1 loss and the SSIM loss;
[0017] Step 10: During training, a self-supervised pose consistency loss is introduced, consisting of two parts: positive and negative consistency constraints and closed-loop consistency constraints; EQ5 is used to calculate the positive and negative consistency constraints. This ensures that the forward and reverse transformations are inverse operations; EQ6 is used to calculate the closed-loop consistency constraints. For three frames The geometric closed-loop relationship between the points constrains the temporal continuity and physical rationality of the camera trajectory; finally, EQ7 is used to calculate the comprehensive pose consistency loss, where λ1 and... 2 is a hyperparameter used to balance the contributions of the two constraints;
[0018] Step 11: During the training phase, apply the edge-aware smoothing loss calculated in Step 8. The image reconstruction loss obtained in step 9 and the pose consistency loss established in step 10 We perform weighted combination to construct an overall optimization objective function, thereby achieving synergistic optimization of depth estimation and pose prediction;
[0019] Step 12: After the model training is complete, only the depth estimation network is retained for inference. The input for the inference stage is a single frame image. Output the corresponding dense depth map This enables high-precision depth estimation under unsupervised conditions.
[0020] Furthermore, a preferred implementation is provided, wherein the channel-spatial dual attention fusion module uses a weighted fusion mechanism to perform feature fusion for each layer. Assign attention weights This makes the interlayer features after fusion into Among them, weight Channel attention and spatial attention are learned together to ensure a balanced integration of local structure and global contextual information.
[0021] Furthermore, a preferred embodiment is provided, in step 4 the fused feature maps of the three frames are... The method for inputting into the temporal attention-based context fusion decoder is as follows:
[0022] Global average pooling is applied to the feature map of each frame to compress the spatial dimension into a global representation vector, thus obtaining the semantic token corresponding to each frame image. t 1, Token t ,Token t+1 The tokens from multiple frames are stacked sequentially to form a temporal feature sequence of shape (B, T, C). F T Where B is the batch size, T is the number of frames, and C is the number of channels.
[0023] Furthermore, a preferred embodiment is provided in which the enhanced temporal feature sequence is used in step 5. T The method for dividing is as follows:
[0024] The first path is a dynamic path, selecting the token corresponding to frame 0 as the temporal enhancement feature of the target frame. F TD The second path is the context path, which is used to perform temporal averaging on the complete token sequence, extract global semantic context information of consecutive frames, and obtain the features after full-frame averaging. F TC .
[0025] Furthermore, a preferred embodiment is provided, wherein step 6 specifically includes:
[0026] The regression head, through several fully connected layers, transforms features using a linear transformation. F fused The mapping is to six values, each representing a three-dimensional rotation vector (θ). x , θ y , θ z Translation vector (t) x , t y , t z ); based on the three-dimensional rotation vector (θ) x ,θ y , θ z Translation vector (t) x , t y , t z The relative camera transformation matrix between the target frame and the reference frame is calculated. (s∈{t) 1,t+1}). Where R is a 3×3 rotation matrix and t is a 3×1 translation vector.
[0027] Furthermore, a preferred embodiment is provided, wherein the method for calculating the edge-aware smoothing constraint term using EQ1 in step 8 is as follows:
[0028]
[0029] in, and These represent depth maps. Gradient changes in the horizontal and vertical directions, and Representing the target image Pixel intensity gradients in the horizontal and vertical directions; For edge-aware smoothing loss.
[0030] Furthermore, a preferred embodiment is provided, wherein step 9 specifically includes:
[0031] EQ2:
[0032] EQ3:
[0033] EQ4:
[0034] in, These are the weights of the L1 loss and the SSIM loss. and This is the reference frame image.
[0035] Furthermore, a preferred embodiment is provided, wherein step 10 specifically includes:
[0036] EQ5:
[0037] EQ6:
[0038] EQ7:
[0039] in, and It's a hyperparameter.
[0040] Option 2: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in Option 1.
[0041] Option 3: A computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in Option 1.
[0042] The advantages of this invention are:
[0043] This invention proposes a self-supervised monocular depth estimation method based on an enhanced multi-scale pose network. By introducing a layer-wise feature fusion pose encoder and a temporal attention-based context fusion decoder, the method enhances the network's ability to model structural details and global semantics. Furthermore, by combining self-supervised pose consistency constraints, it achieves synergistic optimization of depth and pose prediction. This method significantly improves the model's robustness and global consistency while maintaining a lightweight structure. It effectively alleviates the instability issues of existing methods in dynamic scenes and low-texture regions, demonstrating high practical application value and potential for widespread adoption.
[0044] The invention describes a multi-scale pose network that achieves high-precision depth prediction without real depth labels. It fully combines shallow texture and deep semantic information, improving the edge clarity and overall accuracy of depth estimation, and enabling the model to maintain stable performance in low-texture and dynamic scenes.
[0045] This invention is also applicable to application areas such as autonomous driving, robot perception, and augmented reality, which require high efficiency and robustness. Attached Figure Description
[0046] Figure 1 This is a flowchart of the self-supervised monocular depth estimation method based on an enhanced multi-scale pose network as described in Implementation Method 1.
[0047] Figure 2 This is a schematic diagram of the layer-by-layer feature fusion pose encoder structure described in Implementation Method 1.
[0048] Figure 3 This is a schematic diagram of the context fusion pose decoder structure based on temporal attention as described in Implementation Method 1. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.
[0050] Implementation Method 1: The purpose of this implementation method is to provide a self-supervised monocular depth estimation method based on an enhanced multi-scale pose network. It aims to solve the problems of simple pose network structure, insufficient temporal modeling ability, and lack of geometric constraints in existing self-supervised monocular depth estimation methods, thereby overcoming the defects such as discontinuous depth prediction, scale inconsistency, and decreased estimation accuracy caused by these problems, and achieving high-precision and robust monocular depth estimation under the condition of no real depth annotation data.
[0051] To further improve the geometric accuracy and temporal stability of monocular depth estimation in complex dynamic scenes, the technical solution of this invention is as follows: A self-supervised monocular depth estimation method based on an enhanced multi-scale pose network utilizes a deep learning network structure combining multi-scale feature fusion and temporal attention mechanisms. Under the condition of no real depth label, it achieves joint self-supervised optimization of depth and camera pose through image reconstruction constraints and pose consistency constraints. The method is characterized by constructing a self-supervised joint training framework composed of a depth estimation sub-network and an enhanced pose estimation sub-network. The pose estimation sub-network extracts multi-scale spatial structure features through a layer-wise feature fusion encoder (HFFP-Encoder) and models the motion dependencies between consecutive frames using a temporal attention-based context fusion decoder (TACF-Decoder). Simultaneously, a self-supervised pose consistency loss function is introduced to enhance the geometric continuity of the camera trajectory through forward and inverse transformation consistency constraints and closed-loop geometric constraints, achieving collaborative optimization of depth prediction and pose estimation. The method specifically includes the following steps:
[0052] Step 1: Place 39,877 training images in a folder named 'train' and 697 test images in a folder named 'test' as the final test set.
[0053] Step 2: Obtain consecutive monocular video sequence image frame groups from the training images. ,in For the target frame, and Each is the target frame Adjacent reference frames. Before performing the convolution operation, the input tensor size requirement is [Bs, Ic, H, W]. The meaning of each parameter in the input tensor is as follows: Bs=12, input batch size; Ic=3, number of RGB channels; H and W are the height and width of the image, respectively (H×W=256×512).
[0054] Step 3: The data loading process involves traversing the 'train' and 'test' folders to load images as tensors with dimensions [12, 3, 256, 512]. The input images are then standardized to a preset resolution. Furthermore, for image frames... The enhanced input sequence was obtained by randomly applying brightness adjustment, contrast adjustment, saturation adjustment, hue dithering, random horizontal flipping, and data normalization (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]). F s = The preprocessed input sequence F s = Input into the DepthNet depth estimation network.
[0055] Step 4: Enhance the target frame F t Inputting a depth estimation network DepthNet, the target frame is extracted using an alternating stacked structure of convolutional layers and multi-head self-attention layers from the DepthNet encoder. F t Local details F l and global context features F g Next, local detailed features... F l and global context features F g The data is fed into the DepthNet decoder, which employs a multi-scale upsampling structure combined with a skip connection structure to recover the spatial resolution, thereby obtaining a dense depth map. .
[0056] Step 5: In the pose estimation stage, the target frame... F t and its adjacent frames F t-1 and F t+1 (The dimensions are B×3×H×W, where B is the batch size, H is the image height, and W is the image width) are input together into the layer-wise feature fusion encoder (HFFP-Encoder) to extract the corresponding multi-scale feature set: , Indicates the first Feature map of the layer These feature maps represent different levels of spatial resolution and receptive field, providing varying visual information from local details to global semantics. The layer-by-layer feature fusion encoder internally achieves adaptive fusion of shallow structural features and deep semantic features through a channel-spatial dual attention fusion module (AGUF). The AGUF module uses a weighted fusion mechanism to assign weights to the features of each layer. Assign attention weights This makes the interlayer features after fusion into Among them, weight Channel attention and spatial attention are learned together to ensure a balanced integration of local structure and global contextual information.
[0057] Step 6: Combine the feature maps from the three frames The input is fed into the Temporal Attention-Based Context Fusion Decoder (TACF-Decoder). First, global average pooling (GAP) is applied to the feature map of each frame to compress the spatial dimension into a global representation vector, thus obtaining the semantic token corresponding to each frame image. t 1, Token t ,Token t+1 The tokens from multiple frames are stacked sequentially to form a temporal feature sequence of shape (B, T, C). F T Where B is the batch size, T is the number of frames, and C is the number of channels. The generated temporal feature sequence... F T The data is fed into the Temporal GroupAttention module. This module employs a multi-head self-attention mechanism to model inter-frame dependencies in the temporal dimension, capture motion trends and semantic consistency between consecutive frames, and output an enhanced temporal feature sequence. .
[0058] Step 7: Feature sequence after time series modeling T The process is divided into two parallel paths. The first path is the Dynamic Path, which selects the token corresponding to the current frame (i.e., frame 0) as the temporal enhancement feature of the target frame. F TD The second path is the dynamic context path, which involves performing a temporal average on the complete token sequence to extract global semantic context information from consecutive frames, resulting in features after averaging across the entire frame. F TC Output features of these two paths F TD and F TC The features are then fed into the Cross-Gate Fusion module. This module uses a gating mechanism to adaptively weight and fuse the features of the dynamic path and the context path, ensuring effective feature fusion and improving the stability and accuracy of pose estimation. Finally, the fused features are denoted as... F fused , used to represent the pose features of the target frame. Where B is the batch size, 1 indicates that there is only one synthesized feature after fusing the dynamic path and the context path in the time dimension, and C is the number of channels.
[0059] Step 8: Merge the features F fusedThe input is fed into a lightweight fully connected regression head. This regression head passes through several fully connected layers, and uses a linear transformation to convert the features... F fused The mapping is to six values, each representing a three-dimensional rotation vector (θ). x , θ y ,θ z Translation vector (t) x , t y , t z According to the three-dimensional rotation vector (θ) x , θ y , θ z Translation vector (t) x , t y , t z The relative camera transformation matrix between the target frame and the reference frame is calculated. (s∈{t) The image (1, t+1) consists of a rotation matrix R and a translation vector t. The rotation matrix R describes the camera's rotation transformation, and the translation vector t describes the camera's translation transformation. The specific matrix form is as follows: Where R is a 3×3 rotation matrix and t is a 3×1 translation vector. Using this transformation matrix, the 3D point P of the target frame can be transformed. t Transform to the coordinate system of the reference frame to achieve geometric alignment between frames.
[0060] Step 9: Dense depth map obtained from depth estimation network Calculate the dense depth map using expression EQ1 Edge-aware smoothing loss The depth map variation is constrained at the gradient level. Edge strength is obtained by calculating the gradient values of each pixel in the horizontal and vertical directions. Then, smoothing weights are assigned to each pixel based on the gradient magnitude, with smaller weights for edge regions and larger weights for flat regions. and These represent depth maps. The gradient changes in the horizontal and vertical directions are used to measure the magnitude of depth changes; and Representing the target image Pixel intensity gradients in the horizontal and vertical directions.
[0061] EQ1:
[0062] Step 10: Given the camera intrinsic parameter matrix K, extract the target frame. Each pixel p=(x,y) and its corresponding depth value in the image. Based on the expression EQ2, backprojecting to three-dimensional space yields the three-dimensional point P. Wherein, Used to map pixels from the image plane to a camera coordinate system direction vector, depth value This provides the actual scale on the direction vector; then, the 3D point P is transformed to the reference frame coordinate system using expression EQ3, and the transformed point is reprojected onto the reference frame image plane to obtain the mapped position P′ of the target pixel in the reference frame coordinate system. The reference frame image (...) is then processed using bilinear interpolation. and Sampling is performed to generate the target frame. Reconstructed image from reference viewpoint And calculate the image reconstruction loss according to expression EQ4. ,in These are the weights of the L1 loss and the SSIM loss.
[0063] EQ2:
[0064] EQ3:
[0065] EQ4:
[0066] Step 11: To ensure the geometric rationality and trajectory continuity of pose prediction between adjacent frames, pose consistency loss is introduced during training. Including positive and negative consistency constraints Closed-loop consistency constraints Two parts. For the target frame. With the next frame PoseNet predicts the forward pose transformation matrix for each position. With the inverse pose transformation matrix The positive and negative consistency constraints are calculated using expression EQ5. This ensures that forward and inverse transformations are inverse operations, thereby improving the consistency and stability of pose prediction. Closed-loop consistency constraints are calculated using expression EQ6. For three frames The geometric closed-loop relationship between the points constrains the camera trajectory to ensure temporal continuity and physical plausibility. Finally, the comprehensive pose consistency loss is calculated using expression EQ7, where λ1 and... 2 is a hyperparameter used to balance the contributions of the two constraints.
[0067] EQ5:
[0068] EQ6:
[0069] EQ7:
[0070] Step 12: Apply edge-aware smoothing loss during the training phase Image reconstruction loss and pose consistency loss A weighted combination is used to construct an overall optimization objective function, achieving synergistic optimization of depth estimation and pose prediction, thereby simultaneously improving the accuracy of depth map prediction and pose estimation. By minimizing the above three types of constraint terms through a comprehensive loss function, the parameters of the depth estimation network and the pose estimation network are updated synchronously, thereby improving the consistency and convergence stability of predictions at a global scale. These are different scales output by the depth decoder. Set to 1e -3 .
[0071] EQ8:
[0072] Step 13: During training, we optimized the depth estimation network (DepthNet) and pose estimation network (PoseNet) through self-supervised learning. These networks continuously adjust their internal weights during training to extract effective features from the input image and output the depth map (DepthNet) and pose (PoseNet). After training, we save the weights of these two networks to a file, which is the pre-trained model we need to load for subsequent inference.
[0073] Step 14: After model training is complete, proceed to the inference phase. First, load the trained model. PyTorch provides `torch.load()` to load the model and uses the `eval()` method to set the model to inference mode. PyTorch will restore the network structure from the saved file and load the weight parameters learned during training. During inference, only the depth estimation network is used for depth prediction of a single frame image. The input image size is [12, 3, 256, 512], and after processing by the depth estimation network, the output depth map size is [12, 1, 256, 512]. The pose estimation network does not participate in the inference process, so only the depth estimation network learned during training is used for depth map prediction. After preprocessing the input image, gradient calculation is disabled using `torch.no_grad()` because backpropagation is not required during inference. Then, the preprocessed image is input into the model to obtain the corresponding depth map. The resulting depth map is a tensor, typically of shape [1, 1, 256, 512], representing the depth prediction result for a single image. By comparing it with the true depth map, evaluation metrics for depth estimation are calculated, such as absolute relative error (Abs Rel), squared relative error (Sq Rel), and root mean square error (RMSE), thus quantifying the model's performance on different test sets. Finally, matplotlib can be used to visualize the depth map, helping to intuitively understand the prediction results.
[0074] Step 15: Using the above steps, a self-supervised monocular depth estimation method based on an enhanced multi-scale pose network can be obtained.
[0075] In summary, this invention significantly improves the stability and accuracy of the model in complex scenes through the synergistic optimization of depth estimation and pose estimation.
[0076] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
[0077] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.
Claims
1. A self-supervised monocular depth estimation method based on an enhanced multi-scale pose network, characterized in that, The method includes the following steps: Step 1: Obtain consecutive raw monocular video sequence image frame groups from the dataset. and the original monocular video sequence image frame groups Standardization is performed to obtain image frames. For image frames Preprocessing yields the enhanced input sequence F s = ,in, For the target frame, and Each is the target frame Adjacent preceding and following reference frames; Step 2: Enhance the target frame from Step 1. F t The target frame is extracted by using the alternating stacked structure of convolutional layers and multi-head self-attention layers in the DepthNet encoder. F t Local details F l and global context features F g ; local detailed features F l and global context features F g The data is fed into the DepthNet decoder, which employs a multi-scale upsampling structure combined with a skip connection structure to recover the spatial resolution, resulting in a dense depth map. ; Step 3: Set the target frame F t and its adjacent frames F t-1 and F t+1 In the common input layer-by-layer feature fusion encoder, the corresponding multi-scale feature set is extracted: The layer-by-layer feature fusion encoder internally achieves adaptive fusion of shallow structural features and deep semantic features through a channel-spatial dual attention fusion module. Indicates the first Feature map of the layer ; Step 4: Combine the feature maps from the three frames The input is fed into a context fusion decoder based on temporal attention; first, global average pooling is applied to the feature map of each frame to compress the spatial dimension into a global representation vector, thus obtaining the semantics corresponding to each frame image. The tokens from multiple frames are stacked sequentially to form a temporal feature sequence of shape (B, T, C). F T Where B is the batch size, T is the number of frames, and C is the number of channels; the generated temporal feature sequence F T The data is fed into the temporal attention module, which employs a multi-head self-attention mechanism to model inter-frame dependencies in the temporal dimension, capture motion trends and semantic consistency between consecutive frames, and output an enhanced temporal feature sequence. ; Step 5: Enhance the temporal feature sequence Perform dual-path parallel processing. The first path is a dynamic path, selecting the path corresponding to frame 0. Temporal enhancement features of the target frame F TD The second path is the context path, which is used to access the complete... The sequence is temporally averaged to extract global semantic context information from consecutive frames, resulting in features after full-frame averaging. F TC The output features of the two paths are fed into the Cross-Gate Fusion module. The Cross-Gate Fusion module uses a gating mechanism to adaptively weight and fuse the features of the dynamic path and the context path. The fused features are denoted as... F fused ; Step 6: Merge the features F fused The input is fed into a lightweight fully connected regression head to calculate the relative camera transformation matrix between the target frame and the reference frame. , s∈{t 1,t+1}; Step 7: Output the relative camera pose transformation matrix The geometric constraints are optimized to assist in depth prediction in the depth estimation network fed back to the network. The depth maps obtained in steps 8 and 2 EQ1 is used to calculate the edge-aware smoothing constraint term, which constrains the magnitude of depth map changes at the gradient level, and the edge-aware smoothing loss is calculated. ; Step 9: Given the camera intrinsic parameter matrix K, extract the target frame pixel p=(x,y) and its corresponding depth value. The EQ2 backprojection is used to obtain a 3D point P in 3D space; then, the 3D point P is transformed to the reference frame coordinate system using expression EQ3, and the transformed point is reprojected onto the reference frame image plane to obtain the reconstructed pixel P′; the bilinear interpolation method is used to reconstruct the reference frame image. and Perform sampling to generate the target frame. Reconstructed image from reference viewpoint And calculate the image reconstruction loss according to expression EQ4. ,in These are the weights of the L1 loss and the SSIM loss; Step 10: During training, a self-supervised pose consistency loss is introduced, consisting of two parts: positive and negative consistency constraints and closed-loop consistency constraints; EQ5 is used to calculate the positive and negative consistency constraints. This ensures that the forward and reverse transformations are inverse operations; EQ6 is used to calculate the closed-loop consistency constraints. For three frames The geometric closed-loop relationship between the points constrains the temporal continuity and physical rationality of the camera trajectory; finally, EQ7 is used to calculate the comprehensive pose consistency loss, where λ1 and... 2 is a hyperparameter used to balance the contributions of the two constraints; Step 11: During the training phase, apply the edge-aware smoothing loss calculated in Step 8. The image reconstruction loss obtained in step 9 and the pose consistency loss established in step 10 We perform weighted combination to construct an overall optimization objective function, thereby achieving synergistic optimization of depth estimation and pose prediction; Step 12: After the model training is complete, only the depth estimation network is retained for inference. The input for the inference stage is a single frame image. Output the corresponding dense depth map This enables high-precision depth estimation under unsupervised conditions.
2. The self-supervised monocular depth estimation method based on an enhanced multi-scale pose network according to claim 1, characterized in that, The channel-spatial dual attention fusion module uses a weighted fusion mechanism to perform feature fusion for each layer. Assign attention weights This makes the interlayer features after fusion into Among them, weight Channel attention and spatial attention are learned together to ensure a balanced integration of local structure and global contextual information.
3. The self-supervised monocular depth estimation method based on an enhanced multi-scale pose network according to claim 1, characterized in that, Step 6 specifically includes: The regression head passes through several fully connected layers and uses linear transformation to convert features... F fused The mapping is to six values, each representing a three-dimensional rotation vector (θ). x , θ y , θ z Translation vector (t) x , t y , t z ); based on the three-dimensional rotation vector (θ) x , θ y ,θ z Translation vector (t) x , t y , t z The relative camera transformation matrix between the target frame and the reference frame is calculated. , s∈{t 1,t+1}, , where R is a 3×3 rotation matrix and t is a 3×1 translation vector.
4. The self-supervised monocular depth estimation method based on an enhanced multi-scale pose network according to claim 1, characterized in that, The method for calculating the edge-aware smoothing constraint term using EQ1 in step 8 is as follows: in, and These represent depth maps. Gradient changes in the horizontal and vertical directions, and Representing the target image Pixel intensity gradients in the horizontal and vertical directions; For edge-aware smoothing loss.
5. The self-supervised monocular depth estimation method based on an enhanced multi-scale pose network according to claim 1, characterized in that, Step 9 specifically includes: EQ2: EQ3: EQ4: in, These are the weights of the L1 loss and the SSIM loss. and This is the reference frame image.
6. The self-supervised monocular depth estimation method based on an enhanced multi-scale pose network according to claim 1, characterized in that, Step 10 specifically includes: EQ5: EQ6: EQ7: in, and It's a hyperparameter.
7. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.
8. A computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 1-6.