Monocular self-supervision depth estimation method fusing multi-resolution features and global context

Abstract

A monocular self-supervision depth estimation method fusing multi-resolution features and global context belongs to the technical field of computer vision, and comprises the following steps: firstly, providing an encoder based on a cross-stage feature high-resolution network, and enhancing the extraction capability of multi-scale features by improving a cross-stage connection mechanism of an HRNet network; secondly, a visual Transform module is introduced to improve complex down-sampling operation, and a multi-head self-attention mechanism is utilized to establish global pixel association; then, an improved decoder network is provided, a channel attention module is used for re-weighting coding features, multi-level features of an encoder are utilized more efficiently, and the balance between segmentation precision and calculation cost is achieved; and finally, proposing an improved attitude estimation network based on lightweight ResNet18, predicting 6-DoF relative poses of adjacent frames in combination with a four-layer convolutional decoder, and accurately estimating the depth of each object. According to the invention, the accuracy of depth estimation is improved, and the visual effect of the depth view is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a monocular self-supervised depth estimation method fusing multi-resolution features and global context. BACKGROUND

[0002] Depth estimation is to recover the three-dimensional geometric information of a scene from a two-dimensional image or video, that is, to calculate the distance (depth value) from each pixel point in the image to the camera. In the field of computer vision, depth estimation is the cornerstone of many high-level computer vision tasks. Compared with binocular and multi-view depth estimation, monocular depth estimation only needs to predict scene depth information through a single image. In addition to great savings in cost, the most important advantage is that it has stronger generalization ability and robustness to infer the three-dimensional information of the scene from a single perspective. Early monocular depth estimation is based on active sensors, which directly obtain dense and accurate depth data using depth sensors (such as LiDAR, structured light, and ToF cameras), but is constrained by device cost and environmental range. With the development of convolutional neural networks (CNN), monocular depth estimation based on deep learning directly outputs scene depth information from a single RGB image end to end, with higher resolution and faster frame rate, and has become a research hotspot in computer vision and computer graphics, and is widely used in three-dimensional reconstruction, autonomous driving, human-computer interaction virtual reality and other tasks.

[0003] Monocular depth estimation based on deep learning includes supervised, unsupervised and self-supervised learning depth estimation methods. Supervised learning methods can obtain excellent estimation results, but need to rely on a large number of real depth labels. Both unsupervised and self-supervised depth estimation methods do not rely on real depth labels. Unsupervised depth estimation has no learning process at all, and the initial feature matching result has a great impact. In low-texture areas or dynamic scenes, it is easy to fail, and can only generate sparse depth maps. In contrast, self-supervised methods learn depth mapping through neural networks, combining the dense prediction ability of supervised learning and the advantage of not needing real depth labels of unsupervised learning, and are the most practical method in the field of depth estimation. However, since more optimization objectives are needed to limit weak texture, moving objects, different lighting and occlusion, etc., self-supervised training is a task with insufficient constraints; the above methods can only partially solve the defects of self-supervised monocular depth estimation, and still have limitations in global perception and long-distance pixel relationship modeling.

[0004] To solve the above problems, this paper designs a self-supervised monocular depth estimation network (High-Resolution Vision Transformer Depth Network, HR-ViTDepthNet) that fuses multi-resolution features and global context spatial features based on the encoder-decoder architecture, combined with the improved high-resolution network MFS-HRNet and the visual Transformer. First, a high-resolution network MFS-HRNet is designed to build an encoder, which enhances the feature expression ability by fusing multi-resolution features across stages. Then, ViT is introduced to avoid using down-sampling operations in the encoder network, so that the encoder can maintain a global receptive field and capture global information at each stage, reducing the loss of feature map resolution and high-level information. Finally, the encoder is connected to the decoding end of the corresponding scale layer through a skip connection, fusing bottom multi-scale features for up-sampling, combining the loss function of each scale to improve the training effect of the convolutional neural network, thereby improving the accuracy of model prediction. The algorithm in this paper maintains low computational complexity and storage requirements while capturing global context information and establishing long-range dependencies between features and depth, thereby improving the prediction accuracy and generalization performance of the network and meeting the real-time estimation requirements.

[0005] 2. Prior art solutions

[0006] In recent years, self-supervised monocular depth estimation technology has made significant progress under the promotion of multi-modal sensor fusion, neural radiance field expansion, and large model feature extraction. The current mainstream methods are mainly based on the self-supervised framework of view synthesis, which jointly optimizes the depth network and the pose network, and uses the photometric reconstruction loss of multi-view images to realize model training. Typical representatives include Monodepth2, Lite-Mono and other methods. However, due to high scene complexity, insufficient constraints and other problems, the performance of existing methods in weak texture areas, dynamic objects, light changes and occlusions still has limitations, especially in global perception ability and long-distance pixel relationship modeling.

[0007] For example, document [1] proposes a self-supervised monocular depth estimation method based on Laplacian pyramid. First, the Laplacian residual map is used to enrich the encoded features to make up for the loss of feature information in the downsampling process; second, the max pooling layer is used in the downsampling process to highlight and amplify the feature information, making the encoder easier to extract the feature information needed by the training model in the feature extraction process; finally, the residual module is used to solve the overfitting problem and improve the utilization efficiency of the decoder for features. This method reduces the network parameter quantity while achieving more accurate depth detail estimation, but the generalization performance is poor. Document [2] proposes a lightweight unsupervised monocular depth estimation method combining convolutional neural networks, attention mechanisms, and speeded up robust features (SURF). A residual block containing linear self-attention mechanism (CCT-Block) and a residual block containing coordinate attention mechanism (CA-Block) are designed. The two kinds of residual blocks are used alternately in the ResNet framework to build a multi-scale encoder that can capture rich context information while reducing parameter size and memory usage, thus establishing the mapping relationship between depth and image features. The reprojection error of SURF features is introduced to reduce the ambiguity that the depth estimation network and pose estimation network may encounter. Overcome the inherent limitations in existing unsupervised monocular depth estimation frameworks, the network shows good generalization performance in multiple scenarios, but the model complexity is high. Document [3] proposes a multi-scale feature enhanced multi-frame self-supervised monocular depth estimation model. First, the encoder of this model introduces a large kernel attention activation module to improve the encoder's ability to extract global spatial features of the input image and preserve spatial context information. At the same time, a structure enhancement module is proposed to enable it to distinguish important features in the channel dimension and enhance the network's ability to perceive image structural features. Finally, a dynamic upsampling method is used in the decoder instead of the nearest neighbor interpolation upsampling method to restore detail information and optimize the artifact phenomenon of the depth map. Document [4] proposes a lightweight self-supervised monocular depth estimation algorithm combined with multi-scale attention. This algorithm introduces a multi-scale attention module as the encoder main body, taking the local structure and long-range global information capturing ability of convolution operation and self-attention mechanism as the core idea. By combining the gated multi-layer perceptron with multi-branch large kernel hollow convolution with the feedforward neural network, it realizes the aggregation of local and global features with attention mechanism, thereby reducing the network parameter quantity and computational complexity while ensuring the depth estimation accuracy. The algorithm model parameter quantity is only 4.1M, and the algorithm achieves a good balance between depth estimation accuracy and inference speed.

[0008] For example, the application patent number: CN202510064155.X, the application provides a kind of monocular depth estimation method based on space selection sequence feature fusion.The application patent number: CN202410063481.4, the application discloses a kind of intelligent traffic monocular depth estimation method based on orthogonal plane.The application patent number: CN202510261430.7, a kind of lightweight depth estimation method and system are provided.The application patent number: CN202510226151.7, a kind of monocular image depth estimation method, device and equipment based on diffusion model and target prompt are provided.The application patent number: CN202210548361.4, a kind of binocular event depth estimation method based on pulse neural network is disclosed.

[0009] Although the above method has made certain progress in depth estimation technology, there are still problems such as lack of texture details and structure blur in complex environment, insufficient global scene understanding ability and long distance dependence modeling, excessive model parameter quantity, real-time performance deficiency, low modeling precision and poor generalization performance for different scenes, which seriously affect the accuracy and integrity of depth estimation in complex scene, and seriously limit the application of depth estimation technology in more computer vision tasks.

[0010] REFERENCES

[0011] [1] Cao Mingwei, Xing Jingjie, Cheng Yifeng, et al. LpDepth: self-supervised monocular depth estimation based on Laplacian pyramid [J]. Computer Science, 2025, 52(03): 33-40.

[0012] [2] Li Xiyu, Yilimu Yalimu, Xie Li Rong, et al. Lightweight unsupervised monocular depth estimation with attention mechanism fusion [J]. Laser and Optoelectronics Progress, 2025, 62(08): 91-101.

[0013] [3] Kou Qiqi, Wang Weichen, Han Chenggong, et al. Multi-scale feature enhanced multi-frame self-supervised monocular depth estimation [J]. Optics Precision Engineering, 2024, 32(24): 3603-3615.

[0014] [4] Ge Jingrui, Qin Guoxuan, Zhang Wei. Lightweight self-supervised monocular depth estimation combined with multi-scale attention [J]. Journal of Xi'an University of Electronic Science and Technology, 2025, 3(25): 1-13. SUMMARY

[0015] In order to solve the above problems, the application provides a kind of monocular self-supervised depth estimation method fusing multi-resolution features and global context, including the following steps:

[0016] S1, data set pretreatment;

[0017] S2, propose an improved deep network;

[0018] S21, propose a cross-stage feature high-resolution network based on internal feature fusion-MF-HRNet;

[0019] S22, add visual Transformer;

[0020] S23, propose an improved image reconstruction deep decoder;

[0021] S3, propose an improved pose network;

[0022] S4, construct a self-supervised signal: loss function;

[0023] S5, test and evaluate.

[0024] Further, in step S1, the KITTI dataset containing high-resolution images collected in urban, rural and highway scenes is obtained.

[0025] Further, in step S21, all stage features of the same resolution branch are spliced along the channel dimension, so that the features of the same resolution carry details and semantics at the same time, and a cross-stage feature high-resolution network MF-HRNet based on internal feature fusion is proposed. For each resolution stream r, the spliced feature is:

[0026]

[0027] Where, Concat represents the splicing operation, [-] is the connection layer, represents the feature map of the s stage of the r resolution stream, and e represents the feature map generated by the encoder.

[0028] MF-HRNet realizes parallel processing of high and low resolution through multi-scale splicing fusion, realizes complementary of cross-stage multi-resolution features and spatial features, makes high-resolution features obtain semantic information, and low-resolution features retain spatial details.

[0029] Further, in step S22,

[0030] The input feature is first preprocessed by layer normalization to make the value distribution on the feature dimension more uniform and speed up the convergence of the model; then it enters the MSA layer to capture global context information and calculate attention weights in parallel through the multi-head mechanism; then, the normalized feature is sent to the MLP containing two fully connected layers to further enhance the feature representation through nonlinear transformation;

[0031] The ViT is introduced into a hybrid architecture of encoding-decoding, the local prior knowledge is re-injected in a data-driven manner by an encoder, the ViT local deficiency is made up, the ViT structure increases the receptive field, the long-range dependence between pixels in the whole image is established, and the depth map quality is improved;

[0032] The result of the network encoder is input into a ViT module for global information processing, after passing through an L-layer Transformer encoder, the output feature is:

[0033] X l =LayerNorm(X l-1 +MSA(X l-1 )),l=1,...,L (2)

[0034] Wherein, the process of MSA adopts a parallel feature interaction strategy as formula (3), the input feature generates Query (Q), Key (K) and Value (V) matrices through three independent linear projection layers, the dimensions are all (N+1, num, dim), N represents the input sequence length, num is the number of attention heads, which is set to 8, dim is the feature dimension of each attention head, the attention distribution is as formula (4), and the softmax function is used to normalize the dot product result of Q and K to generate the attention distribution.

[0035] MSA(X)=Concat(Attention1,...,Attention H )W O (3)

[0036]

[0037] The multi-head attention mechanism in the ViT basic block allows each feature sequence to obtain context information from other sequences, so that these sequences are equivalent to global information processing of the whole image through each transmission of the ViT module.

[0038] Further, in the step S23,

[0039] The skip connection transmits the FM-HRNet multi-stage fusion feature, so that the decoder receives low-level details (Stage 1-2) and high-level semantics (Stage 3-4) at the same time, reconstructs the sequence output by the encoder into a 2D feature map, and uses a channel attention module (CA) to reweight the encoding feature, and the output thereof is represented as:

[0040]

[0041] Wherein, M c (·) generates channel weights, represents the weighted multiplication of the channel dimension, and the important feature channel is strengthened.

[0042] Next, 3×3 convolution is used for further processing, combined with bilinear upsampling to gradually restore the feature map interpolation to the same resolution as the decoder; at each encoder node, upsampling and data concatenation operations are continuously performed, concatenating the skip connections from the encoder and the disparity map upsampling to fuse multi-resolution features and semantic information, achieving accurate prediction of the disparity map; during the decoding process, each decoder node D i The structure is represented as:

[0043]

[0044] Where μ(·) is the bilinear upsampling operation, and σ(·) is the CA attention module. It is a 3×3 convolutional layer, and [·] is the concatenation of channel dimensions; five such operations are aggregated. Upsampling doubles the resolution of the feature map. The concat operation performs tensor concatenation on the upsampling from the encoder skip connections to fuse multi-scale features and semantic information, thereby making more efficient use of the encoder's multi-level features, achieving a balance between segmentation accuracy and computational cost, and realizing accurate prediction of higher resolution disparity maps.

[0045] Furthermore, in step S3, the PoseNet network uses a ResNet-18 encoder to effectively extract features from two consecutive RGB images. The final high-level semantic features are then input to the decoder. Adjacent frames of the target image are concatenated along the channel dimension (W×H×6) and input into the pose estimation network. This input method allows the network to capture the visual differences and temporal information between the two frames and learn the relative motion of the camera. After the ResNet18 encodes the features, the final convolutional layer outputs the dimension... Vector; at the decoding end, the network first reduces the dimensionality of the feature map through a 1×1 convolutional layer, compressing the channels to... Then, the ReLU function is used for activation; followed by two 3×3 convolutional layers and the ReLU activation function, further extracting and activating the previous-stage features to capture more detailed spatial information; subsequently, a 1×1 convolutional layer is used again to further compress the channels. The channel vector of the last six dimensions is used as Axisangle and Translation, respectively, to represent the three-dimensional displacement components t = [t]. x ,t y ,t z ] T And 3 is the rotation angle ω=[ω x ,ω y ,ω z ] TAnd the pose network output is a 6-DOF pose transformation vector T by using the Rodrigues formula (0) to convert the pose transformation matrix T between adjacent frames;

[0046] R = I + K^sinθ + (K^) 2 (1-cosθ) (7)

[0047] θ =‖ω‖2 (8)

[0048]

[0049] Where I is the identity matrix, ensuring that points on the axis of rotation remain unchanged; K^ is the skew-symmetric matrix of K; K^ sinθ is the linear component of rotation; (K^) 2(1-cosθ) is the quadratic component of rotation; reconstruct the target image by the estimated camera pose, adjacent frame image and depth map; compare the difference between the reconstructed image and the target image, and the model realizes end-to-end self-supervised monocular depth estimation.

[0050] Further, in step S4, an image reconstruction loss is used as a self-supervised signal, and a target image I t is input into the depth estimation network, the predicted depth is d, the target image I0 and the adjacent frame image I t (t∈{-1,1}) are input into the pose network, and the predicted pose is T0→t, the target image I t The reconstructed projection is:

[0051] I t→0 = I t [proj(reproj(I0,d,T 0→t ),K)] (10)

[0052] Where K is the camera intrinsic matrix, proj() is the projection of a 3D point to the 2D coordinates of the source frame I t , reproj() is the back-projection of the pixels of I0 to the 3D space (based on the depth d), and [·] is the bilinear sampling operation, so that the image reconstruction loss lp(2) between the target image and the reconstructed image can be calculated by the structural similarity SSIM and the L1 norm, to measure the difference between the synthesized image I t →0 and the real target frame I0.

[0053]

[0054] Where α = 0.85, and the network is optimized by minimizing the image reconstruction loss L p :

[0055] L p = mint [l p (I0,I t→0 )] (12)

[0056] An edge-aware smoothing loss L s Equation (13) is used to eliminate noise in low gradient areas and suppress depth noise in low texture areas.

[0057]

[0058] The image re-projection loss and the edge smoothing term are reasonably combined to obtain a final loss function Ltotal:

[0059]

[0060] Wherein, s = 4, represents the number of scales of the decoder stage; λ = 10 -3 is the weight of the edge smoothing loss.

[0061] The beneficial effects of the present application are: the present application proposes a self-supervised depth estimation technology based on an encoder-decoder architecture. The technology first proposes a cross-stage feature high-resolution network MFS-HRNet based on internal feature fusion, designs a multi-resolution encoder through the MFS-HRNet network, enhances the coding ability of the outline area and the detail features of the long-distance object; a visual Transformer is introduced at the output end of the decoder, a cross-window interaction module is designed to realize global pixel correlation, a multi-head self-attention mechanism is used to establish global pixel correlation, a hierarchical feature up-sampling architecture is used to gradually restore the spatial resolution, and the structural consistency of the depth map is improved; in addition, a channel attention module (CA) is used to reweight the encoded features, and an improved decoder network is proposed; finally, a lightweight ResNet18 is used to design a pose estimation network, a 4-layer convolutional decoder is used to predict the 6-DoF relative pose of adjacent frames, and the accuracy of depth estimation is improved. High-quality depth estimation is achieved, which provides new technical support for high-precision depth estimation under unlabeled conditions.

[0062] Through a series of experimental verification, it is proved that the technology has good depth estimation ability in complex scenes, can effectively resist complex light changes, capture complex structural information and detail features, and cope with different scene changes; provides a high-precision depth estimation scheme without labeled data for computer vision, automatic driving, AR / VR and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 Structure diagram of the self-supervised depth estimation network;

[0064] Figure 2 Network structure diagram of MF-HRNet;

[0065] Figure 3 ViT structure diagram;

[0066] Figure 4 Decoder structure diagram;

[0067] Figure 5 Pose network structure diagram;

[0068] Figure 6 Depth visualization results of different models on the KITTI dataset. Detailed Implementation

[0069] To make the technical means and objectives of this invention easier to understand, the invention is further described below with reference to specific embodiments. A monocular self-supervised depth estimation method that integrates multi-resolution features and global context includes the following steps:

[0070] S1. Dataset preprocessing;

[0071] S2. Propose an improved deep network;

[0072] S21. Propose a cross-stage feature high-resolution network based on internal feature fusion -- MF-HRNet;

[0073] S22, Add a visual Transformer;

[0074] S23. Propose an improved image reconstruction depth decoder;

[0075] S3. An improved pose network is proposed;

[0076] S4. Construct a self-supervised signal: loss function;

[0077] S5, Testing and Evaluation.

[0078] This invention addresses the shortcomings of existing self-supervised monocular depth estimation methods in global scene understanding, long-range dependency modeling, and object edge sharpness, as well as the difficulty and high cost of obtaining true depth labels in supervised learning. It proposes a self-supervised monocular depth estimation method based on an encoder-decoder architecture, as shown in the following structure. Figure 1An improved multi-stage feature high-resolution network MFS-HRNet based on internal feature fusion is proposed to improve the expression ability of long-range objects and contour details through multi-resolution mechanism across stages. Vision Transformer (ViT) is introduced to avoid using down-sampling operations in the encoder network, so that the encoder can maintain a global receptive field and capture global information at each stage, reducing the loss of feature map resolution and high-level information. In addition, an improved decoder network is proposed, which uses channel attention modules (CA) to reweight the encoded features, enabling the encoder to more efficiently utilize the multi-level features of the encoder, achieving a balance between segmentation accuracy and computational cost. Meanwhile, the encoder is connected to the decoding end of the corresponding scale layer through a skip connection, fusing the bottom multi-scale features for upsampling. Finally, an improved pose estimation network based on lightweight ResNet18 is proposed, which combines a 4-layer convolutional decoder to predict the 6-DoF relative pose of adjacent frames, and uses loss functions of various scales to improve the training effect of the convolutional neural network, thereby improving the accuracy of the model's depth prediction and improving the visual effect of the depth estimation view. This technology improves the prediction accuracy and accuracy of the network while maintaining a low cost, and meets the real-time depth estimation requirements.

[0079] S1, data set preprocessing

[0080] The KITTI data set, which is a real scene shot data set, is obtained. This data set is a data set for computer vision tasks jointly constructed by the Karlsruhe Institute of Technology in Germany and the Toyota Technical Research Institute in the United States, which contains high-resolution images collected in urban, rural and highway scenes. The image resolution is about 1241x376 pixels, of which 39810 groups are used for training, 4424 groups are used for evaluation, and 697 groups are used for testing.

[0081] S2, improved depth network is proposed

[0082] High-resolution low-level features (1 / 4 size) are accurate in spatial positioning but weak in semantic abstraction, while low-resolution high-level features (1 / 32 size) contain rich semantics but spatial information is degraded. Most depth estimation methods are based on ResNet architecture, and ResNet encodes the input into low-resolution feature maps (1 / 32 scale) through cascading down-sampling, resulting in irreversible loss of object edges, spatial details, etc. Therefore, this study introduces a high-resolution network HRNet as the encoder base architecture, and improves the HRNet network combined with ViT for depth estimation. The network maintains the 1 / 4 scale high-resolution main path of the standard HRNet throughout the process, realizes multi-resolution parallel processing across stages, and maintains four feature streams of 1:1, 1:2, 1:4 and 1:8. The global perception ability is enhanced, and the semantic features and global spatial features are fused through information exchange.

[0083] S21. Propose a cross-stage high-resolution feature network based on internal feature fusion -- MF-HRNet

[0084] HRNet consists of four processing stages and four resolution streams, ultimately outputting five feature maps in the final stage. r = 1, 2, 3, 4; This represents the feature map of the s-th stage of the r-resolution stream, with a resolution of... Number of feature map channels C r =64×2 r-1 The HRNet structure is as follows: Figure 3 As shown in the gray boxes, each green box represents a stage, and each row represents a stream. HRNet only uses features from the final stage, Stage 4, discarding intermediate features / semantic information from preceding stages 1–3, resulting in details in the shallow space not being passed to the decoder. To address the problem of shallow feature loss, this study concatenates features from all stages of the same resolution branch along the channel dimension, allowing features of the same resolution to simultaneously carry details and semantics. This leads to the proposal of a cross-stage feature high-resolution network based on internal feature fusion (Multi-stageFeature-HRNet, MF-HRNet). Figure 2 As shown, the red arrows indicate the connections between feature maps. For each resolution stream r, its concatenated features are:

[0085]

[0086] In this context, [-] represents the connecting layers. Shallow features (smaller s) carry more spatial details, while deep features (larger s) contain richer semantics. MF-HRNet fuses shallow and deep features to improve contour accuracy and semantic understanding.

[0087] Furthermore, HRNet incorporates a multi-stage convolution strategy, based on the relationship between feature resolution and spatial information: feature maps with more channels contain richer semantic information. Obtaining semantically rich mid-level feature maps without changing the scale requires increasing the number of convolutional kernels; however, increasing the number of kernels simultaneously increases the number of parameters, leading to a significant increase in computational complexity. MF-HRNet's multi-stage feature fusion simply concatenates features from different stages using channels, without introducing additional convolutional kernels. For example, when fusing four stages of features simultaneously, MF-HRNet only increases the number of channels, keeping the number of parameters at zero; while traditional methods require adding Cin×Cout×3×3 convolutional parameters.

[0088] Therefore, MF-HRNet realizes parallel processing of high and low resolution through multi-scale splicing fusion, achieves complementary of cross-stage multi-resolution features and spatial features, so that high-resolution features obtain semantic information and low-resolution features retain spatial details. Since the feature mapping of different stages contains different semantic information, this non-parametric operation significantly narrows the semantic gap between the encoder and the decoder without increasing the computational burden, improves the semantic richness of the feature map, and more accurately estimates the object boundary and long-range dependence of the object. Figure 2 As shown by the middle green arrow, the multi-stage features of the encoder FM-HRNet are connected to the corresponding levels of the decoder upsampling path. This is used to compensate for the loss of spatial information caused by downsampling, retain high-frequency information such as object edges and textures, and alleviate the gradient vanishing problem in deep network training.

[0089] S22, add visual Transformer

[0090] As shown in FIG. 4, ViT is mainly composed of two sub-layers, multi-head self-attention (MSA) and multi-layer perceptron (MLP). The input features are first preprocessed by layer normalization to make the value distribution in the feature dimension more uniform and speed up the convergence of the model; then they enter the MSA layer to capture global context information and calculate attention weights in parallel through the multi-head mechanism; finally, the normalized features are sent to the MLP containing two fully connected layers to further enhance the feature representation through nonlinear transformation. Figure 3

[0091] ViT relies entirely on the self-attention mechanism, which can establish long-range dependencies between pixels in the entire image, better handle global context information, and solve the problem of blurred depth relationships of distant objects caused by the local receptive field of traditional CNNs. However, the self-attention mechanism itself does not have prior assumptions about local structures in images, so this approach ignores the strong correlation between adjacent pixels in natural images, such as edge continuity and texture consistency, resulting in a lack of local inductive bias in ViT, which is less efficient in handling local details such as some fine-grained edges or small objects, and faces the challenge of insufficient locality. To compensate for the shortcomings of ViT, this paper introduces ViT into a hybrid architecture of encoder-decoder, re-injects local prior knowledge through data-driven encoding, and compensates for the lack of locality in ViT. At the same time, the ViT structure can increase the receptive field and establish long-range dependencies between pixels in the entire image, improving the quality of the depth map.

[0092] ​The result of the network encoder is input into the ViT module for global information processing. After passing through the L-layer Transformer encoder, the output feature is:

[0093] X l = LayerNorm(X l-1 + MSA(X l-1 )), l = 1,..., L (2)

[0094] Where the MSA process adopts a parallel feature interaction strategy as shown in equation (3). The input feature generates Query (Q), Key (K), and Value (V) matrices through three independent linear projection layers, with dimensions of (N+1, num, dim). N represents the input sequence length, num is the number of attention heads (set to 8 in this paper), and dim is the feature dimension of each attention head. The attention distribution is as shown in equation (4), and the softmax function normalizes the dot product result of Q and K to generate the attention distribution.

[0095] MSA(X) = Concat(Attention1,..., Attention H )W O (3)

[0096]

[0097] The multi-head attention mechanism in the ViT basic block allows each feature sequence to obtain context information from other sequences, so that these sequences are equivalent to a global information processing of the entire image through each passing of the ViT module.

[0098] S23, propose an improved image reconstruction deep decoder

[0099] The skip connection passes the FM-HRNet multi-stage fusion feature, so that the decoder receives low-level details (Stage 1-2) and high-level semantics (Stage 3-4) at the same time, and reconstructs the sequence output by the encoder into a 2D feature map, as shown in Figure 4 The channel attention module (CA) is used in this paper to reweight the encoded features. Its output can be represented as:

[0100]

[0101] Where M c (·) generates channel weights. represents the weighted multiplication of the channel dimension, which strengthens important feature channels.

[0102] Then, the 3x3 convolution is used for further processing, and the bilinear up-sampling is used to interpolate the feature map to gradually restore the same resolution as the decoder. In each encoder node, the up-sampling and data splicing operations are continuously performed, and the multi-resolution features and semantic information are fused by splicing the up-sampling of the skip layer connection from the encoder and the disparity map, so as to realize accurate prediction of the disparity map. In the decoding process, each decoder node D i The structure can be represented as:

[0103]

[0104] Wherein, μ(·) is a bilinear up-sampling operation. σ(·) is a CA attention module. is a 3x3 convolution layer. [·] is a channel dimension splicing. The five such operations are aggregated, the up-sampling doubles the resolution of the feature map, and the concat operation splices the tensors of the up-sampling from the encoder skip layer connection to fuse multi-scale features and semantic information, so as to more efficiently utilize the multi-level features of the encoder, balance the segmentation precision and the calculation cost, and realize accurate prediction of the higher resolution disparity map.

[0105] S3, proposing an improved pose network

[0106] In the structure of self-supervised monocular depth estimation, accurate pose estimation is crucial for accelerating the convergence of the entire network and improving the accuracy of the depth estimation network. In order to ensure the accuracy and stability of the pose estimation during the training process, the PoseNet network of the present application uses ResNet-18 to design the encoder, and after effectively extracting the features of the continuous two frames of RGB images, the final high-level semantic features are input to the decoding end, as shown in Figure 5 .

[0107] The adjacent frame images of the target image are spliced along the channel dimension WxHx6 and input into the pose estimation network. This input method enables the network to capture the visual difference and timing information between the two frames and learn the relative motion of the camera. After the ResNet18 encoding feature, the final output dimension of the last convolution layer is a vector. In the decoding end, in order to reduce the dimension and retain important feature information, the network first reduces the dimension of the feature map through a 1x1 convolution layer, and compresses the channel to After that, a RELU function is used for activation. Then, two 3x3 convolution layers and a RELU activation function are used to further extract and activate the previous stage features to capture more detailed spatial information. Subsequently, a 1x1 convolution layer is used again to further compress the channel to a channel vector of dimension. The three dimensions in the last six-dimensional channel vector are used as Axisangle and Translation, respectively, representing the 3D displacement components t=[t x ,ty t z ] T and 3 are rotation angles ω = [ω x ,ω y ,ω z ] T and converted to the pose transformation matrix T between adjacent frames by using the Rodrigues' Rotation Formula (0), the output of the pose network is a 6-DOF pose transformation vector.

[0108] R = I + K^sinθ + (K^) 2 (1-cosθ) (7)

[0109] θ = ‖ω‖2 (8)

[0110]

[0111] where I is the identity matrix, ensuring that points on the rotation axis remain unchanged; K^ is the skew-symmetric matrix of K; K^ sinθ is the linear component of rotation; (K^) 2(1-cosθ) is the quadratic component of rotation. The reconstructed target image is performed by the estimated camera pose, adjacent frame image and depth map; by comparing the difference between the reconstructed image and the target image, the model realizes end-to-end self-supervised monocular depth estimation.

[0112] S4, constructing a self-supervised signal-loss function

[0113] The unsupervised monocular depth estimation model does not need the depth ground truth of the scene during training, only needs a continuous image sequence as a supervision signal, so a reasonable loss function is crucial for network convergence. In this paper, the image reconstruction loss is used as a self-supervised signal, the target image I t is input into the depth estimation network, and the predicted depth is d. The target image I0 and the adjacent frame image I t (t ∈ {-1, 1}) are input into the pose network, and the predicted pose is T0→t, the target image I t is reconstructed as follows:

[0114] I t→0 = I t [proj(reproj(I0, d, T 0→t ), K)] (10)

[0115] where K is the camera intrinsic matrix, and proj() is the projection of 3D points to the source frame I tThe 2D coordinates of I0 are used, and reproj() backprojects the pixels of I0 to the 3D space (based on depth d). [·] is a bilinear sampling operation. Thus, the image reprojection loss lp(2) between the target image and the reconstructed image can be calculated using structural similarity (SSIM) and L1 norm, which is used to measure the synthetic image I. t →0 is the difference between the actual target frame I0 and the target frame I0.

[0116]

[0117] Where α = 0.85. To reduce the impact of occlusion on the results, the network is optimized by minimizing the image reconstruction loss. p :

[0118] L p =min t [l p (I0,I t→0 (12)

[0119] To regularize the model's depth prediction in structurally ambiguous low-gradient image regions, the model also employs an edge-aware smoothing loss Li. s Equation 13 is used to eliminate noise in low-gradient regions and suppress depth noise in low-texture regions.

[0120]

[0121] By appropriately combining the image reprojection loss and the edge smoothing term, the final loss function Ltotal is obtained:

[0122]

[0123] Where s = 4 represents the number of scales in the decoder stage; λ = 10 -3 It represents the weight of the edge smoothing loss.

[0124] S5, Testing and Evaluation

[0125] To ensure fair comparison with other methods and to accurately analyze the model's accuracy, a universally applicable metric is used to evaluate the proposed method. As follows:

[0126] Absolute Relative Error (Abs Rel):

[0127]

[0128] Square Relative Error (Sq Rel):

[0129]

[0130] Root Mean Square Error (RMSE):

[0131]

[0132] log-root-mean-square error (RMSE_log):

[0133]

[0134] The accuracy δ is the ratio of the predicted depth to the true depth and the inverse of all pixels in the image, and the maximum value is taken as δ. The proportion of pixel points with δ less than threshold 1.25, 1.25 2 , 1.25 3 to the total pixel points is the prediction accuracy, and the closer the value is to 1, the more accurate the result is.

[0135]

[0136] Where, d i is the predicted depth of pixel i; d i * is the true depth of pixel i; and N is the total number of effective depth pixels.

[0137] Experimental results of the method

[0138] (1) Comparative experiment of detection method

[0139] In order to better reflect the superiority and effectiveness of the method of the present application, the method in this paper is tested on the KITTI dataset, and compared with other representative self-supervised monocular depth estimation methods, and the results are shown in Table 1. All algorithms use images of 640x192 size as input, and the optimal and suboptimal results are represented in bold and italic respectively.

[0140] Table 1 Comparison of experimental results on KITTI dataset

[0141]

[0142]

[0143] As shown in Table 1, the method in this paper shows a significant advantage in the comprehensive accuracy performance of depth estimation. Compared with the baseline method Lite-Mono, the method in this paper has obvious improvement in all key indicators: the Abs Rel error is reduced by 5.94%, indicating that it is more sensitive to small depth values and more robust to depth outliers; Sq Rel, RMSE, and RMSE-log are all reduced. In addition, especially in the most challenging threshold indicator δ < 1.25, it is improved by 7.22%, which verifies that the model has more stable prediction ability in the depth fluctuation area (such as the occlusion boundary, weak texture area). The performance improvement is mainly due to: (1) the cross-stage high-resolution network (MF-HRNet) based on internal feature fusion effectively aggregates feature flow information at different levels through multi-scale feature interaction; (2) the global dependency modeling module fused with vit realizes long-range context awareness while maintaining computational efficiency. Compared with the current most advanced method, the model in this paper also shows overall competitiveness. The method in this paper is better than other methods or equal in comprehensive accuracy performance. However, in the extreme motion blur scene (such as fast moving objects), the RMSE-log indicator of the method in this paper still has a slight gap with the methods of Suanders et al. and MonoDiffusion, which may be due to the lack of use of time series information.

[0144] (2) Visual analysis

[0145] In order to further verify the depth estimation effect of the proposed method, in addition to quantitative comparison and analysis, qualitative comparative analysis is also carried out on the present application, such as Figure 6 The depth estimation results of the present application, Monodepth2, Lite-Mono and MonoDiffusion method are shown.

[0146] As shown in Figure 6 , the visualized scene contains a variety of extremely challenging areas, such as weak light and strong shadow, strong light environment, etc. When feature matching is performed in these areas, the model will be affected by factors such as light, texture, occlusion, etc., resulting in large errors in depth prediction results. Other methods have different degrees of scene object missing and boundary blur phenomenon, for example, (1) the depth prediction of the tree trunk and sign area in the first column is broken; (2) the depth estimation of the wall texture in the second column is blurred and deformed; (3) the geometric details of the tree trunk and street lamp pole in the third column are lost; (4) the depth value of the backlight area in the fourth column deviates from the true distribution. In contrast, the method in this paper shows a significant advantage in these challenging areas, and the estimation result not only fits the true outline of the target object better, but also retains more fine depth edges and geometric details. This is due to the multi-scale spatial feature fusion mechanism of MF-HRNet, which effectively preserves local detail information by aggregating feature maps at different scales, making the model more sensitive to depth boundaries and microstructures.

[0147] In addition, the existing method generally has confusion problem in depth prediction of distant objects (such as the far view house in the second column, the distant trees in the third and fourth columns), which is mainly due to the insufficient ability to model the global scene structure. By introducing the ViT visual attention mechanism, the present application realizes the robust modeling of global features: on the one hand, the attention mechanism can capture the spatial dependence relationship between long-distance pixels, thereby reconstructing the complete scene structure; on the other hand, the global context information provides strong constraints for local depth prediction, especially in the light abnormal or texture deficient area, significantly improving the accuracy and visual rationality of depth estimation. Experimental results show that the depth prediction error of the method in this paper is reduced compared with the baseline model (see Table 1), and the visual results also show higher geometric consistency and detail fidelity. Therefore, the present application can provide more reliable and accurate depth information for visual perception tasks, and has a wider application prospect in three-dimensional scene reconstruction, autonomous driving and other scenes.

[0148] The above merely describes the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes within the technical range disclosed in the present application according to the technical solution and concept of the present application, which should be covered within the protection scope of the present application.

Claims

1. A monocular self-supervised depth estimation method that integrates multi-resolution features and global context, characterized in that, Includes the following steps: S1. Dataset preprocessing; S2. Propose an improved deep network; S21. Propose a cross-stage feature high-resolution network based on internal feature fusion -- MF-HRNet; S22, Add a visual Transformer; S23. Propose an improved image reconstruction depth decoder; S3. An improved pose network is proposed; S4. Construct a self-supervised signal: loss function; S5, Testing and Evaluation.

2. The monocular self-supervised depth estimation method fusing multi-resolution features and global context as described in claim 1, characterized in that, In step S1, the KITTI dataset, which is a dataset of real-world scene images, is obtained. It contains high-resolution images collected in scenes such as urban areas, rural areas, and highways.

3. The monocular self-supervised depth estimation method fusing multi-resolution features and global context as described in claim 2, characterized in that, In step S21, all stage features of the same resolution branch are concatenated along the channel dimension, allowing features of the same resolution to simultaneously carry details and semantics. This leads to the proposal of MF-HRNet, a cross-stage feature high-resolution network based on internal feature fusion. For each resolution stream r, its concatenated features are: Where Concat represents the concatenation operation, and [-] indicates the connection layer. Let e ​​represent the feature map of the s-th stage of the r-th resolution stream, and let e represent the feature map generated by the encoder. MF-HRNet processes high-resolution and low-resolution features in parallel through multi-scale stitching and fusion, enabling cross-stage multi-resolution features and spatial features to complement each other, so that high-resolution features can obtain semantic information and low-resolution features can retain spatial details.

4. The monocular self-supervised depth estimation method fusing multi-resolution features and global context as described in claim 3, characterized in that, In step S22, the input features are first preprocessed by layer normalization to make the value distribution in the feature dimension more uniform and accelerate the convergence speed of the model. Then, they enter the MSA layer to capture global context information and calculate attention weights in parallel through a multi-head mechanism. Next, the normalized features are fed into an MLP containing two fully connected layers to further enhance the feature representation using nonlinear transformation. Introducing ViT into the encoder-decoder hybrid architecture, local prior knowledge is re-injected through encoder data-driven approach to compensate for the locality deficiency of ViT. At the same time, the ViT structure increases the receptive field, establishes long-range dependencies between pixels across the entire image, and improves the quality of the depth map. The results from the network encoder are input into the ViT module for global information processing. After passing through the L-layer Transformer encoder, the output features are: X l =LayerNorm(X l-1 +MSA(X l-1 )),l=1,...,L (2) Among them, the MSA process adopts a parallel feature interaction strategy as shown in Equation (3). The input features generate Query(Q), Key(K) and Value(V) moments through three independent linear projection layers. The dimensions are (N+1, num, dim), where N represents the length of the input sequence, num is the number of attention heads, which is set to 8, and dim is the feature dimension of each attention head. The attention distribution is shown in Equation (4). The softmax function normalizes the dot product result of Q and K to generate the attention distribution. MSA(X)=Concat(Attention1,...,Attention H )W O (3) The multi-head attention mechanism in the ViT basic block allows each feature sequence to acquire contextual information from other sequences, making each pass of these sequences through the ViT module equivalent to a global information processing of the entire image.

5. The monocular self-supervised depth estimation method fusing multi-resolution features and global context as described in claim 4, characterized in that, In step S23 Skip-layer connections pass multi-stage fusion features from FM-HRNet, allowing the decoder to simultaneously receive low-level details (Stage 1-2) and high-level semantics (Stage 3-4). This reconstructs the encoder's output sequence into a 2D feature map, which is then reweighted using a channel attention module (CA). The output is represented as follows: Among them, M c (·) Generate channel weights, This represents a weighted multiplication along the channel dimension, which strengthens important feature channels; Next, 3×3 convolution is used for further processing, combined with bilinear upsampling to gradually restore the feature map interpolation to the same resolution as the decoder; at each encoder node, upsampling and data concatenation operations are continuously performed, concatenating the skip connections from the encoder and the disparity map upsampling to fuse multi-resolution features and semantic information, achieving accurate prediction of the disparity map; during the decoding process, each decoder node D i The structure is represented as: Where μ(·) is the bilinear upsampling operation, and σ(·) is the CA attention module. It is a 3×3 convolutional layer, and [·] is the concatenation of channel dimensions; five such operations are aggregated. Upsampling doubles the resolution of the feature map. The concat operation performs tensor concatenation on the upsampling from the encoder skip connections to fuse multi-scale features and semantic information, thereby making more efficient use of the encoder's multi-level features, achieving a balance between segmentation accuracy and computational cost, and realizing accurate prediction of higher resolution disparity maps.

6. The monocular self-supervised depth estimation method fusing multi-resolution features and global context as described in claim 5, characterized in that, In step S3, the PoseNet network uses ResNet-18 to design the encoder. After effectively extracting features from two consecutive RGB images, the final high-level semantic features are input to the decoder. The adjacent frames of the target image are stitched together along the channel dimension in a W×H×6 format and input into the pose estimation network. This input method enables the network to capture the visual differences and temporal information between the two frames and learn the relative motion of the camera. After ResNet18 encodes the features, the final convolutional layer outputs the dimension. Vector; at the decoding end, the network first reduces the dimensionality of the feature map through a 1×1 convolutional layer, compressing the channels to... Then, the ReLU function is used for activation; followed by two 3×3 convolutional layers and the ReLU activation function, the previous features are further extracted and activated to capture more detailed spatial information; Subsequently, a 1×1 convolutional layer was used again to further compress the channels. The channel vector of the last six dimensions is used as Axisangle and Translation, respectively, to represent the three-dimensional displacement components t = [t]. x ,t y ,t z ] T And 3 is the rotation angle ω=[ω x ,ω y ,ω z ] T And using the Rodriguez formula (0), it is transformed into the pose transformation matrix T between adjacent frames. The output of the pose network is a 6-DOF pose transformation vector. R=I+K^sinθ+(K^) 2 (1-cosθ) (7) θ=‖ω‖2 (8) Where I is the identity matrix, ensuring that points on the rotation axis remain unchanged; K^ is the antisymmetric matrix of K; K^ sinθ It is a linear component of rotation; (K^) 2(1-cosθ) It is the second component of rotation; the target image is reconstructed by estimating the camera pose, adjacent frame images and depth map; by comparing the difference between the reconstructed image and the target image, the model achieves end-to-end self-supervised monocular depth estimation.

7. The monocular self-supervised depth estimation method fusing multi-resolution features and global context as described in claim 6, characterized in that, In step S4, image reconstruction loss is used as a self-supervised signal, and the target image I... t As input to the depth estimation network, the predicted depth is d, and the target image I0 and neighboring frame images I... t (t∈{-1,1}) is used as the input to the pose network, the predicted pose is T0→t, and the target image I t The reconstructed projection is: I t→0 =I t [proj(reproj(I0,d,T 0→t ),K)] (10) Where K is the camera intrinsic parameter matrix, and proj() projects 3D points onto the source frame I. t The 2D coordinates of I0 are used, and `reproj()` backprojects the pixels of I0 into 3D space (based on depth d). `[·]` is a bilinear sampling operation, allowing the image reprojection loss between the target image and the reconstructed image to be calculated using structural similarity SSIM and the L1 norm. (2) is used to measure the synthesized image I. t →The difference between 0 and the actual target frame I0; Where α = 0.85, the network is optimized by minimizing the image reconstruction loss. p : L p =who t [l p (I0,I t→0 )] (12) Using edge-aware smoothing loss L s Equation (13) is used to eliminate noise in low gradient regions and suppress depth noise in low texture regions. By appropriately combining the image reprojection loss and the edge smoothing term, the final loss function Ltotal is obtained: Where s = 4 represents the number of scales in the decoder stage; λ = 10 -3 It represents the weight of the edge smoothing loss.

Images