Gradient-aware self-supervised monocular depth estimation method and device for dynamic scene

Abstract

The application discloses a gradient-aware self-supervised monocular depth estimation method and device for dynamic scenes, and relates to the field of image processing, and comprises the following steps: constructing a monocular depth estimation model, a pre-trained pseudo-depth prediction model and a pose estimation network, training the monocular depth estimation model by using the pre-trained pseudo-depth prediction model and the pose estimation network, and obtaining a trained monocular depth estimation model; the monocular depth estimation model adopts a gradient-aware dense skip connection structure, and a loss function used in the training process comprises a mask weighted photometric loss, a geometric consistency loss, a global depth ordering constraint loss, a normal matching loss, an edge relative normal loss and a multi-scale feature alignment loss; a single-frame image to be estimated is acquired and input into the trained monocular depth estimation model, and a predicted depth map is obtained through the gradient-aware dense skip connection structure. The application solves the problems of inconsistent semantics and insufficient spatial alignment of an existing depth estimation model.

Description

Technical Field

[0001] This invention relates to the field of image processing, and specifically to a gradient-aware self-supervised monocular depth estimation method and apparatus for dynamic scenes. Background Technology

[0002] Monocular depth estimation aims to recover the geometric depth information of a scene from a single RGB image. Traditional supervised methods rely on costly ground truth depth data from LiDAR or structured light, which are expensive to acquire and label, and have limited generalization capabilities across scenes. To reduce reliance on ground truth, self-supervised monocular depth estimation typically utilizes the geometric consistency between adjacent frames, synthesizing differentiable views through "depth and camera pose," and using photometric reconstruction error as a training signal.

[0003] However, self-supervised frameworks are usually based on the assumption of static rigid body scenes: when there are dynamic objects such as vehicles and pedestrians or occlusion / revealing phenomena in the scene, the cross-frame correspondence cannot be explained by the rigid body pose, resulting in photometric reconstruction errors containing a large number of residuals that are unrelated to the real geometry; if these residuals are directly backpropagated, they will form incorrect supervision, causing serious distortion of dynamic region depth or even overall training instability.

[0004] On the other hand, commonly used encoder-decoder structures are prone to introducing interpolation smoothing and spatial misalignment during the decoder's upsampling to restore resolution, especially in regions with drastic changes in depth gradients, such as object boundaries and occlusion boundaries. This often results in blurred boundaries, depth discontinuities, or local artifacts in the prediction results. The encoder continuously aggregates high-level semantic information while losing low-level details during multiple downsampling processes. Traditional U-Net-style skip connections often employ direct concatenation or simple addition, forcibly fusing features from different semantic levels and receptive fields at the same decoding layer. This easily leads to semantic inconsistencies and insufficient spatial alignment, weakening the role of skip connections in detail restoration.

[0005] In addition, externally pre-trained deep models can provide single-frame pseudo-depth priors to compensate for insufficient supervision in dynamic regions; however, pseudo-depth usually suffers from scale bias, boundary ambiguity, and local noise. Without screening and constraint mechanisms, direct strong supervision may introduce noisy geometry into the training, thereby destroying geometric consistency. Summary of the Invention

[0006] The purpose of this application is to propose a gradient-aware self-supervised monocular depth estimation method and device for dynamic scenes to address the aforementioned technical problems.

[0007] In a first aspect, the present invention provides a gradient-aware self-supervised monocular depth estimation method for dynamic scenes, comprising the following steps:

[0008] A monocular depth estimation model, a pre-trained pseudo-depth prediction model, and a pose estimation network are constructed. The pre-trained pseudo-depth prediction model and pose estimation network are then used to train the monocular depth estimation model, resulting in a trained monocular depth estimation model. The monocular depth estimation model includes an encoder, an intermediate combination layer, and a decoder, forming a gradient-aware dense skip connection structure. The loss functions used in the training process of the monocular depth estimation model include mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss. The construction process of the mask-weighted photometric loss includes the following steps: acquiring a monocular video sequence and dividing adjacent frames... Monocular images are used as reference and source frames. The reference and source frames are input into a monocular depth estimation model to obtain predicted depth maps for the reference and source frames, respectively. The reference and source frames are then input into a pose estimation network to obtain the corresponding relative camera pose. Camera intrinsic parameters are obtained, and a photometric consistency loss is constructed based on the predicted depth map of the source frame, the relative camera pose, the camera intrinsic parameters, and the reference frame. The predicted depth map of the reference frame is reprojected onto the source frame coordinate system based on the relative camera pose and aligned with the predicted depth map of the source frame. Pixel-level normalized depth inconsistency is calculated, and a self-discovery mask is generated based on the pixel-level normalized depth inconsistency. The photometric consistency loss is weighted using the self-discovery mask to obtain a mask-weighted photometric loss.

[0009] The single-frame image to be estimated is acquired and input into the trained monocular depth estimation model. After processing by a gradient-aware dense skip connection structure, the corresponding predicted depth map is obtained.

[0010] As a preferred method, the process for constructing the photometric uniformity loss is as follows:

[0011] The predicted depth map of the source frame, the relative camera pose, and the camera intrinsic parameters are input into the reprojection and bilinear differentiable sampling algorithm to obtain the corresponding reconstructed image.

[0012] The photometric consistency loss is calculated based on the reference frame and the reconstructed image, as shown in the following formula:

[0013] ;

[0014] in, Indicates the reference frame and the reconstructed image at the 1st Luminous consistency loss of each effective pixel Indicates the weighting coefficient. Indicates the reference frame and the reconstructed image at the 1st Structural similarity of each effective pixel and These represent the reference frame and the reconstructed image at the 1st... The pixel value at each valid pixel location. Represents the L1 norm;

[0015] The calculation process for mask-weighted photometric loss is as follows:

[0016] The predicted depth value of each pixel in the predicted depth map of the reference frame is reprojected into 3D space and mapped to the source frame coordinate system relative to the camera pose to obtain the predicted depth map of the reference frame in the source frame coordinate system. Then, bilinear interpolation sampling is performed on the predicted depth map of the source frame using the predicted depth map of the reference frame in the source frame coordinate system to obtain the aligned depth map of the source frame, as shown in the following formula:

[0017] ;

[0018] ;

[0019] in, This represents the relative camera pose from the reference frame to the source frame. The first in the predicted depth map of the reference frame The predicted depth value of each valid pixel. This represents reprojection and mapping operations. This represents the reference frame in the predicted depth map in the source frame coordinate system. The predicted depth value of each valid pixel. This indicates a bilinear interpolation sampling operation. The first element on the predicted depth map of the source frame The predicted depth value of each valid pixel. The first element on the alignment depth map of the source frame Predicted depth values ​​for each valid pixel;

[0020] The pixel-level normalized depth inconsistency between the predicted depth map of the source frame and the predicted depth map of the reference frame is calculated using the following formula:

[0021] ;

[0022] in, This indicates that the predicted depth map of the source frame and the predicted depth map of the reference frame are compared at the [missing information]th [missing information]. Pixel-level normalized depth inconsistency of each valid pixel;

[0023] The self-discovery mask is calculated based on the pixel-level normalized depth inconsistency, as shown in the following formula:

[0024] ;

[0025] in, Indicates the first Self-discovery mask for each valid pixel;

[0026] The photometric consistency loss is weighted using a self-discovery mask and then averaged to obtain the mask-weighted photometric loss, as shown in the following formula:

[0027] ;

[0028] in, This represents the set of valid pixels that were successfully aligned during the projection and interpolation process. Represents the set of valid pixels The total number of valid pixels in the middle. This represents the mask-weighted luminosity loss;

[0029] The geometric consistency loss is calculated using the following formula:

[0030] ;

[0031] in, This represents the geometric consistency loss.

[0032] As a preferred approach, the construction process for the global depth ranking constraint loss is as follows:

[0033] The reference frame is input into the pre-trained pseudo-depth prediction model to obtain the corresponding pseudo-depth map;

[0034] The self-discovery mask is compared with a threshold. Based on the comparison result, the reference frame is divided into dynamic and static regions. Valid pixels are sampled from the dynamic region and paired with valid pixels from the static region. Additionally, any two valid pixels are sampled from the reference frame and paired. An ordinal label is defined based on the ratio of the pseudo-depth of the two paired valid pixels, as shown in the following formula:

[0035] ;

[0036] in, The depth difference threshold, and These represent pseudo-depth maps. Two valid pixels paired in the middle and pseudo-depth value, Indicates ordinal number label;

[0037] The set of valid pixel pairs consisting of two paired valid pixels in the pseudo-depth map is determined based on the ordinal labels, as shown in the following formula:

[0038] ;

[0039] in, Represents the set of valid pixel pairs;

[0040] The global depth ranking constraint loss is calculated based on the predicted depth values ​​of the paired valid pixels in the pseudo-depth map in the predicted depth map of the reference frame, as shown in the following formula:

[0041] ;

[0042] ;

[0043] in, This represents the total number of valid pixel pairs in the set of valid pixel pairs. and These represent the predicted depth maps of the reference frame. Two valid pixels paired in the middle and The predicted depth value, Represents the sorting loss. This represents the global depth-ordering constraint loss;

[0044] The process of constructing the normal matching loss is as follows:

[0045] The gradients of the predicted depth map and pseudo depth map of the reference frame at each pixel are calculated using finite difference, and the unnormalized normals of the predicted depth map and pseudo depth map of the reference frame are constructed, as shown in the following equation:

[0046] ;

[0047] ;

[0048] in, and These represent the predicted depth maps of the reference frame at the [number]th [year]. The unnormalized normal and pseudo-depth map of the n pixels in the nth The unnormalized normals of each valid pixel. and These represent the first and second predicted depth maps of the reference frame, respectively. The gradient of each pixel in the x-axis and y-axis directions. and These represent the first and second parts of the pseudo-depth map. The gradient of each pixel in the x-axis and y-axis directions;

[0049] Normalize the unnormalized normals of the predicted depth map and the pseudo depth map of the reference frame to obtain the normalized normals of the predicted depth map and the pseudo depth map of the reference frame. Then, calculate the normal matching loss as shown in the following formula:

[0050] ;

[0051] in, This represents the normal matching loss. and These represent the predicted depth maps of the reference frame at the [number]th [year]. The normalized normal and pseudo-depth map of each pixel in the first... Normalized normals of each pixel Indicates the total number of pixels;

[0052] The process of constructing the edge relative normal loss is as follows:

[0053] Edge-guided sampling was used to construct point pairs in the edge regions of the predicted depth map and pseudo depth map of the reference frame, respectively. Point pairs in the predicted depth map of the reference frame The corresponding normal In the point pairs of the pseudo-depth map The corresponding normal Then the edge relative normal loss is:

[0054] ;

[0055] in, Represents point pairs The total number, This represents the loss relative to the edge normal.

[0056] As a preferred method, the construction process of the multi-scale feature alignment loss is as follows:

[0057] The backbone network in the pre-trained pseudo-deep prediction model is used as the teacher model. The reference frame is input into the backbone network of the pre-trained pseudo-deep prediction model, and the i-th token of the teacher feature is extracted from the l-th layer of the backbone network of the pre-trained pseudo-deep prediction model, as shown in the following formula:

[0058] ;

[0059] in, Indicates the reference frame. Let l represent the l-th layer of the backbone network in the pre-trained pseudo-deep prediction model. This represents the i-th token among the teacher features extracted from the l-th layer of the backbone network in the pre-trained pseudo-deep prediction model. Indicates the total number of tokens;

[0060] The encoder of the monocular depth estimation model is used as the student model. The reference frame is input into the encoder of the monocular depth estimation model, and the student features are extracted from the corresponding k-th layer in the encoder of the monocular depth estimation model, as shown in the following formula:

[0061] ;

[0062] in, This represents the k-th layer in the encoder of the monocular depth estimation model. This represents the student feature extracted from the k-th layer in the encoder of the monocular depth estimation model;

[0063] Construct a semantic projector to project the student features extracted from the k-th layer of the encoder in the monocular depth estimation model onto the teacher feature space, thus obtaining the i-th token of the student features extracted from the k-th layer of the encoder in the monocular depth estimation model after projection, as shown in the following formula:

[0064] ;

[0065] in, Represents a semantic projector. This represents the i-th token after projection of the student features extracted from the k-th layer in the encoder of the monocular depth estimation model.

[0066] The multi-scale feature alignment loss is calculated using the following formula:

[0067] ;

[0068] in, Represents the L2 norm. This represents the multi-scale feature alignment loss;

[0069] During the training of the monocular depth estimation model, the total loss function used is a weighted sum of mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss. During the training of the monocular depth estimation model, the parameters of the pre-trained pseudo-depth prediction model are fixed, and the pose estimation network is trained simultaneously.

[0070] Preferably, the semantic projector comprises a multilayer perceptron, a SiLU activation function layer, and an L2 normalization layer connected in sequence. The student features extracted from the k-th layer of the encoder in the monocular depth estimation model are input into the semantic projector, where bilinear interpolation and flattening operations are performed sequentially, followed by the multilayer perceptron, SiLU activation function layer, and L2 normalization layer, as shown in the following equation:

[0071] ;

[0072] in, This represents the bilinear interpolation operation. This represents the spatial dimensions of the teacher model feature map. Indicates the flattening operation. This represents a multilayer perceptron. This indicates the L2 normalized layer.

[0073] Preferably, the pre-trained pseudo-depth prediction model includes the Depth Anything V2 model, where both the encoder and pose estimation networks use the ResNet18 backbone network; the decoder uses the Monodepth2 network, and the intermediate combination layer includes a 1×1 convolutional layer; the gradient-aware dense skip connection structure includes c rows of dense skip connections, where the output feature of the encoder in each row is the feature output of the input image after passing through the encoder in that row; the dense skip connection structure in the e-th row includes the encoder in the e-th row, ce intermediate combination layers in the e-th row, and the decoder in the e-th row, where e=1,2,…,c; in the dense skip connection structure in the e-th row, the input feature of the g-th intermediate combination layer in the e-th row is the output feature of the encoder in the e-th row. The output features of the g-1th intermediate combination layer in row e and the g-1th intermediate combination layer in row e+1 are concatenated to form a feature, where g=2,…,ce; the input features of the first intermediate combination layer in row e are concatenated to form a feature, where g=2,…,ce; the output features of the encoder in row e and the encoder in row e+1 are concatenated to form a feature, and the output features of the encoder in row c are directly input into the decoder in row c. When e is not equal to c, the input features of the decoder in row e are concatenated to form a feature, where e is the output features of the ceth intermediate combination layer in row e and the decoder in row e+1; and the output features of the decoder in row 1 are then input into another decoder to obtain the corresponding predicted depth map.

[0074] Secondly, the present invention provides a gradient-aware self-supervised monocular depth estimation device for dynamic scenes, comprising:

[0075] The model building module is configured to construct a monocular depth estimation model, a pre-trained pseudo-depth prediction model, and a pose estimation network. The pre-trained pseudo-depth prediction model and pose estimation network are then used to train the monocular depth estimation model, resulting in a trained monocular depth estimation model. The monocular depth estimation model includes an encoder, intermediate combination layers, and a decoder, forming a gradient-aware dense skip connection structure. The loss functions used during the training of the monocular depth estimation model include mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss. The construction process of the mask-weighted photometric loss includes the following steps: acquiring monocular video sequences and... Two adjacent monocular images are used as the reference frame and the source frame, respectively. The reference frame and the source frame are input into the monocular depth estimation model to obtain the predicted depth map of the reference frame and the predicted depth map of the source frame. The reference frame and the source frame are input into the pose estimation network to obtain the corresponding relative camera pose. The camera intrinsic parameters are obtained, and a photometric consistency loss is constructed based on the predicted depth map of the source frame, the relative camera pose, the camera intrinsic parameters, and the reference frame. Based on the relative camera pose, the predicted depth map of the reference frame is reprojected onto the coordinate system of the source frame and aligned with the predicted depth map of the source frame. The pixel-level normalized depth inconsistency is calculated, and a self-discovery mask is generated based on the pixel-level normalized depth inconsistency. The photometric consistency loss is weighted using the self-discovery mask to obtain the mask-weighted photometric loss.

[0076] The prediction module is configured to acquire a single-frame image to be estimated and input it into a trained monocular depth estimation model. After processing through a gradient-aware dense skip connection structure, the corresponding predicted depth map is obtained.

[0077] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0078] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0079] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] (1) The gradient-aware self-supervised monocular depth estimation method proposed in this invention for dynamic scenes adopts a gradient-aware dense skip connection structure. The gradient-aware dense skip connection structure provides more sufficient and aligned multi-scale details and semantic fusion in the upsampling stage, which can significantly improve the depth continuity and clarity of high gradient regions such as object boundaries and occlusion boundaries.

[0082] (2) The gradient-aware self-supervised monocular depth estimation method proposed in this invention uses mask-weighted photometric loss and geometric consistency loss during the training process. In the process of constructing mask-weighted photometric loss and geometric consistency loss, the self-discovery mask is automatically constructed using geometric consistency. This can adaptively suppress the pollution of photometric loss by dynamic objects and occluded pixels during the training process, significantly improving the robustness of dynamic scenes without the need for semantic annotation or additional segmentation network.

[0083] (3) The gradient-aware self-supervised monocular depth estimation method for dynamic scenes proposed in this invention utilizes a pre-trained pseudo-depth prediction model to construct hierarchical pseudo-supervision from two aspects: global depth ranking constraints and local geometric structure. Thus, in the training process of the monocular depth estimation model, global depth ranking constraint loss, normal matching loss, and edge relative normal loss are constructed. Compared with direct strong supervision, it is less susceptible to scale bias and noise, and can provide supplementary supervision for dynamic regions and suppress noise propagation. In addition, a multi-scale feature alignment loss is constructed. In this multi-scale feature alignment loss, the ability of the pre-trained teacher model is transferred to the student model, so that it inherits the visual understanding ability of the teacher model and improves the cross-domain generalization ability. Aligning shallow features forces the retention of high-frequency information and solves the feature smoothing problem. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 This is a flowchart illustrating a gradient-aware self-supervised monocular depth estimation method for dynamic scenes, as an embodiment of this application.

[0086] Figure 2 This is a schematic diagram of the training process of the monocular depth estimation model of the gradient-aware self-supervised monocular depth estimation method for dynamic scenes, which is an embodiment of this application.

[0087] Figure 3This is a schematic diagram illustrating the processing of the intermediate combination layer in the gradient-aware self-supervised monocular depth estimation method for dynamic scenes, as described in an embodiment of this application.

[0088] Figure 4 This is a schematic diagram of the decoder structure of the gradient-aware self-supervised monocular depth estimation method for dynamic scenes, as shown in an embodiment of this application.

[0089] Figure 5 This is a schematic diagram of a gradient-aware self-supervised monocular depth estimation device for dynamic scenes, as an embodiment of this application.

[0090] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0091] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0092] Figure 1 This application illustrates an embodiment of a gradient-aware self-supervised monocular depth estimation method for dynamic scenes, comprising the following steps:

[0093] S1. Construct a monocular depth estimation model, a pre-trained pseudo-depth prediction model, and a pose estimation network. Train the monocular depth estimation model using the pre-trained pseudo-depth prediction model and pose estimation network to obtain the trained monocular depth estimation model. The monocular depth estimation model includes an encoder, an intermediate combination layer, and a decoder, forming a gradient-aware dense skip connection structure. The loss functions used in training the monocular depth estimation model include mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss. The construction process of the mask-weighted photometric loss includes the following steps: acquiring a monocular video sequence and using two adjacent monocular images as the reference frame and the source frame; inputting the reference frame and the source frame into the monocular depth estimation model to obtain the predicted depth map of the reference frame and the predicted depth map of the source frame, respectively; and inputting the reference frame and the source frame into the pose estimation network to obtain the corresponding relative camera pose. Obtain camera intrinsics, construct photometric consistency loss based on the predicted depth map of the source frame, relative camera pose, camera intrinsics, and reference frame; reproject the predicted depth map of the reference frame onto the coordinate system of the source frame based on the relative camera pose and align it with the predicted depth map of the source frame, calculate pixel-level normalized depth inconsistency, and generate a self-discovery mask based on the pixel-level normalized depth inconsistency; use the self-discovery mask to weight the photometric consistency loss to obtain the mask-weighted photometric loss.

[0094] In a specific embodiment, the pre-trained pseudo-depth prediction model includes the Depth Anything V2 model, where both the encoder and pose estimation networks use the ResNet18 network as their backbone; the decoder includes the Monodepth2 network, and the intermediate combination layer includes a convolutional layer with a kernel size of 1×1; the gradient-aware dense skip connection structure includes c rows of dense skip connection structures, where the output feature of the encoder in each row is the feature output of the input image after passing through the encoder in that row; the dense skip connection structure in the e-th row includes the encoder in the e-th row, the ce intermediate combination layers in the e-th row, and the decoder in the e-th row, where e=1,2,…,c; in the dense skip connection structure in the e-th row, the input feature of the g-th intermediate combination layer in the e-th row is the output feature of the encoder in the e-th row, the g-th intermediate combination layer, and the g-th intermediate combination layer in the e-th row, the g-th intermediate combination layer, and the g-th intermediate combination layer in the e-th row are the output features of the encoder in the e-th row, the g-th intermediate combination layer, and the g-th intermediate combination layer in the e-th row. The feature is formed by concatenating the output features of the (g-1)th intermediate combination layer in row e with the output features of the (g-1)th intermediate combination layer in row e+1, where g=2,…,ce; the input feature of the first intermediate combination layer in row e is formed by concatenating the output features of the encoder in row e and the output features of the encoder in row e+1; the output features of the encoder in row c are directly input into the decoder in row c; when e is not equal to c, the input feature of the decoder in row e is formed by concatenating the output features of the ce-th intermediate combination layer in row e and the output features of the decoder in row e+1; and the output features of decoder D1 in row 1 are then input into decoder D0 to obtain the corresponding predicted depth map.

[0095] Specifically, embodiments of this application first construct a monocular depth estimation model, a pre-trained pseudo-depth prediction model, and a pose estimation network. In one example, the pre-trained pseudo-depth prediction model uses the Depth Anything V2 model, which is pre-trained. Since masks suppress a large number of dynamic pixels, dynamic regions may suffer from insufficient supervision. Therefore, embodiments of this application introduce a pre-trained pseudo-depth prediction model to generate corresponding pseudo-depth maps for reference frames. Prior knowledge is injected in a hierarchical manner.

[0096] The embodiments of this application design a gradient-aware dense skip connection structure in a monocular depth estimation model. The design goal is to enable regions with significant depth gradient changes to exhibit a smoother transition and more stable response during multi-scale feature fusion. On one hand, a progressive fusion of multi-level features is introduced between the encoder and decoder to reduce the semantic and spatial distribution differences between features at different scales, providing more consistent contextual information near boundaries. On the other hand, dense skip paths continuously deliver detailed features and high-level semantic representations to locations with significant depth changes, such as object contours and occlusion boundaries, alleviating boundary blurring and local artifacts during the upsampling stage. To this end, the original skip connections are reconstructed, and multi-level intermediate combination layers are arranged between the encoder and decoder to form interleaved paths.

[0097] refer to Figure 2 The monocular depth estimation model consists of an encoder, intermediate combination layers, and a decoder, and exists in the form of a gradient-aware dense skip connection structure. The specific structure of the gradient-aware dense skip connection structure is composed of c rows of dense skip connections. The e-th row of dense skip connections consists of the e-th row encoder, ce intermediate combination layers in the e-th row, and the e-th row decoder. Taking c=5 as an example, the first row of dense skip connections consists of the first row encoder, the first row four intermediate combination layers, and the first row decoder. The four intermediate combination layers in the first row are denoted as C1, C2, C3, C4, C5, C6, C7, C8, C9, C1 ... 1,1 C 1,2 C 1,3 and C 1,4 The dense skip connection structure in row 2 consists of the encoder in row 2, the three intermediate combination layers in row 2, and the decoder in row 2. The three intermediate combination layers in row 2 are denoted as C. 2,1 C 2,2 and C 2,3 And so on. The feature formed by concatenating the output features of the encoder in the first row and the output features of the encoder in the second row is input into the intermediate combination layer C. 1,1 The features are fused through a convolutional layer with a kernel size of 1×1. The output features of the encoder in the first row and the intermediate combination layer C are used for feature fusion. 1,1 Output characteristics and intermediate combination layer C 2,1 The features formed by concatenating the output features are input into the intermediate combination layer C. 1,2The features are fused through a 1×1 convolutional layer. Similarly, other intermediate combination layers are connected in the same way until the output feature of the fourth intermediate combination layer in the first row is obtained. The output feature of the fourth intermediate combination layer in the first row is then concatenated with the output feature of the decoder in the second row to form a feature that is input into the decoder in the first row. The calculation process in the dense skip connection structure in the second row is similar, until the dense skip connection structure in the fifth row. The dense skip connection structure in the fifth row has no intermediate combination layers; the output feature of the encoder in the fifth row is directly input into the decoder in the fifth row. The output features are concatenated with the output features of the first intermediate combination layer in row 4 and then input into the decoder in row 4 to obtain the output features of the decoder in row 4. The output features of the decoder in row 4 are then concatenated with the output features of the second intermediate combination layer in row 3 and then input into the decoder in row 3 to obtain the output features of the decoder in row 3, and so on, until the output features of the decoder in row 1 are obtained. The output features of the decoder in row 1 are then input into a decoder to obtain the corresponding predicted depth map. During upsampling to restore resolution, depth gradients and structural details at object boundaries and occlusion boundaries are preserved as much as possible. In one example, both the encoder and pose estimation network use the standard ResNet18 backbone network, which includes several sequentially connected residual blocks. To accommodate dual-frame input, the number of channels in the first convolutional layer of the pose estimation network is increased from 3 to 6. (Reference...) Figure 3 The intermediate combination layers are all densely aggregated using convolutional layers with a kernel size of 1×1. (Reference) Figure 4 The decoder follows the Monodepth2 architecture, specifically comprising a globally average pooling layer, two fully connected layers, a sigmoid activation function layer, and a 1×1 convolutional layer connected in sequence. Channel attention weighting is achieved through the globally average pooling layer, the two fully connected layers, and the sigmoid activation function layer, and then densely aggregates all skip features in the same row through the 1×1 convolutional layer. In other embodiments, the encoder and decoder can be replaced with ResNet50, MobileNet, ConvNeXt, or Transformer encoder structures; the intermediate combining layers in the gradient-aware dense skip connection structure can be implemented using deformable convolutions or attention gating. The training input can be expanded from two frames to multiple frames, and the pose estimation network can use a stronger temporal model; optional semantic priors or optical flow branches can also be added to further improve the occlusion handling capability in extreme dynamic scenes.

[0098] In terms of network structure, the monocular depth estimation model mentioned in the embodiments of this application utilizes a gradient-aware dense skip connection structure to enhance boundary detail recovery. During the training of this monocular depth estimation model, two network models are needed for assistance: a pre-trained pseudo-depth prediction model and a pose estimation network. Simultaneously, the pose estimation network is trained during the training of the monocular depth estimation model, and the parameters of the pre-trained pseudo-depth prediction model are fixed. The training of this monocular depth estimation model requires the use of six losses: mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss. The total loss function used in the training of this monocular depth estimation model is a weighted sum of the mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss, as shown in the following equation:

[0099] ;

[0100] in, A mask-weighted luminosity loss is used to provide constraints for view reconstruction of static regions; This is the geometric consistency loss, used to provide cross-frame depth consistency constraints; This is the normal matching loss, used to provide constraints on the local surface tilt direction; The global depth ranking constraint loss is used to provide scale-invariant depth proximity constraints. The edge relative normal loss is used to provide geometric abrupt change constraints at the boundary; This is a multi-scale feature alignment loss used to provide distillation constraints for intermediate features. The weighting coefficients represent the mask-weighted photometric loss. The weighting coefficients representing the geometric consistency loss. The weighting coefficients represent the normal matching loss. The weight coefficients representing the global depth-ordering constraint loss are: The weighting coefficients represent the edge loss relative to the normal. These represent the weight coefficients of the multi-scale feature alignment loss. In one example, , , Backpropagation and the Adam optimizer are used to update the parameters of the monocular depth estimation model and the pose estimation network.

[0101] The following section details the construction process of five losses: mask-weighted photometric loss, geometric consistency loss, global depth sorting constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss.

[0102] In a specific embodiment, the process of constructing the photometric consistency loss is as follows:

[0103] The predicted depth map of the source frame, the relative camera pose, and the camera intrinsic parameters are input into the reprojection and bilinear differentiable sampling algorithm to obtain the corresponding reconstructed image.

[0104] The photometric consistency loss is calculated based on the reference frame and the reconstructed image, as shown in the following formula:

[0105] ;

[0106] in, Indicates the reference frame and the reconstructed image at the 1st Luminous consistency loss of each effective pixel Indicates the weighting coefficient. Indicates the reference frame and the reconstructed image at the 1st Structural similarity of each effective pixel and These represent the reference frame and the reconstructed image at the 1st... The pixel value at each valid pixel location. Represents the L1 norm;

[0107] The calculation process for mask-weighted photometric loss is as follows:

[0108] The predicted depth value of each pixel in the predicted depth map of the reference frame is reprojected into 3D space, and then relative to the camera pose. Mapping to the source frame coordinate system, we obtain the predicted depth map of the reference frame in the source frame coordinate system. Then, we perform bilinear interpolation sampling on the predicted depth map of the source frame using the predicted depth map of the reference frame in the source frame coordinate system to obtain the aligned depth map of the source frame, as shown in the following formula:

[0109] ;

[0110] ;

[0111] in, This represents the relative camera pose from the reference frame to the source frame. The first in the predicted depth map of the reference frame The predicted depth value of each valid pixel. This represents reprojection and mapping operations. This represents the reference frame in the predicted depth map in the source frame coordinate system. The predicted depth value of each valid pixel. This indicates a bilinear interpolation sampling operation. The first element on the predicted depth map of the source frame The predicted depth value of each valid pixel. The first element on the alignment depth map of the source frame Predicted depth values ​​for each valid pixel;

[0112] The pixel-level normalized depth inconsistency between the predicted depth map of the source frame and the predicted depth map of the reference frame is calculated using the following formula:

[0113] ;

[0114] in, This indicates that the predicted depth map of the source frame and the predicted depth map of the reference frame are compared at the [missing information]th [missing information]. Pixel-level normalized depth inconsistency of each valid pixel;

[0115] The self-discovery mask is calculated based on the pixel-level normalized depth inconsistency, as shown in the following formula:

[0116] ;

[0117] in, Indicates the first Self-discovery mask for each valid pixel;

[0118] The photometric consistency loss is weighted using a self-discovery mask and then averaged to obtain the mask-weighted photometric loss, as shown in the following formula:

[0119] ;

[0120] in, This represents the set of valid pixels that were successfully aligned during the projection and interpolation process. Represents the set of valid pixels The total number of valid pixels in the middle. This represents the mask-weighted luminosity loss;

[0121] The geometric consistency loss is calculated using the following formula:

[0122] ;

[0123] in, This represents the geometric consistency loss.

[0124] Specifically, two adjacent video frames in a monocular video sequence are obtained as reference frames. With source frame and obtain camera intrinsic parameters Reference frame With source frame The data are input into a monocular depth estimation model to obtain predicted depth maps for two frames. and Reference frame With source frame The input is fed into the pose estimation network, and the output is a 6-dimensional relative camera pose between the two. Based on predicted depth map Relative camera pose With camera internal reference The source frame is reprojected using a bilinear differentiable sampling algorithm. Transform to the reference frame coordinate system to obtain the reconstructed image. Compare with reference frames. With reconstructed images The photometric consistency loss is obtained by analyzing the image differences. The photometric consistency loss adopts a hybrid form of structural similarity (SSIM) and L1 pixel difference.

[0125] In self-supervised monocular depth estimation, the scene is typically assumed to be a rigid body. However, this assumption is difficult to hold in real-world scenes. On one hand, dynamic objects such as foreground vehicles and pedestrians produce non-rigid displacements inconsistent with the camera's own motion; on the other hand, changes in viewpoint lead to frequent occlusion and re-occlusion. These two types of pixels often exhibit significant photometric residuals after reprojection. If these residuals are directly included in the photometric reconstruction loss, they will be mistakenly treated as supervisory signals and backpropagated, introducing a large number of unreliable gradients during training and weakening the network's ability to model real geometric relationships. Therefore, geometric consistency constraints are explicitly introduced into the training process of the monocular depth estimation model, and a self-discovery mask is constructed accordingly to suppress the interference of dynamic objects and occluded regions on training. The specific steps are as follows:

[0126] First, calculate the alignment depth of the source frame that is pixel-aligned with the reference frame. Then, align each pixel on the reference frame according to the predicted depth map. Reprojected into 3D space and relative to the camera pose Mapping to the source frame coordinate system yields the depth map of the reference frame from the source frame's perspective. Because the projection is not completely located Within the grid, therefore, in the predicted depth map of the source frame. Bilinear interpolation sampling is performed on the source frame to obtain the alignment depth map.

[0127] Secondly, to achieve a relatively balanced penalty effect across different depth ranges, a pixel-level normalized depth inconsistency is defined. The formula for this pixel-level normalized depth inconsistency uses the absolute depth difference as the numerator and the sum of depths as the denominator, making the inconsistencies across different distance ranges comparable, thus achieving a scale-invariant consistency measure.

[0128] Finally, geometric consistency loss Defined as the average pixel-level normalized depth inconsistency of all valid pixels. This is achieved by minimizing... The predicted depth of adjacent frames is constrained to be geometrically consistent in three-dimensional space, thereby gradually eliminating depth instability caused by scale drift or local error estimation during training.

[0129] Besides serving as a geometric constraint, pixel-level normalized depth inconsistency can also be used to automatically locate pixels that violate static scene assumptions. Since dynamic objects and occluded regions are difficult to align across views using rigid body transformations, their corresponding... The size of the background area is often significantly larger; while the static background area exhibits better geometric consistency across multiple views, with corresponding pixel-level normalized depth inconsistency. Based on this phenomenon, a self-discovery mask is developed. The result is analytically derived from the normalized depth inconsistency. Pixels violating the static scene assumption are automatically located based on the pixel-level normalized depth inconsistency, and a self-discovery mask is calculated based on the pixel-level normalized depth inconsistency. The numerical range of the self-discovery mask is... At pixels that satisfy the rigid body assumption, The corresponding self-discovery mask is close to 0. Approximately 1; however, in areas such as dynamic objects and occlusion boundaries, geometric inconsistencies lead to... The value is relatively large, thus yielding a self-discovery mask close to 0. Compared to explicit semantic segmentation or additional motion branches, this geometrically consistent self-discovery mask is directly derived from the network's predicted depth and pose, requiring no additional network or annotation overhead. Considering that the photometric consistency assumption is severely violated in dynamic and occluded regions, the discovery mask... Introducing photometric consistency loss and utilizing self-discovery masks The photometric consistency loss is weighted and averaged to obtain the mask-weighted photometric loss. During training, the mask is self-discovered. It adaptively suppresses the contributions of dynamic objects, occlusion boundaries, and other geometrically inconsistent regions, making the optimization process primarily dominated by static, geometrically consistent regions. This significantly reduces the interference of noise supervision on the monocular depth estimation model and improves the robustness of depth prediction in complex dynamic scenes. Compared with self-supervised frameworks that rely solely on photometric reconstruction, it provides a cleaner and more reliable supervised foundation for subsequent feature modeling and loss design without introducing additional networks and semantic annotations.

[0130] In a specific embodiment, the process of constructing the global depth ranking constraint loss is as follows:

[0131] The reference frame is input into the pre-trained pseudo-depth prediction model to obtain the corresponding pseudo-depth map;

[0132] The self-discovery mask is compared with a threshold. Based on the comparison result, the reference frame is divided into dynamic and static regions. Valid pixels are sampled from the dynamic region and paired with valid pixels from the static region. Additionally, any two valid pixels are sampled from the reference frame and paired. An ordinal label is defined based on the ratio of the pseudo-depth of the two paired valid pixels, as shown in the following formula:

[0133] ;

[0134] in, The depth difference threshold, and These represent pseudo-depth maps. Two valid pixels paired in the middle and pseudo-depth value, Indicates ordinal number label;

[0135] The set of valid pixel pairs consisting of two paired valid pixels in the pseudo-depth map is determined based on the ordinal labels, as shown in the following formula:

[0136] ;

[0137] in, Represents the set of valid pixel pairs;

[0138] The global depth ranking constraint loss is calculated based on the predicted depth values ​​of the paired valid pixels in the pseudo-depth map in the predicted depth map of the reference frame, as shown in the following formula:

[0139] ;

[0140] ;

[0141] in, This represents the total number of valid pixel pairs in the set of valid pixel pairs. and These represent the predicted depth maps of the reference frame. Two valid pixels paired in the middle and The predicted depth value, Represents the sorting loss. This represents the global depth-ordering constraint loss;

[0142] The process of constructing the normal matching loss is as follows:

[0143] The gradients of the predicted depth map and pseudo depth map of the reference frame at each pixel are calculated using finite difference, and the unnormalized normals of the predicted depth map and pseudo depth map of the reference frame are constructed, as shown in the following equation:

[0144] ;

[0145] ;

[0146] in, and These represent the predicted depth maps of the reference frame at the [number]th [year]. The unnormalized normal and pseudo-depth map of the n pixels in the nth The unnormalized normals of each valid pixel. and These represent the first and second predicted depth maps of the reference frame, respectively. The gradient of each pixel in the x-axis and y-axis directions. and These represent the first and second parts of the pseudo-depth map. The gradient of each pixel in the x-axis and y-axis directions;

[0147] Normalize the unnormalized normals of the predicted depth map and the pseudo depth map of the reference frame to obtain the normalized normals of the predicted depth map and the pseudo depth map of the reference frame. Then, calculate the normal matching loss as shown in the following formula:

[0148] ;

[0149] in, This represents the normal matching loss. and These represent the predicted depth maps of the reference frame at the [number]th [year]. The normalized normal and pseudo-depth map of each pixel in the first... Normalized normals of each pixel Indicates the total number of pixels;

[0150] The process of constructing the edge relative normal loss is as follows:

[0151] Edge-guided sampling was used to construct point pairs in the edge regions of the predicted depth map and pseudo depth map of the reference frame, respectively. Point pairs in the predicted depth map of the reference frame The corresponding normal In the point pairs of the pseudo-depth map The corresponding normal Then the edge relative normal loss is:

[0152] ;

[0153] in, Represents point pairs The total number, This represents the loss relative to the edge normal.

[0154] Specifically, during the training process of the monocular depth estimation model, although geometric consistency loss and self-discovery masks can effectively reduce noise introduced by dynamic objects and occluded regions, these mechanisms also remove a large number of dynamic pixels from the supervision, resulting in these regions having almost no effective constraints and significantly weaker depth prediction quality. In contrast, the pedestal model trained on large-scale data exhibits better prior characteristics in single-frame depth estimation. From a global perspective, the distance relationships between objects are basically correct; at the local scale, the surface shape is smooth and the normal structure is reasonable; at the object boundaries, the depth contour is clear and the boundary transition is natural. Based on this, the embodiments of this application further introduce an externally pre-trained pseudo-depth prediction model, Depth Anything v2, to generate corresponding pseudo-depth maps for reference frames on top of the self-supervised training framework. A hierarchical pseudo-supervision is constructed from two aspects: global depth ranking constraints and local geometric structure. This provides additional constraints for dynamic regions that are originally under-supervised, while minimizing the accumulation and propagation of pseudo-label noise in the network. In other embodiments, the pseudo-depth map can be generated by other basic depth models or by stereo imaging; in addition to ranking loss, global constraints can take the form of piecewise monotonic constraints, contrastive learning, etc.

[0155] First, a global depth ranking constraint is applied, meaning that instead of forcing the fitting of absolute values ​​from the pseudo-depth map, its proximity information is utilized. This is based on a self-discovery mask. The reference frame is coarsely divided into dynamic and static regions. Specifically, the self-discovery mask is... The pixels with the lowest values ​​are considered potential dynamic regions, while the rest are considered static pixels. Pixels are sampled from the dynamic regions. Pixels in static areas Pairing is performed, and several additional point pairs are randomly sampled throughout the reference frame to form a set of pixel point pairs. To reduce false label noise, only ordinal labels are retained. The set of pixel pairs is defined as the set of valid pixel pairs. , and will Pixel pairs are ignored, and a ranking constraint is imposed on the pseudo-depth map, defining a ranking loss. The ranking loss of all valid pixel pairs is averaged to obtain the global depth ranking constraint loss. This global depth ranking constraint loss only utilizes the proximity relationship of pseudo-depth to explicitly regulate the depth order between dynamic objects and static backgrounds, without overly relying on the absolute value of pseudo-depth.

[0156] Secondly, the predicted depth map of the reference frame is refined using the geometric prior of the pseudo-depth map at a local scale. Specifically, the consistency of surface normal trends is constrained in non-edge regions to regulate local surface orientation; and the relative bends and contour shapes at image edges or high-gradient regions are constrained to make boundaries sharper and reduce upsampling smoothing and artifacts.

[0157] In a specific embodiment, the construction process of the multi-scale feature alignment loss is as follows:

[0158] The backbone network in the pre-trained pseudo-deep prediction model is used as the teacher model. The reference frame is input into the backbone network of the pre-trained pseudo-deep prediction model, and the i-th token of the teacher feature is extracted from the l-th layer of the backbone network of the pre-trained pseudo-deep prediction model, as shown in the following formula:

[0159] ;

[0160] in, Indicates the reference frame. Let l represent the l-th layer of the backbone network in the pre-trained pseudo-deep prediction model. This represents the i-th token among the teacher features extracted from the l-th layer of the backbone network in the pre-trained pseudo-deep prediction model. Indicates the total number of tokens;

[0161] The encoder of the monocular depth estimation model is used as the student model. The reference frame is input into the encoder of the monocular depth estimation model, and the student features are extracted from the corresponding k-th layer in the encoder of the monocular depth estimation model, as shown in the following formula:

[0162] ;

[0163] in, This represents the k-th layer in the encoder of the monocular depth estimation model. This represents the student feature extracted from the k-th layer in the encoder of the monocular depth estimation model;

[0164] Construct a semantic projector to project the student features extracted from the k-th layer of the encoder in the monocular depth estimation model onto the teacher feature space, thus obtaining the i-th token of the student features extracted from the k-th layer of the encoder in the monocular depth estimation model after projection, as shown in the following formula:

[0165] ;

[0166] in, Represents a semantic projector. This represents the i-th token after projection of the student features extracted from the k-th layer in the encoder of the monocular depth estimation model.

[0167] The multi-scale feature alignment loss is calculated using the following formula:

[0168] ;

[0169] in, Represents the L2 norm. This represents the multi-scale feature alignment loss;

[0170] The total loss function used in the training process of the monocular depth estimation model is a weighted sum of mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss and multi-scale feature alignment loss. In the training process of the monocular depth estimation model, the parameters of the pre-trained pseudo-depth prediction model are fixed and the pose estimation network is trained simultaneously.

[0171] In a specific embodiment, the semantic projector includes a multilayer perceptron, a SiLU activation function layer, and an L2 normalization layer connected in sequence. The student features extracted from the k-th layer of the encoder in the monocular depth estimation model are input into the semantic projector. First, bilinear interpolation and flattening operations are performed sequentially, and then the signal passes through the multilayer perceptron, SiLU activation function layer, and L2 normalization layer sequentially, as shown in the following equation:

[0172] ;

[0173] in, This represents the bilinear interpolation operation. This represents the spatial dimensions of the teacher model feature map. Indicates the flattening operation. This represents a multilayer perceptron. This indicates the L2 normalized layer.

[0174] Specifically, the embodiments of this application, through a hierarchical constraint mechanism, robustly utilize pseudo-depth map priors from two dimensions: global depth ranking relationships and local geometry, effectively alleviating the problem of missing supervision signals in dynamic regions. However, the above constraints essentially still operate on the depth value space, that is, using the topological structure and geometric attributes of the pseudo-depth map as the supervision target. This depth value-level constraint has inherent limitations: firstly, the backbone network DINOv2 of the pre-trained pseudo-depth prediction model Depth AnythingV2 extracts rich semantic, texture, and object component information in the intermediate layers, while the depth map, as the final scalar output, inevitably compresses the above high-dimensional features into single-channel values, creating an information bottleneck; secondly, constraining only the proximity relationships and geometry of depth values ​​cannot guarantee that the student model learns a feature representation consistent with the teacher model, which may lead to a shift in feature mapping rules. To overcome the above limitations of the depth value space constraint, the embodiments of this application further introduce multi-scale feature alignment loss, extending the supervision dimension from the depth value space to the feature representation space, realizing explicit inheritance of the intermediate layer knowledge of the teacher model.

[0175] In one example, in an embodiment of this application, the backbone network DINOv2 of the pre-trained pseudo-deep prediction model DepthAnything V2 is selected as the teacher model. The reference frame is input into the teacher model, and the teacher features output from three representative layers are selected for feature alignment. =5 layers of shallow features are used to capture high-frequency geometric information such as edges and textures; =12 layers of mid-layer features are used to capture object parts and local shape information; =24 layers of deep features are used to capture semantic categories and global contextual information.

[0176] In one example, the encoder in the monocular depth estimation model is selected as the student model in the embodiment of this application. Taking the ResNet18 network as an example, after the reference frame is input into the student model, the student features corresponding to the teacher level are selected for feature alignment: In shallow feature alignment, the teacher model selects Layer 5 (1024-dimensional) of the backbone network DINOv2, and the student model selects Layer 2 (128 channels) of the ResNet18 network; In mid-level feature alignment, the teacher model selects Layer 12 (1024-dimensional) of the backbone network DINOv2, and the student model selects Layer 3 (256 channels) of the ResNet18 network; In deep feature alignment, the teacher model selects Layer 24 (1024-dimensional) of the backbone network DINOv2, and the student model selects Layer 4 (512 channels) of the ResNet18 network.

[0177] Due to the architectural heterogeneity between the teacher and student models, embodiments of this application design a semantic projector. The semantic projector maps student features to the teacher feature space. It consists of the following components: spatial resampling adjusts the spatial size through bilinear interpolation; flattening and transpose convert the two-dimensional feature map into a token sequence; a three-layer multilayer perceptron (MLP) with SiLU activation function performs nonlinear feature transformation; and L2 normalization normalizes the features to a unit hypersphere.

[0178] The multi-scale feature alignment loss is a weighted sum of the losses from each layer. This multi-scale feature alignment loss has four technical advantages: by aligning the features output from the intermediate layers of DINOv2, the student model can learn the feature representation of the teacher model, thus inheriting the visual understanding ability of the teacher model; aligning shallow features forces the network to retain high-frequency spatial information, solving the feature smoothing problem; inheriting the general visual priors learned by DINOv2 on large-scale data improves cross-domain generalization ability; and the semantic projector is used only during the training phase, achieving enhancement during training without additional overhead during inference.

[0179] The embodiments of this application automatically discover unreliable pixels through geometric consistency and generate self-discovery masks, reducing their weight in photometric supervision and thus purifying the training signal. Since supervision becomes insufficient after suppressing dynamic regions, the embodiments of this application introduce pseudo-depth maps generated by an externally pre-trained pseudo-depth prediction model to supplement constraints in a hierarchical manner of global distance relationships and local geometry. Furthermore, to fully inherit the visual representation capabilities of the pre-trained model, the embodiments of this application extend supervision from the depth value space to the feature representation space, achieving explicit distillation of intermediate layer knowledge through multi-scale feature alignment loss.

[0180] A total loss function is constructed, consisting of mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss. A monocular depth estimation model is trained based on this total loss function to obtain the trained monocular depth estimation model.

[0181] S2: Acquire the single-frame image to be estimated and input it into the trained monocular depth estimation model. After processing by the gradient-aware dense skip connection structure, the corresponding predicted depth map is obtained.

[0182] Specifically, the trained monocular depth estimation model is deployed, and a single frame image to be estimated is input. The trained monocular depth estimation model directly outputs the corresponding predicted depth map. During the inference phase, it is not necessary to run training structures such as pose estimation networks and pre-trained pseudo-depth prediction models, thereby shortening the inference chain, reducing computing power and latency overhead, and facilitating deployment in resource-constrained devices such as vehicles or robots.

[0183] Further reference Figure 5 As an implementation of the methods shown in the above figures, this application provides an embodiment of a gradient-aware self-supervised monocular depth estimation device for dynamic scenes. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0184] This application provides a gradient-aware self-supervised monocular depth estimation device for dynamic scenes, including:

[0185] Model building module 1 is configured to construct a monocular depth estimation model, a pre-trained pseudo-depth prediction model, and a pose estimation network. The pre-trained pseudo-depth prediction model and pose estimation network are then used to train the monocular depth estimation model, resulting in a trained monocular depth estimation model. The monocular depth estimation model includes an encoder, intermediate combination layers, and a decoder, forming a gradient-aware dense skip connection structure. The loss functions used during the training of the monocular depth estimation model include mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss. The construction process of the mask-weighted photometric loss includes the following steps: acquiring monocular video sequences and... Two adjacent monocular images are used as the reference frame and the source frame, respectively. The reference frame and the source frame are input into the monocular depth estimation model to obtain the predicted depth map of the reference frame and the predicted depth map of the source frame. The reference frame and the source frame are input into the pose estimation network to obtain the corresponding relative camera pose. Camera intrinsic parameters are obtained, and a photometric consistency loss is constructed based on the predicted depth map of the source frame, the relative camera pose, the camera intrinsic parameters, and the reference frame. Based on the relative camera pose, the predicted depth map of the reference frame is reprojected onto the coordinate system of the source frame and aligned with the predicted depth map of the source frame. Pixel-level normalized depth inconsistency is calculated, and a self-discovery mask is generated based on the pixel-level normalized depth inconsistency. The photometric consistency loss is weighted using the self-discovery mask to obtain the mask-weighted photometric loss.

[0186] Prediction module 2 is configured to acquire a single-frame image to be estimated and input it into a trained monocular depth estimation model. After processing by a gradient-aware dense skip connection structure, the corresponding predicted depth map is obtained.

[0187] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 6 As shown, the electronic device of this embodiment includes a processor 601 and a memory 602; wherein the memory 602 is used to store computer execution instructions; and the processor 601 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0188] Alternatively, the memory 602 can be either standalone or integrated with the processor 601.

[0189] When the memory 602 is set up independently, the electronic device also includes a bus 603 for connecting the memory 602 and the processor 601.

[0190] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 601, implement the above method.

[0191] This invention also provides a computer program product, including a computer program that, when executed by a processor 601, implements the above-described method.

[0192] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0193] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0194] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0195] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 601 to execute some steps of the methods of the various embodiments of this application.

[0196] It should be understood that the processor 601 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 601 can be any conventional processor 601. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 601 executing the steps, or as a combination of hardware and software modules within the processor 601 executing the steps.

[0197] The memory 602 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0198] Bus 603 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 603 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 603 in the accompanying drawings of this application is not limited to only one bus 603 or one type of bus 603.

[0199] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0200] An exemplary storage medium is coupled to a processor 601, enabling the processor 601 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor 601. The processor 601 and the storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor 601 and the storage medium can exist as discrete components in an electronic device or a host device.

[0201] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A gradient-aware self-supervised monocular depth estimation method for dynamic scenes, characterized in that, Includes the following steps: A monocular depth estimation model, a pre-trained pseudo-depth prediction model, and a pose estimation network are constructed. The monocular depth estimation model is then trained using the pre-trained pseudo-depth prediction model and pose estimation network to obtain a trained monocular depth estimation model. The monocular depth estimation model includes an encoder, an intermediate combination layer, and a decoder, forming a gradient-aware dense skip connection structure. The loss functions used in the training process of the monocular depth estimation model include mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss. The process of constructing the mask-weighted photometric loss includes the following steps: acquiring a monocular video sequence and using two adjacent monocular images as the reference frame and the source frame; inputting the reference frame and the source frame into the monocular depth estimation model respectively to obtain the predicted depth map of the reference frame and the predicted depth map of the source frame; The reference frame and the source frame are input into the pose estimation network to obtain the corresponding relative camera pose; camera intrinsic parameters are obtained, and a photometric consistency loss is constructed based on the predicted depth map of the source frame, the relative camera pose, the camera intrinsic parameters, and the reference frame. The construction process of the photometric consistency loss is as follows: The predicted depth map of the source frame, the relative camera pose, and the camera intrinsic parameters are input into the reprojection and bilinear differentiable sampling algorithm to obtain the corresponding reconstructed image; The photometric consistency loss is calculated based on the reference frame and the reconstructed image, as shown in the following formula: ； in, This indicates that the reference frame and the reconstructed image are in the first... Luminous consistency loss of each effective pixel Indicates the weighting coefficient. This indicates that the reference frame and the reconstructed image are in the first... Structural similarity of each effective pixel and The reference frame and the reconstructed image respectively represent the first... The pixel value at each valid pixel location. Represents the L1 norm; Based on the relative camera pose, the predicted depth map of the reference frame is reprojected onto the source frame coordinate system and aligned with the predicted depth map of the source frame. Pixel-level normalized depth inconsistency is calculated, and a self-discovery mask is generated based on the pixel-level normalized depth inconsistency. The photometric consistency loss is weighted using the self-discovery mask to obtain the mask-weighted photometric loss. The calculation process of the mask-weighted photometric loss is as follows: The predicted depth value of each pixel in the predicted depth map of the reference frame is reprojected into 3D space and mapped to the source frame coordinate system through the relative camera pose to obtain the predicted depth map of the reference frame in the source frame coordinate system. Then, bilinear interpolation sampling is performed on the predicted depth map of the source frame using the predicted depth map of the reference frame in the source frame coordinate system to obtain the aligned depth map of the source frame, as shown in the following formula: ； ； in, This represents the relative camera pose from the reference frame to the source frame. The first reference frame is represented on the predicted depth map. The predicted depth value of each valid pixel. This indicates reprojection and mapping operations. The reference frame represents the first reference frame in the predicted depth map of the source frame coordinate system. The predicted depth value of each valid pixel. This indicates a bilinear interpolation sampling operation. The first element on the predicted depth map of the source frame. The predicted depth value of each valid pixel. The first element on the alignment depth map of the source frame Predicted depth values ​​for each valid pixel; The pixel-level normalized depth inconsistency between the predicted depth map of the source frame and the predicted depth map of the reference frame is calculated using the following formula: ； in, This indicates that the predicted depth map of the source frame and the predicted depth map of the reference frame are compared at the 1st... Pixel-level normalized depth inconsistency of each valid pixel; The self-discovery mask is calculated based on the pixel-level normalized depth inconsistency, as shown in the following formula: ； in, Indicates the first Self-discovery mask for each valid pixel; The photometric consistency loss is weighted using the self-discovery mask and the mean value is calculated to obtain the mask-weighted photometric loss, as shown in the following formula: ； in, This represents the set of valid pixels that were successfully aligned during the projection and interpolation process. Represents the set of valid pixels The total number of valid pixels in the middle. This represents the mask-weighted luminosity loss; The geometric consistency loss is calculated using the following formula: ； in, This represents the geometric consistency loss; A single-frame image to be estimated is acquired and input into the trained monocular depth estimation model. After processing by a gradient-aware dense skip connection structure, the corresponding predicted depth map is obtained.

2. The gradient-aware self-supervised monocular depth estimation method for dynamic scenes according to claim 1, characterized in that, The construction process of the global depth ranking constraint loss, normal matching loss, and edge relative normal loss is as follows: The reference frame is input into the pre-trained pseudo-depth prediction model to obtain the corresponding pseudo-depth map; The self-discovery mask is compared with a preset mask threshold. Based on the comparison result, the reference frame is divided into dynamic and static regions. Valid pixels are sampled from the dynamic region and paired with valid pixels from the static region. Additionally, any two valid pixels are sampled from the reference frame for pairing. An ordinal label is defined based on the ratio of the pseudo-depth of the two paired valid pixels, as shown in the following formula: ； in, The depth difference threshold, and These respectively represent the pseudo-depth maps Two valid pixels paired in the middle and pseudo-depth value, Indicates ordinal number label; The set of valid pixel pairs consisting of two paired valid pixels in the pseudo-depth map is determined based on the ordinal labels, as shown in the following formula: ； in, Represents the set of valid pixel pairs; The global depth ranking constraint loss is calculated based on the predicted depth values ​​of the paired valid pixels in the pseudo-depth map and their corresponding values ​​in the predicted depth map of the reference frame, as shown in the following formula: ； ； in, This represents the total number of valid pixel pairs in the set of valid pixel pairs. and These represent the predicted depth maps of the reference frames, respectively. Two valid pixels paired in the middle and The predicted depth value, Represents the sorting loss. This represents the global depth-ordering constraint loss; The process of constructing the normal matching loss is as follows: The gradients of the predicted depth map and the pseudo depth map of the reference frame at each pixel are calculated using finite difference, and the unnormalized normals of the predicted depth map and the pseudo depth map of the reference frame are constructed, as shown in the following equation: ； ； in, and The predicted depth map of the reference frame is respectively represented in the th... The unnormalized normal of each pixel and the pseudo-depth map at the The unnormalized normals of each pixel and These represent the first and second predicted depth maps of the reference frame, respectively. The gradient of each pixel in the x-axis and y-axis directions. and These respectively represent the first and second parts of the pseudo-depth map. The gradient of each pixel in the x-axis and y-axis directions; The unnormalized normals of the predicted depth map of the reference frame and the unnormalized normals of the pseudo depth map are normalized to obtain the normalized normals of the predicted depth map of the reference frame and the normalized normals of the pseudo depth map. The normal matching loss is then calculated as shown in the following formula: ； in, This represents the normal matching loss. and The predicted depth map of the reference frame is respectively represented in the th... The normalized normal of each pixel and the pseudo-depth map at the 1st Normalized normals of each pixel Indicates the total number of pixels; The process for constructing the edge-relative-normal loss is as follows: Edge-guided sampling is used to construct point pairs in the edge regions of the predicted depth map and the pseudo depth map of the reference frame, respectively. The points in the predicted depth map of the reference frame are paired The corresponding normal is denoted as The point pairs of the pseudo-depth map The corresponding normal is denoted as Then the edge relative normal loss is: ； in, Represents point pairs The total number, This represents the loss relative to the edge normal.

3. The gradient-aware self-supervised monocular depth estimation method for dynamic scenes according to claim 1, characterized in that, The construction process of the multi-scale feature alignment loss is as follows: The backbone network in the pre-trained pseudo-deep prediction model serves as the teacher model, and the reference frame is input into the backbone network of the pre-trained pseudo-deep prediction model. The i-th token of the teacher feature is extracted from the l-th layer of the teacher model, as shown in the following formula: ； in, Indicates the reference frame. This represents the l-th layer of the backbone network in the pre-trained pseudo-deep prediction model. This represents the i-th token among the teacher features extracted from the l-th layer of the backbone network in the pre-trained pseudo-deep prediction model. Indicates the total number of tokens; The encoder of the monocular depth estimation model serves as the student model, and the reference frame is input into the encoder of the monocular depth estimation model. The student features are extracted from the k-th layer corresponding to the encoder of the monocular depth estimation model, as shown in the following formula: ； in, This represents the k-th layer in the encoder of the monocular depth estimation model. This represents the student feature extracted from the k-th layer of the encoder in the monocular depth estimation model; A semantic projector is constructed to project the student features extracted from the k-th layer of the encoder of the monocular depth estimation model into the space of teacher features, thereby obtaining the i-th token of the student features extracted from the k-th layer of the encoder of the monocular depth estimation model after projection, as shown in the following formula: ； in, Represents a semantic projector. This represents the i-th token after projection of the student features extracted from the k-th layer of the encoder in the monocular depth estimation model; The multi-scale feature alignment loss is calculated using the following formula: ； in, Describing the L2 norm, This represents the multi-scale feature alignment loss; The total loss function used in the training process of the monocular depth estimation model is a weighted sum of mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss. In the training process of the monocular depth estimation model, the parameters of the pre-trained pseudo-depth prediction model are fixed, and the pose estimation network is trained simultaneously.

4. The gradient-aware self-supervised monocular depth estimation method for dynamic scenes according to claim 3, characterized in that, The semantic projector comprises a multilayer perceptron, a SiLU activation function layer, and an L2 normalization layer connected in sequence. The student features extracted from the k-th layer corresponding to the encoder of the monocular depth estimation model are input into the semantic projector, and bilinear interpolation, flattening, multilayer perceptron processing, SiLU activation function, and L2 normalization are performed sequentially, as shown in the following equation: ； in, This represents the bilinear interpolation operation. This represents the spatial dimensions of the teacher model feature map. Indicates the flattening operation. This represents a multilayer perceptron. This represents the SiLU activation function layer. This indicates the L2 normalized layer.

5. The gradient-aware self-supervised monocular depth estimation method for dynamic scenes according to claim 1, characterized in that, The pre-trained pseudo-depth prediction model includes the Depth Anything V2 model, and both the encoder and the pose estimation network include the backbone network of the ResNet18 network; the decoder includes the Monodepth2 network, and the intermediate combination layer includes a convolutional layer with a kernel size of 1×1; the gradient-aware dense skip connection structure includes c rows of dense skip connection structures, where the output feature of the encoder in each row is the feature obtained after the input image passes through the encoder in that row, and the dense skip connection structure in the e-th row includes the encoder in the e-th row, ce intermediate combination layers in the e-th row, and the decoder in the e-th row, where e=1,2,…,c; in the dense skip connection structure in the e-th row, the input feature of the g-th intermediate combination layer in the e-th row is the output feature of the encoder in the e-th row. The output features are formed by concatenating the output features of the g-1th intermediate combination layer in the e-th row with the output features of the g-1th intermediate combination layer in the e+1th row, where g=2,…,ce; the input features of the first intermediate combination layer in the e-th row are formed by concatenating the output features of the encoder in the e-th row with the output features of the encoder in the e+1th row; the output features of the encoder in the c-th row are directly input into the decoder in the c-th row; when e is not equal to c, the input features of the decoder in the e-th row are formed by concatenating the output features of the ce-th intermediate combination layer in the e-th row with the output features of the decoder in the e+1th row; and the output features of the decoder in the first row are then input into another decoder to obtain the corresponding predicted depth map.

6. A gradient-aware self-supervised monocular depth estimation device for dynamic scenes, characterized in that, include: The model building module is configured to construct a monocular depth estimation model, a pre-trained pseudo-depth prediction model, and a pose estimation network. The pre-trained pseudo-depth prediction model and pose estimation network are then used to train the monocular depth estimation model, resulting in a trained monocular depth estimation model. The monocular depth estimation model includes an encoder, an intermediate combination layer, and a decoder, forming a gradient-aware dense skip connection structure. The loss functions used during the training of the monocular depth estimation model include mask-weighted photometric loss, geometric consistency loss, global depth ranking constraint loss, normal matching loss, edge relative normal loss, and multi-scale feature alignment loss. The process of constructing the mask-weighted photometric loss includes the following steps: acquiring a monocular video sequence and using two adjacent monocular images as the reference frame and the source frame; inputting the reference frame and the source frame into the monocular depth estimation model respectively to obtain the predicted depth map of the reference frame and the predicted depth map of the source frame; The reference frame and the source frame are input into the pose estimation network to obtain the corresponding relative camera pose; camera intrinsic parameters are obtained, and a photometric consistency loss is constructed based on the predicted depth map of the source frame, the relative camera pose, the camera intrinsic parameters, and the reference frame. The construction process of the photometric consistency loss is as follows: The predicted depth map of the source frame, the relative camera pose, and the camera intrinsic parameters are input into the reprojection and bilinear differentiable sampling algorithm to obtain the corresponding reconstructed image; The photometric consistency loss is calculated based on the reference frame and the reconstructed image, as shown in the following formula: ； in, This indicates that the reference frame and the reconstructed image are in the first... Luminous consistency loss of each effective pixel Indicates the weighting coefficient. This indicates that the reference frame and the reconstructed image are in the first... Structural similarity of each effective pixel and The reference frame and the reconstructed image respectively represent the first... The pixel value at each valid pixel location. Represents the L1 norm; Based on the relative camera pose, the predicted depth map of the reference frame is reprojected onto the source frame coordinate system and aligned with the predicted depth map of the source frame. Pixel-level normalized depth inconsistency is calculated, and a self-discovery mask is generated based on the pixel-level normalized depth inconsistency. The photometric consistency loss is weighted using the self-discovery mask to obtain the mask-weighted photometric loss. The calculation process of the mask-weighted photometric loss is as follows: The predicted depth value of each pixel in the predicted depth map of the reference frame is reprojected into 3D space and mapped to the source frame coordinate system through the relative camera pose to obtain the predicted depth map of the reference frame in the source frame coordinate system. Then, bilinear interpolation sampling is performed on the predicted depth map of the source frame using the predicted depth map of the reference frame in the source frame coordinate system to obtain the aligned depth map of the source frame, as shown in the following formula: ； ； in, This represents the relative camera pose from the reference frame to the source frame. The first reference frame is represented on the predicted depth map. The predicted depth value of each valid pixel. This indicates reprojection and mapping operations. The reference frame represents the first reference frame in the predicted depth map of the source frame coordinate system. The predicted depth value of each valid pixel. This indicates a bilinear interpolation sampling operation. The first element on the predicted depth map of the source frame. The predicted depth value of each valid pixel. The first element on the alignment depth map of the source frame Predicted depth values ​​for each valid pixel; The pixel-level normalized depth inconsistency between the predicted depth map of the source frame and the predicted depth map of the reference frame is calculated using the following formula: ； in, This indicates that the predicted depth map of the source frame and the predicted depth map of the reference frame are compared at the 1st... Pixel-level normalized depth inconsistency of each valid pixel; The self-discovery mask is calculated based on the pixel-level normalized depth inconsistency, as shown in the following formula: ； in, Indicates the first Self-discovery mask for each valid pixel; The photometric consistency loss is weighted using the self-discovery mask and the mean value is calculated to obtain the mask-weighted photometric loss, as shown in the following formula: ； in, This represents the set of valid pixels that were successfully aligned during the projection and interpolation process. Represents the set of valid pixels The total number of valid pixels in the middle. This represents the mask-weighted luminosity loss; The geometric consistency loss is calculated using the following formula: ； in, This represents the geometric consistency loss; The prediction module is configured to acquire a single-frame image to be estimated and input it into the trained monocular depth estimation model. After processing by the gradient-aware dense skip connection structure, the corresponding predicted depth map is obtained.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images