Monocular self-supervision depth estimation method based on credible area depth controllable generation

Abstract

The invention relates to the technical field of monocular depth estimation, and discloses a monocular self-supervision depth estimation method based on credible region depth controllable generation. Firstly, a monocular image is collected and input into a depth estimation module for multi-scale feature extraction, an initial depth map of a whole scene is generated, and relative camera attitude transformation between adjacent frames is estimated through an attitude estimation module; identifying a credible area and an uncredible area according to the obtained initial depth map and relative camera attitude transformation; depth estimation of a dynamic region is guided through a pseudo depth and contrast learning method; then, constructing a multi-task joint loss function, and optimizing depth estimation of the untrusted region by utilizing depth information of the trusted region through a comparison loss mechanism; and iteratively optimizing network parameters of the depth estimation module and the attitude estimation module, and outputting a final optimized depth map. According to the method, the depth estimation precision and robustness of the model in a complex dynamic environment are improved through a credible region depth controllable generation strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of monocular depth estimation, and in particular to a monocular self-supervised depth estimation method based on controllable generation of credible region depth. Background Art

[0002] Monocular depth estimation, as a fundamental and challenging task in the field of computer vision, aims to recover the three-dimensional depth information of a scene from a single two-dimensional image. In recent years, with the rapid development of deep learning technology, the accuracy and robustness of monocular depth estimation have been significantly improved. However, most existing methods assume that the scene is static, that is, the objects in the scene remain stationary during the capture process. This assumption does not hold in many practical application scenarios, such as autonomous driving, robot navigation, and augmented reality, which often contain a large number of dynamic objects such as pedestrians, vehicles, and animals. The presence of dynamic objects can seriously affect the accuracy of depth estimation because they violate the static scene assumption, causing the self-supervised methods based on photometric consistency to fail.

[0003] Specifically, in self-supervised depth estimation frameworks, the photometric consistency between adjacent frames is typically used as a supervisory signal to train the network. When dynamic objects are present in the scene, the motion of the objects causes the photometric values ​​of corresponding pixels in adjacent frames to become inconsistent, which can cause the network to learn incorrect depth information. Furthermore, dynamic objects often have complex shapes and motion patterns, further complicating depth estimation. Therefore, effectively handling dynamic objects and improving the robustness and accuracy of monocular depth estimation in dynamic scenes has become a critical issue that needs to be addressed.

[0004] Traditional methods for processing dynamic objects mainly rely on motion segmentation or optical flow estimation. Motion segmentation methods attempt to segment the image into a static background and dynamic objects, and then estimate their depth separately. However, accurate motion segmentation itself is a challenging problem, especially in complex scenes. Optical flow estimation methods attempt to estimate the motion vector of each pixel and then use this motion information to compensate for the influence of dynamic objects. However, the accuracy of optical flow estimation is also easily affected by factors such as lighting changes, occlusion, and texture loss.

[0005] In recent years, some deep learning-based methods have attempted to combine motion segmentation or optical flow estimation with depth estimation to improve performance in dynamic scenes. For example, some methods use a pre-trained optical flow network to estimate pixel motion and then provide this motion information as an additional input to the depth estimation network. Other methods jointly train the depth estimation network and the motion segmentation network to achieve end-to-end depth estimation. However, these methods still have some limitations. Because they are very sensitive to the accuracy of motion segmentation or optical flow estimation, any errors may propagate to the depth estimation network, resulting in inaccurate final results. In addition, these methods usually assume that dynamic objects are rigid, that is, the relative positions within the object remain unchanged. However, in the real world, many dynamic objects are non-rigid, such as pedestrians and animals, and their shapes change with movement. Therefore, how to effectively handle non-rigid dynamic objects remains an open problem. Summary of the Invention

[0006] To address the above issues, the present invention aims to provide a monocular self-supervised depth estimation method based on controllable depth generation of trusted regions. This strategy improves the accuracy and robustness of depth estimation in complex dynamic environments. Multi-frame depth information is used to guide and optimize depth estimation in dynamic regions, effectively constraining the depth values ​​in dynamic regions and mitigating the negative impact of dynamic object motion on the self-supervised learning process. The technical solution is as follows:

[0007] Step 1: Collect a monocular image and input it into a lightweight self-supervised depth estimation module. The depth estimation module extracts multi-scale features from the input monocular image to generate an initial depth map of the entire scene.

[0008] Step 2: Take two consecutive frames of images as input to the pose estimation module and estimate the relative camera pose transformation between adjacent frames;

[0009] Step 3: Based on the obtained initial depth map and the relative camera pose transformation, the dynamic area in the initial depth map that may contain dynamic objects is identified as an untrustworthy area by calculating the difference coefficient based on the credible region detection module, while the static background area is identified as a credible area. The depth estimation of the dynamic area is guided by pseudo-depth and contrastive learning methods.

[0010] Step 4: Construct a multi-task joint loss function, including photometric consistency loss, depth consistency loss, dynamic region contrast loss, and smoothness loss. Through the contrast loss mechanism, the depth information of the credible area is used to optimize the depth estimation of the untrustworthy area.

[0011] Step 5: Iteratively optimize the network parameters of the depth estimation module and the pose estimation module, and output the final optimized depth map.

[0012] The beneficial effects of the present invention are:

[0013] 1) This paper designs a lightweight, self-supervised depth estimation network based on the fusion of a convolutional neural network and a Transformer architecture to initially generate a depth map of the entire scene. Leveraging the powerful local feature extraction capabilities of CNNs and the global context modeling capabilities of Transformers, it can effectively learn the geometric structure of the scene. While reducing the number of model parameters by approximately 80%, the performance surpasses the current state-of-the-art.

[0014] 2) This paper proposes a credible region detection method based on motion consistency, which identifies areas in the depth map that may contain dynamic objects as untrustworthy areas, while identifying static background areas as credible areas. On this basis, the depth values ​​of untrustworthy areas are optimized using multi-frame depth information as a guidance signal.

[0015] 3) This paper proposes an improved loss function optimization strategy, which improves the accuracy of network depth estimation in dynamic areas by introducing a contrast loss mechanism;

[0016] 4) The present invention does not require additional network modules for motion segmentation or optical flow estimation, so it has lower computational cost and higher efficiency; it can handle non-rigid dynamic objects because the method of the present invention does not rely on any prior assumptions about the shape or motion of the object and is end-to-end, and can be easily integrated into various computer vision applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A monocular self-supervised depth estimation framework for dynamic objects.

[0018] Figure 2 It is the depth estimation network structure.

[0019] Figure 3 The network structure for pose estimation.

[0020] Figure 4 Visualization of depth estimation results on the NYU Depth Dataset V2 dataset.

[0021] Figure 5 Visualization of depth estimation results on the TUMRGB-D dataset.

[0022] Figure 6 The results of credible area detection on the TUMRGB-D dataset are as follows: the first column is the original image, the second column is the predicted depth map, and the third column is the mask of credible area detection. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] To address the challenges of monocular self-supervised depth estimation in dynamic scenes, this paper proposes a novel monocular self-supervised depth estimation framework for dynamic objects. This framework aims to improve the accuracy and robustness of depth estimation in complex dynamic environments by leveraging a controllable depth generation strategy for trusted regions. The core idea of ​​this framework is to utilize multi-frame depth information to guide the optimization of depth estimation in dynamic regions, thereby effectively constraining the depth values ​​in dynamic regions and mitigating the negative impact of dynamic object motion on the self-supervised learning process.

[0025] like Figure 1 As shown in Figure 1, the overall framework of the proposed method consists of four core modules: depth estimation module, pose estimation module, trusted region detection module, and loss function module. These four modules work together to form an end-to-end self-supervised learning process.

[0026] First, the depth estimation module is responsible for preliminarily predicting the depth map of the entire scene from the input monocular image. This module uses a network architecture based on the fusion of convolutional neural networks and Transformers, fully leveraging the powerful local feature extraction capabilities of CNNs and the global context modeling capabilities of Transformers to better understand the geometric structure and semantic information of the scene.

[0027] Secondly, the pose estimation module is responsible for estimating the camera motion between adjacent frames, providing the necessary geometric constraints for self-supervised learning. This module adopts an encoder-decoder structure, takes two consecutive frames as input, and outputs the relative camera pose transformation matrix between the two frames.

[0028] The trusted region detection module then uses motion consistency to mark areas in the preliminary depth map that may contain dynamic objects as untrustworthy, while marking static background areas as trustworthy. This step is crucial because it focuses the subsequent depth optimization process on static background areas, avoiding interference from dynamic objects.

[0029] Finally, the loss function module guides the training process of the entire network through a carefully designed loss function.

[0030] In particular, the present invention introduces a contrastive loss mechanism, which uses the depth information of the credible area as a supervisory signal to optimize the depth of the untrustworthy area, thereby improving the accuracy of the network's depth estimation in dynamic areas.

[0031] During the training phase, the present invention proposes a depth estimation method based on a continuous image sequence. For each frame of the image in the input sequence, the depth estimation module first predicts its corresponding depth map. At the same time, the posture estimation module uses the relationship between adjacent frames to estimate the transformation parameters of the camera posture. In addition, the trusted area detection module divides the depth map into two parts: a trusted area and an untrusted area based on the generated depth map and camera posture information. The loss function module then combines the predicted depth map, camera posture and trusted area mask to calculate the training loss and perform backpropagation to update the network parameters. In particular, the depth information extracted from different frames by the deep network is used as a supervisory signal to guide the optimization of the depth estimation of the untrusted area. In this way, the method can effectively utilize the depth information between multiple frames, constrain the depth estimation of dynamic areas, and thus significantly improve the accuracy of the overall depth estimation.

[0032] During the testing phase, the method of the present invention requires only a single frame as input to complete depth estimation. The depth estimation module directly generates a depth map corresponding to that frame. Because depth estimation is optimized for dynamic areas during the training phase, the testing phase does not require multiple frames of input. Furthermore, the method of the present invention can generate accurate depth maps even when dynamic objects are present in the input image.

[0033] 1. Depth Estimation Module

[0034] The depth estimation module is the core component of this framework, responsible for predicting a dense depth map from the input monocular image. In order to better capture local details and global contextual information in the image, the present invention proposes a depth estimation network architecture that integrates convolutional neural networks and Transformer. The network is based on ResNet as the backbone network and introduces the Multi-Query Attention (MQA) mechanism to enhance the network's ability to model long-distance dependencies. At the same time, in order to improve the detail recovery ability of the depth map, the Pixel Shuffle operation is used in the upsampling stage to obtain smoother gradients and finer depth edges.

[0035] like Figure 2 As shown in Figure 1, the depth estimation network mainly consists of two parts: the encoder and the decoder. The encoder is responsible for extracting multi-scale feature representations from the input image, while the decoder is responsible for gradually upsampling these features and fusing them into the final depth map.

[0036] In the encoder, we first downsample the input image using a 3×3 convolutional layer and a max pooling layer to generate a feature map. We then stack multiple residual blocks (ResNet blocks) and multi-query attention modules (MQABlocks) to extract deeper features. Each residual block contains two 3×3 convolutional layers and a skip connection to enhance the network's nonlinear representation and gradient propagation efficiency. Each MQA module includes a multi-query attention layer and a feedforward network to capture long-range dependencies in the feature map.

[0037] It is worth noting that in order to successfully apply Transformer to the image field and reduce the number of parameters to speed up training and inference, the present invention uses convolution operations as mappings before calculating attention values. This design takes advantage of the local receptive field and weight sharing characteristics of convolution operations, enabling the model to better capture the local structural information of the image and reduce the number of parameters of the model. At the same time, the introduction of the multi-query attention mechanism enables the model to pay attention to the long-distance dependencies between different regions in the image, thereby improving the model's ability to model global context information. Specifically, given an input feature map Where C represents the number of channels, H and W represent the height and width of the feature map respectively. The multi-query attention layer first maps it into a query vector Q, a key vector K, and a value vector V through three different convolutional layers:

[0038] Q = Conv Q (X), K = Conv K (X), V = Conv V (X) (1)

[0039] Among them, Conv Q 、Conv K and Conv V Represent different convolutional layers. To reduce computational complexity, we use a multi-query attention mechanism, where all attention heads share the same key vector and value vector and only use different query vectors. This reduces the dimensions of the key vector and value vector to Where N represents the number of attention heads. Then calculate the dot product attention weight between the query vector and the key vector:

[0040]

[0041] in, Represents the dimension of each attention head. Finally, the attention weight is multiplied by the value vector to obtain the weighted value vector, and then mapped back to the original dimension through a convolution layer:

[0042] MQA(X)=Conv(AV) (3)

[0043] In the decoder part, multiple residual blocks and upsampling modules are stacked to gradually restore the details of the depth map. In order to obtain smoother gradients and finer depth edges, the PixelShuffle operation is used for upsampling. PixelShuffle is a non-learnable upsampling method that achieves upsampling by rearranging the channel dimensions of the feature map, avoiding the checkerboard effect caused by traditional bilinear interpolation or deconvolution operations. Specifically, given an input feature map X∈R C×H×W , PixelShuffle rearranges it into an output feature map Where r represents the upsampling factor. This process can be expressed as:

[0044]

[0045] in, i,j∈[0,r-1]. Y c,rh+i,rw+j Represents the value of the output feature map Y at the rh+i row, rw+j column, and c channel; Represents the value of the input feature map X at the hth row, wth column, and r^2c+ri+jth channel; the PixelShuffle operation rearranges the values ​​of multiple channels (r^2c+ri+j) of the input feature map X at a certain spatial position (h,w) to a larger spatial area of ​​a channel c on the output feature map Y, consisting of r×r pixels, specifically at (rh+i,rw+j).

[0046] Through the PixelShuffle operation, smoother gradients can be obtained, which is beneficial to network training and detail recovery of the depth map.

[0047] Finally, we use the depth output module to map the final feature map into a single-channel disparity map. The depth output module first further refines the features through two convolutional layers and a SiLU activation function, then uses a Sigmoid activation function to limit the output between (0, 1) and uses a linear transformation to map it to a predefined disparity range. This process can be expressed as:

[0048] Disp=αSigmoid(Conv(SiLU(Conv(Y))))+β (5)

[0049] Among them, α and β are non-learnable parameters, which are set to 10.0 and 0.01 respectively. The final depth map D is calculated by the inverse of the disparity map Disp:

[0050]

[0051] Since monocular depth estimation has scale uncertainty, the depth map we obtain is a scale-free depth representation.

[0052] 2. Posture Estimation Module

[0053] The pose estimation module plays a crucial role in this framework. Its primary task is to estimate the camera motion between adjacent frames, providing the necessary geometric constraints for self-supervised learning. To achieve this goal, we design a pose estimation network based on an encoder-decoder architecture. This network takes two consecutive frames as input and outputs the relative camera pose transformation matrix between the two frames.

[0054] like Figure 3 As shown, the encoder portion of the pose estimation network shares the same network architecture as the depth estimation module, but we have made some key improvements to adapt to the characteristics of the pose estimation task. First, in addition to the original input image, we also add two downsampled versions of the input image to the encoder input. Specifically, the original monocular image is downsampled to 1 / 2 and 1 / 4 of its original size through two average pooling operations. These two downsampled versions, together with the original monocular image, form the input to the pose estimation module encoder. This multi-scale input design accelerates network learning and improves the robustness of pose estimation. This is because images of different scales provide different levels of contextual information, helping the network better understand the scene geometry and motion patterns. Smaller-scale images provide a wider field of view, helping the network capture larger motions, while larger-scale images provide finer details, enabling the network to estimate more precise motions.

[0055] Within the encoder, the feature extraction process is the same as that of the depth estimation module, which uses stacked residual blocks and multi-query attention modules to extract multi-scale feature representations. These feature representations capture local details and global contextual information in the input image, providing rich information for subsequent pose estimation.

[0056] The decoder portion of the pose estimation network is relatively simple, consisting of a global average pooling layer and a pose output module. The global average pooling layer averages the feature maps output by the encoder across the spatial dimensions to produce a fixed-length feature vector. This feature vector can be viewed as a global motion representation of the input image pair. The pose output module maps this feature vector to the final camera pose transformation.

[0057] To better train the pose estimation network, the present invention features a special design for the pose output module. Specifically, the camera pose transformation is decoupled into two components: the translation vector and the rotation vector, which are predicted separately. This decoupled design makes it easier for the network to learn, as translation and rotation are two different types of motion, with different ranges of variation and impacts on the scene. By predicting them separately, the network can learn their characteristics, thereby improving the accuracy of pose estimation.

[0058] The posture output module consists of two independent multilayer perceptrons, which are used to predict the translation vector and the rotation vector respectively. Each MLP (multilayer perceptron) contains two fully connected layers and a SiLU activation function. The input feature vector is first predicted by an MLP to obtain a 3D translation vector t = (t x ,t y ,t z ), which represents the translation of the camera on the three coordinate axes. Then, the input feature vector is predicted by another MLP to obtain a 3D rotation vector r = (r x ,r y ,r z ), which represents the rotation angle of the camera around the three coordinate axes. These two MLPs can be expressed as:

[0059] t=MLP t (f) = SiLU(W t2 ·SiLU(W t1 f+b t1 )+b t2 ) (7)

[0060] r = MLP r (f) = SiLU(W r2 ·SiLU(W r1 f+b r1 )+b r2 ) (8)

[0061] Where f represents the input feature vector, W t1 ,W t2 ,W r1 ,W r2 Represents the fully connected layer weight matrix of the two MLPs, b t1 ,b t2 ,b r1 ,b r2 represent the bias vectors respectively.

[0062] Finally, the predicted translation vector t and rotation vector r are combined into a 6-dimensional pose vector p = (t, r), which represents the relative camera pose transformation between adjacent frames. This pose vector can be used to transform pixel coordinates in one view to another, thus achieving self-supervised learning.

[0063] 3. Trusted Area Detection

[0064] In dynamic scenes, due to the presence of object motion, the photometric consistency assumption and the depth consistency assumption fail in dynamic areas, which seriously affects the training effect of self-supervised depth estimation. To address this problem, it is necessary to identify dynamic areas in the image and perform special processing on them. This paper adopts a credible region detection method based on the difference coefficient to distinguish between dynamic and static areas in the image, and further introduces pseudo-depth and contrastive learning methods to guide depth estimation in dynamic areas.

[0065] First, the difference coefficient (diff) is used to preliminarily determine the dynamic area. This coefficient is used to compare the depth map D at time t. t Projection to I t+1 The depth map obtained from the viewing angle and the depth map D at time t+1 t+1 The specific calculation formula is as follows:

[0066]

[0067] in, is by adding D t The camera pose transformation is projected to the viewing angle at time t+1. In static areas, since both depth estimation and camera pose estimation are relatively accurate, and D t+1 Should be very close, so the diff value is small; in the dynamic area, due to the inconsistency of light intensity and depth caused by the movement of the object, and D t+1 The difference between them is larger, so the diff value is larger.

[0068] According to the size of the difference coefficient diff, we can preliminarily determine the dynamic area. We assume that 10% of the area in the image is the dynamic area, and the dynamic area dynamic is defined as:

[0069] dynamic=diff top10％ (10)

[0070] That is, the regions with the top 10% diff values ​​are considered dynamic regions.

[0071] To guide self-supervised depth estimation, a pseudo depth F is introduced. t (pt ) and proposed a method based on contrastive learning. Pseudo-depth can be obtained through a deep network. In dynamic areas, the difference in pseudo-depth values ​​is used to guide network learning. The calculation method for defining the contrast value is as follows:

[0072]

[0073] Among them, p 0 and p 1 Represents two pixels in a picture, and τ is a preset threshold used to control the sensitivity of the contrast value. t (p 0 ) represents pixel p 0 The pseudo-depth value can be obtained by the exponential moving average of the depth estimation module or other methods that can generate depth. The comparison value of the pseudo-depth can be used to guide the network so that the network prediction value has a similar size relationship in the dynamic area.

[0074] For any two pixels p 0 and p 1 , their loss function L p (p 0 ,p 1 ) is defined as follows:

[0075]

[0076] The calculation logic of this loss function is as follows: when contrast≠0, it indicates that the pseudo-depth difference between the two pixels is large. We want the network's predicted depth difference to be consistent with the pseudo-depth difference, so we use the cross-entropy loss. When contrast=0, it indicates that the pseudo-depth difference between the two pixels is small. Since we want the network's predicted depth difference to be as small as possible, we use the mean squared error loss. This design can effectively utilize pseudo-depth information to guide depth estimation in dynamic areas.

[0077] Finally, the dynamic area can be sampled to obtain the dynamic area contrast loss function:

[0078]

[0079] Among them, w ij is the weight calculated based on the diff value, which is used to balance the importance of different dynamic areas. Specifically, where diff ij Represents a pixel pair (p i ,p j This weighted sampling strategy allows the network to pay more attention to areas with larger diff values, thereby improving the accuracy of depth estimation.

[0080] 4. Loss Function

[0081] The loss function is the key to guiding network training. A good loss function can help the network converge to the optimal solution faster and better. The present invention adopts a multi-task joint training method, so the total loss function is composed of a weighted combination of multiple sub-loss functions to constrain the learning of different aspects of the network. Specifically, the loss function of the present invention consists of four parts: photometric consistency loss L photo , depth consistency loss L depth , dynamic area contrast loss L dynamic And the smoothing loss L smooth The total loss function can be expressed as:

[0082] L=λ photo L photo +λ depth L depth +λ dynamic L dynamic +λ smooth L smooth (14)

[0083] Among them, λ photo ,λ depth ,λ dynamic and λ smooth are the weight coefficients of each loss function, which are used to balance the importance of different loss functions. In the experiment, we set λ photo =1.0,λ depth = 0.1 and λ smooth =0.001. And λ dynamic The value of is dynamically adjusted according to the value of contrast in the dynamic area contrast loss, see formula (21) for details.

[0084] Photometric consistency loss is one of the most commonly used loss functions in self-supervised depth estimation, which is based on the assumption of photometric consistency of static regions between adjacent frames. Specifically, for the image I at time t t A pixel p in t , the depth D predicted by the depth estimation module can be used t (p t ) and the camera pose transformation T predicted by the pose estimation module t→t′ , project it to the adjacent frame I t′ The corresponding position p in t′ :

[0085] p t′ ~KT t→t′ D t (p t )K -1 p t(15)

[0086] Where K represents the camera intrinsic parameter matrix, and ~ represents the equality of homogeneous coordinates. If the depth estimation and pose estimation are both accurate enough, and the pixel p t Located in the static region, then I t (p t ) and I t′ (p t′ ) should be very close. Therefore, we can minimize I t (p t ) and I t′ (p t′ ) to optimize the depth estimation network and the pose estimation network.

[0087] In order to improve the robustness of the photometric consistency loss, the grayscale difference and structural similarity (Structural Similarity Index Measure, SSIM) of the image are considered at the same time. Specifically, the photometric consistency loss can be expressed as:

[0088]

[0089] Where ω is a weight coefficient used to balance the importance of grayscale difference and SSIM loss, which we set to 0.85. t′ represents the frames adjacent to t, including t-1 and t+1.

[0090] In addition, in order to avoid the false supervision signal caused by occlusion problems, a strategy of minimizing the reprojection error is adopted. Specifically, for each pixel, only the minimum value of its reprojection error in all adjacent frames is calculated as the final photometric consistency loss. This strategy can be expressed as:

[0091]

[0092] This minimization strategy can effectively alleviate the impact of occlusion problems on the training process, because in the occluded area, there is at least one adjacent frame that can provide a more accurate photometric consistency supervision signal.

[0093] The depth consistency loss is used to constrain the consistency of depth estimation between adjacent frames. Specifically, for the image I at time t t A pixel p in t , which can be projected to the adjacent frame I t′ The corresponding position p in t′ , and use I t′ The depth map D t′ Get p t′ Depth D t′ (p t′ ). Then, Dt′ (p t′ ) projected back to I again t , and get p t Projection depth If the depth estimation and pose estimation are both accurate enough, then D t (p t )and should be very close. Therefore, we can minimize D t (p t )and The difference between them is used to optimize the depth estimation network and the pose estimation network. The depth consistency loss can be expressed as:

[0094]

[0095] in, It can be calculated by the following formula:

[0096]

[0097] Among them, proj represents the projection operation, D t′ (p t′ ) According to the camera posture transformation T t→t′ And the camera intrinsic parameter matrix K is projected onto I t p in t Location.

[0098] Similar to the photometric consistency loss, a minimization strategy is also used to alleviate the impact of the occlusion problem. Specifically, for each pixel, only the minimum value of its depth consistency loss in all adjacent frames is calculated as the final depth consistency loss:

[0099]

[0100] Dynamic area contrast loss L dynamic The calculation of has been introduced in detail in the section on trusted region detection. The core idea is to use pseudo-depth information to guide depth estimation in dynamic regions. In contrast to the original calculation method, different calculation strategies and weights are used here according to different contrast values. In practice, the strategies and weights need to be adjusted according to different contrast values, which is the following formula: and That is the weight.

[0101] For any two points p 0 and p 1 , their loss function L p (p 0 ,p 1 ) is defined as follows:

[0102]

[0103] Among them, when contrast≠0, it means that the difference in pseudo depth between the two pixels is large and the relationship is clear. It is hoped that the depth difference predicted by the network is consistent with the difference in pseudo depth. Therefore, the cross entropy loss is adopted and the weight is set. When contrast = 0, it means that the pseudo depth difference between the two pixels is small, and the true depth relationship between them cannot be determined. However, we still hope that the depth difference predicted by the network is as small as possible. Therefore, the mean square error loss is used and a smaller weight is set. This design can more finely utilize pseudo-depth information to guide depth estimation in dynamic areas, taking into account both clear size relationships and constraints in uncertain situations.

[0104] Finally, the dynamic area can be sampled to obtain the dynamic area contrast loss function:

[0105]

[0106] Among them, w ij is the weight calculated based on the diff value, which is used to balance the importance of different dynamic areas. Specifically, where diff ij Represents a pixel pair (p i ,p j This weighted sampling strategy allows the network to pay more attention to areas with larger diff values, thereby improving the accuracy of depth estimation.

[0107] Smoothing loss L smooth Used to constrain the local smoothness of the depth map. Since the surfaces of objects in natural scenes are usually smooth, we expect the predicted depth map to have similar smoothness. The smoothness loss achieves this goal by penalizing the depth differences between adjacent pixels in the depth map. Specifically, the predicted depth map and the input image are first Gaussian blurred, and then their gradients in the horizontal and vertical directions are calculated. The calculation formula of the smoothness loss is as follows:

[0108]

[0109] Where N is the total number of pixels, D i,j Represents the depth value of pixel (i, j) in the depth map, I i,j represents the pixel value of pixel (i, j) in the input image, and Represent the gradient operators in the horizontal and vertical directions respectively. and It is an edge weight term used to reduce the penalty of depth gradient at the edge of the image, because the edge of the image usually corresponds to a depth discontinuous area.

[0110] In summary, the loss function of this embodiment achieves effective constraints on the depth estimation network and the posture estimation network by combining photometric consistency loss, depth consistency loss, dynamic area contrast loss and smoothness loss. Among them, photometric consistency loss and depth consistency loss utilize the geometric relationship between adjacent frames and provide the main supervisory signal; dynamic area contrast loss adopts different calculation strategies and weights according to different contrast values, and makes more refined use of pseudo-depth information, thereby improving the accuracy of depth estimation in dynamic areas; smoothness loss ensures the local smoothness of the depth map, making the prediction results more consistent with the real scene. This multi-task joint training strategy enables the method of the present invention to obtain more accurate and robust depth estimation results in complex dynamic scenes.

[0111] 5. Dataset and evaluation metrics

[0112] (1) Dataset

[0113] The NYU Depth Dataset V2 was created by Silberman et al. from the Department of Computer Science at New York University. It contains video sequences of indoor scenes captured by Microsoft Kinect and provides pixel-level depth information. The dataset consists of 1,449 RGB images and corresponding depth maps, 795 of which are used for training and 654 for testing. These images cover a variety of indoor scenes, such as bedrooms, living rooms, and offices, and include a variety of common furniture and objects. All images have a resolution of 640×480 pixels.

[0114] (2) TUM dataset

[0115] The TUMRGB-D dataset was created by Sturm et al. from the Computer Vision Group at the Technical University of Munich. It contains RGB-D sequences of indoor scenes with precise camera pose information. The dataset includes multiple sequences covering different scenes, textures, and lighting conditions. Each sequence contains synchronized RGB images, depth maps, and camera trajectory data. The size of the dataset depends on the specific sequence selected; commonly used sequences, such as the fr1 series, contain hundreds to thousands of frames. The image resolution is typically 640 × 480 pixels.

[0116] In contrast, NYUDepthDatasetV2 focuses more on static environments with large camera motion, making it suitable for tasks such as SLAM and visual odometry; while the TUMRGB-D dataset focuses more on dynamic environments with small camera motion. Combining the two datasets allows for a more comprehensive evaluation of the performance of depth estimation models.

[0117] (3) Evaluation indicators

[0118] In depth estimation tasks, the performance of the evaluation model is usually evaluated using multiple metrics to measure the difference between the predicted depth map and the true depth map. The following are commonly used evaluation metrics:

[0119] 1) Absolute Relative Error (AbsRel): The average absolute error between the predicted depth and the true depth, divided by the average true depth. The smaller the value of this indicator, the more accurate the model's prediction results.

[0120] 2) Root Mean Square Error (RMS): The square root of the average of the squares of the differences between the predicted depth and the true depth. The smaller the value of this indicator, the more accurate the model's prediction results.

[0121] 3) δ1, δ2, δ3 (Delta1, Delta2, Delta3): The proportion of pixels for which the relative error between the predicted depth and the true depth is less than a threshold (usually 1.25). Larger values ​​of δ1, δ2, and δ3 indicate a greater number of pixels within the predicted error range, indicating better model performance.

[0122] 6. Experimental setup and results analysis

[0123] Experimental settings and result analysis were carried out, covering software environment configuration, comparison of experimental results on NYU and TUM datasets, and ablation experiments. The software and hardware configurations of the experiment were listed to ensure the reproducibility of the experiment. On the NYU dataset, quantitative indicators and qualitative visualization results verified that the proposed method surpassed the existing self-supervised methods, especially in the details and edges of the depth map, showing good generalization and robustness. On the TUM dataset, in addition to the overall performance comparison, dynamic and static regions were evaluated separately. The results showed that the method of the present invention has significant advantages in depth estimation in dynamic regions, proving the effectiveness of credible region detection and contrast loss function, and analyzing the visualization results of credible region detection. Ablation experiments further verified the contribution of contrast loss function, model structure and training strategy to performance improvement. Overall, the experimental results fully demonstrated the effectiveness and superiority of the method proposed in this invention.

[0124] (1) Software conditions and environment configuration

[0125] The software conditions and environment configuration used in the experiment of this embodiment are as follows:

[0126] Interpreter: Python 3.10.15; Dependent libraries: numpy 1.26.4, torch 2.1.2, opencv-python 4.10.0.84; Operating platform: Ubuntu 20.04 LTS; Training platform: PyTorch; Training rounds: 60 epochs; Optimizer: Adam; Data processing batch size: 24; Learning rate setting: Cosine annealing is used after two epochs of WarmUp, and the maximum learning rate is 0.001.

[0127] (2) Analysis of experimental results

[0128] 1) Experimental results on the NYU dataset

[0129] Table 1 shows the experimental results on the NYU Depth Dataset V2 dataset, where ↓ indicates that the smaller the value, the better, and ↑ indicates that the larger the value, the better. The best results of the supervised and self-supervised models are displayed in bold.

[0130] Table 1 Experimental results on NYU Depth Dataset V2

[0131]

[0132] On the NYU Depth Dataset V2 dataset, the performance of supervised and self-supervised depth estimation models is compared. The experimental results are shown in Table 1. As can be seen from the table, the self-supervised depth estimation model proposed in the present invention has achieved better performance in absolute relative error, root mean square error, and δ1 and δ2 indicators than the best self-supervised model. Specifically, the model of the present invention has improved by 4.8%, 3.3%, 1.4%, and 0.3% in absolute relative error, root mean square error, and δ1 and δ2 indicators respectively compared with the best self-supervised model. It has reached the same level as the best self-supervised model in terms of δ3 indicator. This shows that the model of the present invention has achieved better performance in the depth estimation task, and has better generalization ability and robustness. In addition to quantitative indicators, the depth estimation results of the model are visualized and analyzed to show the qualitative comparison results with other self-supervised models, such as Figure 4 As shown. Figure 4 As can be seen from the figure, the model of the present invention achieves better performance in the depth estimation task. Compared with other self-supervised models, the model of the present invention more accurately predicts the details and edges of the depth map, without any obvious depth estimation errors.

[0133] 2) Experimental results on the TUM dataset

[0134] The performance of self-supervised depth estimation models was compared on the TUMRGB-D dataset. The experimental results are shown in Table 2, where ↓ indicates that smaller values ​​are better, and ↑ indicates that larger values ​​are better. As can be seen from the table, the self-supervised depth estimation model achieved better performance than the supervised model in terms of absolute relative error, root mean square error, and δ1, δ2, and δ3. Specifically, the model of the present invention improved the absolute relative error, root mean square error, and δ1, δ2, and δ3 indicators by 27.6%, 25.2%, 4.5%, 6.9%, and 5.3%, respectively, compared to the best self-supervised model.

[0135] Table 2 Experimental results of self-supervised depth estimation model on TUMRGB-D dataset

[0136]

[0137] Similarly, the depth estimation results of the model are visualized and analyzed to show the qualitative comparison results with other self-supervised models, such as Figure 5 As shown in the figure, it can also be seen that the model of the present invention is more accurate in predicting the details and edges of the depth map.

[0138] In addition, because the TUM RGB-D dataset includes dynamic scenes, this embodiment analyzes the depth estimation results for dynamic and static regions separately, considering the area around the person as dynamic and the rest of the area as static. The experimental results are shown in Table 3, where ↓ indicates a smaller value is better and ↑ indicates a larger value is better. As can be seen from the table, the model of the present invention achieves better performance in depth estimation in dynamic regions, but lags behind the best self-supervised depth estimation model in static regions. Specifically, the model of the present invention improves the absolute relative error and δ1 metric by 29.5% and 14.0% respectively compared to the best self-supervised model in dynamic regions; while it lags behind the best self-supervised model in absolute relative error and δ1 metric by 16.7% and 11.4% respectively in static regions. This demonstrates that the contrastive loss function proposed in this embodiment can effectively guide depth estimation in dynamic regions, improving the accuracy of depth estimation in dynamic scenes. In the last row of the table, this embodiment also conducts experiments on a model that does not use the contrastive loss for dynamic regions. The results show that the contrastive loss for dynamic regions plays a significant role in improving depth estimation performance.

[0139] Table 3 Experimental results of the self-supervised depth estimation model on the dynamic and static regions of the TUMRGB-D dataset

[0140]

[0141] Figure 6The qualitative experimental results of credible region detection on the TUMRGB-D dataset are shown. The first column is the original RGB image, the second column is the predicted depth map, and the third column is the mask obtained by credible region detection. As can be seen from the figure, the method of the present invention can effectively detect dynamic areas in the image, such as moving human bodies. This is the same as our assumption that the moving area is the human body. At the same time, we can also find that in addition to the moving human body, the mask also identifies some edge areas as dynamic areas, which may be due to the occlusion problem caused by the front and back of the movement. This shows that the method of the present invention has further room for improvement in dynamic region detection, and the accuracy of detection can be improved by introducing more prior knowledge.

[0142] (3) Ablation experiment

[0143] In order to comprehensively evaluate the effectiveness of the contrast loss function and model structure introduced in the present invention, an ablation experiment was conducted on the TUM dataset. Specifically, the performance of the depth estimation model with or without the adaptive loss function and model structure was compared. The experimental results are shown in Table 4, where √ indicates use, × indicates not use, ↓ indicates the smaller the value, the better, and ↑ indicates the larger the value, the better. It can be seen from the table that the introduction of the contrast loss function and the model structure can improve the performance of the depth estimation model. Specifically, the introduction of the contrast loss function improved by 56.9% compared to not introducing the contrast loss function; the introduction of the model structure improved by 5.1% compared to not introducing the model structure. This shows that the contrast loss function and model structure proposed in the present invention have important guiding significance for the depth estimation task and can improve the performance of the depth estimation model.

[0144] Table 4 Ablation experiment results on the TUM dataset

[0145]

[0146] In summary, the present invention proposes a monocular self-supervised depth estimation framework for dynamic objects. By utilizing multi-frame depth information to guide the optimization of depth estimation in dynamic areas, it effectively solves the depth estimation problem in dynamic scenes. The framework integrates depth estimation, pose estimation, trusted region detection and loss function modules, adopts a fusion CNN and Transformer architecture to capture image details and global information, and combines contrastive learning methods to optimize depth estimation in dynamic areas. Experimental results show that this method significantly outperforms existing methods on the NYU and TUM datasets, especially in depth estimation in dynamic areas. Ablation experiments verify the effectiveness of the contrastive loss function and model structure, and the visualization results of trusted region detection further confirm its reliability.

Claims

1. A monocular self-supervised depth estimation method based on controllable generation of credible region depth, characterized by: The following steps are involved: Step 1: Collect a monocular image and input it into a lightweight self-supervised depth estimation module. The depth estimation module extracts multi-scale features from the input monocular image to generate an initial depth map of the entire scene. Step 2: Take two consecutive frames of images as input to the pose estimation module and estimate the relative camera pose transformation between adjacent frames; Step 3: Based on the obtained initial depth map and the relative camera pose transformation, the dynamic area in the initial depth map that may contain dynamic objects is identified as an untrustworthy area by calculating the difference coefficient based on the credible region detection module, while the static background area is identified as a credible area. The depth estimation of the dynamic area is guided by pseudo-depth and contrastive learning methods. Step 4: Construct a multi-task joint loss function, including photometric consistency loss, depth consistency loss, dynamic region contrast loss, and smoothness loss. Through the contrast loss mechanism, the depth information of the credible area is used to optimize the depth estimation of the untrustworthy area. Step 5: Iteratively optimize the network parameters of the depth estimation module and the pose estimation module, and output the final optimized depth map.

2. The monocular self-supervised depth estimation method based on controllable generation of credible region depth according to claim 1, characterized in that In step 1, the depth estimation module adopts a hybrid architecture that integrates CNN and Transformer, including an encoder and a decoder; 1) In the encoder part of the depth estimation module, the input image is first downsampled using convolutional layers and maximum pooling layers to obtain a feature map. Multiple residual blocks and multi-query attention modules are then stacked to extract deeper features. Each residual block includes two convolutional layers and a skip connection to enhance the network's nonlinear expression capability and gradient propagation efficiency. Each multi-query attention module includes a multi-query attention layer and a feedforward network to capture long-range dependencies in the feature map. The specific process is as follows: First, given an input feature map X∈R C×H×W , where C represents the number of channels, H and W represent the height and width of the feature map respectively. The multi-query attention layer first maps it into a query vector Q, a key vector K, and a value vector V through different convolutional layers: Q=Conv Q (X),K=Conv K (X),V=Conv V (X) (1) Among them, Conv Q 、Conv K and Conv V Represent different convolutional layers respectively; Then, the dot product attention weight A between the query vector Q and the key vector K is calculated: in, Represents the dimension of each attention head; Finally, the dot product attention weight A is multiplied by the value vector V to obtain the weighted value vector, which is then mapped back to the original dimension through a convolutional layer: MQA(X)=Conv(AV) (3) 2) In the decoder part of the depth estimation module, multiple residual blocks and upsampling modules are stacked to gradually recover the details of the depth map. PixelShuffle operation is used for upsampling. The specific process is as follows: The output data of the encoder is used as the input feature map X∈R of the decoder C×H×W , use PixelShuffle to rearrange it into an output feature map Where r represents the upsampling factor; it is expressed as: in, i,j∈[0,r-1];Y c,rh+i,rw+j Represents the value of the output feature map Y at the rh+i row, rw+j column, and c channel; Represents the value of the input feature map X at the hth row, wth column, and r^2c+ri+jth channel; Finally, the final feature map is mapped into a single-channel disparity map using a depth output module. The depth output module first further refines the features through two convolutional layers and a SiLU activation function, then uses a Sigmoid activation function to limit the output between (0, 1), and maps it to a predefined disparity range through a linear transformation. It can be expressed as: Disp=αSigmoid(Conv(SiLU(Conv(Y))))+β (5) Among them, α and β are non-learnable parameters; Disp is the disparity map; The initial depth map D is calculated by the inverse of the disparity map Disp:

3. The monocular self-supervised depth estimation method based on controllable generation of credible region depth according to claim 1, characterized in that In step 2, the posture estimation module includes an encoder and a decoder; The encoder part of the pose estimation module: the original monocular image is average-pooled twice to obtain two downsampled versions of the image; The two downsampled versions of the image are used together with the original monocular image as the input image of the pose estimation module encoder; The pose estimation module encoder extracts multi-scale feature maps through stacked residual blocks and multi-query attention modules, which capture local details and global context information in the input image; The decoder part of the pose estimation module includes a global average pooling layer and a pose output module. The global average pooling layer averages the feature map output by the pose estimation module encoder in the spatial dimension to obtain a fixed-length feature vector; the pose output module maps the feature vector to the final camera pose transformation.

4. The monocular self-supervised depth estimation method based on controllable generation of credible region depth according to claim 3, characterized in that The posture output module includes two independent multi-layer perceptrons, which are used to predict the translation vector and rotation vector of the camera posture respectively; Each multilayer perceptron consists of two fully connected layers and a SiLU activation function; the input feature vector is first predicted by a multilayer perceptron to obtain a 3D translation vector t = (t x ,t y ,t z ), which represents the translation of the camera on the three coordinate axes; then, the input feature vector is predicted by another multi-layer perceptron to obtain a 3D rotation vector r = (r x ,r y ,r z ), which represents the rotation angle of the camera around the three coordinate axes; the two multi-layer perceptrons are represented as: t=MLP t (f)=SiLU(W t2 ·SiLU(W t1 ·f+b t1 )+b t2 ) (7) r=MLP r (f)=SiLU(W r2 ·SiLU(W r1 ·f+b r1 )+b r2 ) (8) Among them, MLP t (·) represents a multi-layer perceptron for predicting translation vectors; MLP r (·) represents a multilayer perceptron used to predict the rotation vector; f represents the input feature vector; W t1 ,W t2 ,W r1 ,W r2 Represents the fully connected layer weight matrix of two multi-layer perceptrons; b t1 ,b t2 ,b r1 ,b r2 They represent the bias vectors respectively; Finally, the predicted translation vector t and rotation vector r are combined into a 6-dimensional posture vector p = (t, r), which represents the relative camera posture transformation between adjacent frames.

5. The monocular self-supervised depth estimation method based on controllable generation of credible region depth according to claim 3, characterized in that Step 3 specifically includes: Step 3.1: Use the coefficient of difference to preliminarily determine the dynamic area; By comparing the depth map D at time t t Projected to image I at time t+1 t+1 The depth map obtained from the viewing angle and t+1 Depth map D at the moment t+1 The coefficient of variation is calculated as follows: in, is achieved by converting the depth map D t Obtained by projecting the relative camera pose transformation to the viewing angle at time t+1; Assuming that the area of ​​a set proportion in the image is the dynamic area, the dynamic area is defined as: dynamic=diff top γ ％ (10) Among them, γ% is the threshold of the dynamic area, that is, the area with the top γ% of the difference coefficient value is considered to be the dynamic area; Step 3.2: Introduce pseudo-depth F t (p t ), a contrastive learning-based approach to guide self-supervised depth estimation; In dynamic areas, the network learning is guided by the difference in pseudo-depth values. The calculation method of the contrast value is defined as follows: Among them, p 0 and p 1 Represents two pixels in a picture, τ is a preset threshold used to control the sensitivity of the contrast value; F t (p 0 ) represents pixel p 0 The pseudo depth value of Step 3.3: Define any two pixel points p 0 and p 1 The loss function L p (p 0 ,p 1 )as follows: Among them, D(p 0 ) and D(p 1 ) are the pixel points p obtained by the depth estimation module 0 The value at and pixel p 1 The value at Step 3.4: Sample the dynamic area and obtain the dynamic area contrast loss function L dynamic : Among them, L p (p i ,p j ) is the pixel point p in the dynamic area i and p j The loss function, w ij is the weight calculated based on the difference coefficient value, which is used to balance the importance of different dynamic areas; specifically, where diff ij Represents a pixel pair (p i ,p j ) of the coefficient of variation.

6. The monocular self-supervised depth estimation method based on controllable generation of credible region depth according to claim 5, characterized in that In step 4, construct the multi-task joint loss function as follows: L=λ photo L photo +λ depth L depth +λ dynamic L dynamic +λ smooth L smooth (14) Among them, L photo is the photometric consistency loss, L depth is the depth consistency loss, L dynamic is the dynamic region contrast loss, L smooth is the smoothing loss; photo ,λ depth ,λ dynamic and λ smooth are the weight coefficients of each loss function.

7. The monocular self-supervised depth estimation method based on controllable generation of credible region depth according to claim 6, characterized in that The photometric consistency loss is calculated as follows: For the image I at time t t A pixel p in the static area t , using the depth D predicted by the depth estimation module t (p t ) and the camera pose transformation T predicted by the pose estimation module t→t′ , project it to the adjacent frame I t′ The corresponding position p in t′ : p t′ ~KT t→t′ D t (p t )K -1 p t (15) Where K represents the camera intrinsic parameter matrix, and ~ indicates that the homogeneous coordinates are equal; By minimizing the image I t Medium pixel p t The pixel value I t (p t ) and adjacent frame I t′ The corresponding position p in t′ The pixel value I t′ (p t′ ) to optimize the depth estimation network and the pose estimation network, while considering the grayscale difference and structural similarity of the image; for each pixel, only the minimum value of its reprojection error in all adjacent frames is calculated as the final photometric consistency loss, as follows: Where ω is a weight coefficient used to balance the importance of grayscale difference and structural similarity loss; t′ represents the adjacent frames of the image at time t, including time t-1 and time t+1; SSIM(·) represents structural similarity.

8. The monocular self-supervised depth estimation method based on controllable generation of credible region depth according to claim 7, characterized in that The depth consistency loss is calculated as follows: For the image I at time t t A pixel p in t , project it to the adjacent frame I t′ The corresponding position p in t′ , and use the adjacent frame I t′ The depth map D t′ Get the corresponding position p t′ Depth D t′ (p t′ );Then the depth D t′ (p t′ ) projected back to image I t , get pixel p t Projection depth By minimizing the depth D t (p t ) and projection depth The difference between them is used to optimize the depth estimation network and the pose estimation network; For each pixel, only the minimum value of its depth consistency loss in all adjacent frames is calculated as the final depth consistency loss, as follows: Among them, the projection depth The calculation is as follows: Among them, proj represents the projection operation, that is, the depth D t′ (p t′ ) According to the camera posture transformation T t→t′ And the camera intrinsic parameter matrix K is projected onto the image I t Pixel p in t Location.

9. The monocular self-supervised depth estimation method based on controllable generation of credible region depth according to claim 8, characterized in that The dynamic area contrast loss is calculated as follows: Based on step 3.3, different calculation strategies and weights are adopted according to the different contrast values; define that for any two points p 0 and p 1 The loss function L p (p 0 ,p 1 )as follows: in, and are the weights when contrast≠0 and contrast=0 respectively; Then the dynamic area is sampled to obtain the dynamic area contrast loss function.

10. The monocular self-supervised depth estimation method based on controllable generation of credible region depth according to claim 8, characterized in that The smoothing loss is calculated as follows: First, the predicted depth map and input image are Gaussian blurred respectively, and then their gradients in the horizontal and vertical directions are calculated; the calculation formula of the smoothing loss is as follows: Where N is the total number of pixels, D i,j Represents the depth value of pixel (i, j) in the depth map, I i,j represents the pixel value of pixel (i, j) in the input image, and Represent the gradient operators in the horizontal and vertical directions respectively; and is the edge weight term, which is used to reduce the penalty of depth gradient at the edge of the image.

Images