A self-supervised monocular depth estimation method based on spatial-semantic prior feature enhancement

Abstract

The application belongs to the technical field of computer vision, and particularly relates to a self-supervised monocular depth estimation method based on space-semantic prior feature enhancement. A perception model constructed offline comprises a depth encoder and a depth decoder of a monocular depth estimation network and a reinforced depth feature generation module. The module is used to acquire scene semantics and spatial prior features of a single frame, compress and fuse the two prior features, adopt a dual structure, perform convolution feature extraction on the two compressed and fused prior features respectively and align the features with initial depth feature channels, adopt an attention mechanism to perform feature interaction and fusion on the aligned prior features and the initial depth features, perform shape and channel processing on the two interaction and fusion features respectively, perform addition operation on the three features and the initial depth features to obtain reinforced depth features, and decode the depth features by the depth decoder. The application can improve the robustness of monocular depth estimation in a complex feature degradation environment.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and more specifically, relates to a self-supervised monocular depth estimation method based on spatial-semantic prior feature enhancement. Background Technology

[0002] Depth perception is a core foundation for computer vision applications in various fields such as autonomous driving, robotics, and augmented reality. In recent years, with the continuous advancement of deep learning and the release of related datasets, supervised monocular depth estimation (MDE) algorithms can effectively infer environmental depth from a single RGB image. However, these supervised algorithms require a large collection of images with depth labels for training, but the cost of acquiring such data is usually high, which greatly limits the application capabilities of the algorithms. Therefore, methods that utilize large amounts of unlabeled real-world data for training have gradually attracted more attention, and this trend has greatly promoted the research of self-supervised algorithms using monocular video sequences as input.

[0003] While self-supervised monocular depth estimation algorithms perform well in many applications, they often suffer from instability in their self-supervised photometric training signals when faced with numerous visually blurred regions in feature-degraded environments. This leads to decreased depth estimation accuracy and insufficient geometric modeling. Furthermore, traditional vision methods rely excessively on low-level texture features, making it difficult to accurately capture relative depth information between instances in complex scenes, especially in areas lacking clear edges or textures, where the algorithm's performance may significantly degrade.

[0004] This shows that existing self-supervised monocular depth estimation algorithms have insufficient robustness in complex feature degradation environments. Summary of the Invention

[0005] To address the above-mentioned deficiencies or improvement needs of existing technologies, this invention provides a self-supervised monocular depth estimation method based on spatial-semantic prior feature enhancement, aiming to improve the robustness of self-supervised monocular depth estimation in complex feature degradation environments.

[0006] To achieve the above objectives, according to one aspect of the present invention, a self-supervised monocular depth estimation method based on spatial-semantic prior feature enhancement is provided, comprising:

[0007] (1) In the offline stage, a monocular depth perception model is obtained through photometric consistency self-supervised training, including: a depth encoder of the monocular depth estimation network, used to obtain the initial depth features of a single frame image; an enhanced depth feature generation module, used to extract the scene spatial prior features corresponding to a single frame image through a pre-trained visual base model, extract the scene semantic prior features corresponding to a single frame image through a pre-trained visual-semantic alignment multimodal model, compress and fuse the spatial prior features and semantic prior features respectively through a multi-level feature fusion unit, extract convolutional features from the two compressed and fused prior features respectively through a cross-modal feature fusion unit and align them with the initial depth feature channels, use an attention mechanism to perform feature interaction fusion between each aligned prior feature and the initial depth feature, process the shape and channels of the two interactively fused features respectively, and perform addition operations between the three features with the initial depth feature to obtain the spatial-semantic fusion enhanced depth features; and a depth decoder of the monocular depth estimation network, used to perform upsampling and cross-layer connection synchronously at each level on the enhanced depth features to reconstruct multi-size depth maps.

[0008] During the photometric consistency self-supervised training process, the model training loss based on the self-supervised training objective function is calculated, and the model parameters are updated by backpropagating gradients until convergence, thus completing the construction of the perception model.

[0009] (2) In the online stage, monocular depth estimation: input a single frame image into the constructed monocular depth perception model to estimate the monocular depth.

[0010] Furthermore, the visual base model uses the DepthAnythingv2 encoder, and the visual-semantic aligned multimodal model uses the SigLIP encoder; the DepthAnythingv2 encoder and the SigLIP encoder respectively extract the hierarchical features of the input image, which are used as semantic prior features and spatial prior features respectively.

[0011] Furthermore, the multi-level feature fusion unit compresses and fuses each prior feature in the following way:

[0012] Layer normalization is performed on each level of the prior features. 1×1 convolution is used to align the channels of the standardized prior features between different levels to achieve single-modal feature channel dimension remapping. Spatial dimension alignment of single-modal features is completed through interpolation and reshape operations to achieve spatial resolution matching.

[0013] Furthermore, the cross-modal feature fusion unit also processes the shape and channel of each prior feature after compression and fusion. Before the addition operation, the prior feature after shape and channel processing is spliced ​​with the interactive fusion feature corresponding to the prior feature after shape and channel processing, thereby preserving the effect of the original prior feature.

[0014] Furthermore, the self-supervised training objective function includes a photometric consistency constraint loss function and a gradient-aware smoothing loss function;

[0015] The model training loss is a weighted sum of the values ​​of the photometric consistency constraint loss function and the gradient-aware smoothing loss function.

[0016] Furthermore, the photometric consistency constraint loss function L p for:

[0017]

[0018] In the formula, pe(I a ,I b ) indicates the luminosity consistency between two images, I a I represents the keyframe image in the training samples to be used for monocular depth estimation. b Indicates that during the training process, I a The reference frame is projected onto I a The reprojected image; α represents the hyperparameters of the equilibrium structural similarity index and L1 norm; SSIM(I a ,I b ) indicates I a ,I b Structural similarity index between them; ||I a -I b ||1 represents I a ,I b The L1 norm between them;

[0019] Gradient-aware smoothing loss function L smooth for:

[0020]

[0021] In the formula, These represent the multi-scale depth maps I output by the perception model. t Partial derivatives in the x and y directions; This represents the inverse depth of the average normalization; and Representing multi-size depth maps I t The absolute values ​​of the gradients in the x and y directions.

[0022] Furthermore, in the offline stage, the method also includes: using the constructed monocular depth perception model as the teacher model, using the spatial-semantic fusion enhanced depth features obtained from the teacher model as intermediate features, using the multi-size depth map obtained from the teacher model as a pseudo-label depth map, constraining the training of the student model through knowledge distillation, and using the resulting student model for monocular depth estimation in the online stage.

[0023] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0024] According to another aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the storage medium is located to perform the steps of the method described above.

[0025] In summary, compared with the prior art, the technical solutions conceived by this invention have the following main advantages:

[0026] 1. This invention proposes a self-supervised monocular depth estimation method based on spatial-semantic prior feature enhancement. In the offline-trained perception model, this invention proposes that the perception model not only includes a depth encoder and a depth decoder of the monocular depth estimation network, but also adds an enhanced depth feature generation module after the output of the depth encoder. This module is used to acquire scene semantic prior features and scene spatial prior features of the input image. The spatial and semantic prior features are compressed and fused separately. Then, a dual structure is adopted to extract convolutional features from the two compressed and fused prior features and align them with the initial depth feature channels. An attention mechanism is used to perform feature interaction fusion between each aligned prior feature and the initial depth features. The two interactively fused features are processed for shape and channel, respectively, to perform a three-feature addition operation with the initial depth features, resulting in spatial-semantic fused enhanced depth features, which are input into the depth decoder for depth map decoding. The prior spatial features obtained by the visual base model through training on large-scale geometric data can be effectively compensated for, and the language alignment model can construct semantic associations through cross-modal contrastive learning, providing high-level abstract cues for scene understanding. Furthermore, compression fusion and cross-modal fusion enable spatial features to maintain the continuity of scene representation through pixel-level geometric constraints, semantic features to enhance region boundary perception through object-level concept segmentation, and the attention mechanism to achieve non-linear interaction between the two through dynamic weight allocation. This fusion strategy is equivalent to introducing regularization of prior knowledge, which can correct gradient biases caused by interference factors such as illumination changes, motion blur, and sparse regional textures in self-supervised training based on photometric consistency. Essentially, it implicitly constructs structured constraint terms in the loss function, enabling the model to simultaneously satisfy the three convergence conditions of geometric consistency, semantic rationality, and photometric smoothness during the optimization process, greatly improving the robustness of self-supervised monocular depth estimation in complex feature degradation environments.

[0027] 2. To ensure the effectiveness of prior features, this invention also proposes a cross-modal feature fusion unit that processes the shape and channels of each compressed and fused prior feature. Before the addition operation, the prior feature processed by shape and channels is concatenated with the interactively fused feature corresponding to the prior feature processed by shape and channels, thereby preserving the effect of the original prior feature. In the final fusion stage, it is concatenated with the enhanced feature, thereby better realizing the triple deep fusion of space-semantics-photometry.

[0028] 3. To enhance the application capability of the design model of this invention, a joint distillation strategy based on intermediate features and pseudo-label depth maps is also introduced. Attached Figure Description

[0029] Figure 1 A flowchart illustrating a self-supervised monocular depth estimation method based on spatial-semantic prior feature enhancement, provided in an embodiment of the present invention;

[0030] Figure 2 This is a diagram illustrating the overall framework for training the perception model provided in an embodiment of the present invention.

[0031] Figure 3 The structural diagrams of the multi-level feature fusion unit and the cross-modal feature fusion unit provided in the embodiments of the present invention are as follows;

[0032] Figure 4 A framework diagram of the joint distillation strategy for intermediate features and pseudo-label depth maps provided in embodiments of the present invention;

[0033] Figure 5 The image shows the effect of depth estimation on the KITTI dataset provided in the embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0035] Example 1

[0036] A self-supervised monocular depth estimation method based on spatial-semantic prior feature enhancement includes:

[0037] (1) In the offline stage, a monocular depth perception model is obtained through photometric consistency self-supervised training, including: a depth encoder of the monocular depth estimation network, used to obtain the initial depth features of a single frame image; an enhanced depth feature generation module, used to extract the scene spatial features corresponding to a single frame image through a pre-trained visual base model as spatial prior features, and extract the scene semantic features corresponding to a single frame image through a pre-trained visual-semantic alignment multimodal model as semantic prior features, and compress and fuse the spatial prior features and semantic prior features respectively through a multi-level feature fusion unit, and extract convolutional features from the two compressed and fused prior features respectively through a cross-modal feature fusion unit and align them with the initial depth feature channels, and use an attention mechanism to perform feature interaction fusion between each aligned prior feature and the initial depth feature, and process the shape and channels of the two interactively fused features respectively, so as to perform the three-feature addition operation with the initial depth feature to obtain the spatial-semantic fusion enhanced depth features; and a depth decoder of the monocular depth estimation network, used to perform upsampling and cross-layer connection synchronously on the enhanced depth features to reconstruct multi-size depth maps;

[0038] During the photometric consistency self-supervised training process, the model training loss based on the self-supervised training objective function is calculated, and the model parameters are updated by backpropagating gradients until convergence, thus completing the construction of the perception model.

[0039] (2) In the online stage, monocular depth estimation: input a single frame image into the constructed monocular depth perception model to estimate the monocular depth.

[0040] In the enhanced deep feature generation module, the cross-modal feature fusion unit adopts a dual structure to simultaneously process spatial prior features and semantic prior features. Specifically, spatial prior features are combined with the initial deep features and enhanced through convolution and attention mechanisms; simultaneously, semantic prior features are also combined with the initial deep features and enhanced using the same convolution and attention mechanisms; finally, the two interactively fused features are fused with the initial deep features to achieve a triple deep fusion of spatial, semantic, and photometric features.

[0041] As a preferred approach, the visual base model uses the DepthAnythingv2 encoder, and the visual-semantic aligned multimodal model uses the SigLIP encoder. The DepthAnythingv2 encoder and the SigLIP encoder extract the hierarchical features (Layer 1 to Layer 4) of the input image, which are used as semantic prior features and spatial prior features, respectively.

[0042] The DepthAnythingv2 model extracts multi-level spatial geometric features through hybrid synthesis and pre-training with real-world scene data; the SigLIP model achieves zero-shot mapping between image semantic features and text descriptions through cross-modal contrastive learning.

[0043] Taking the SigLIP model and the DepthAnythingv2 model as examples, the photometric consistency self-supervised training method is as follows:

[0044] Step 1: Input a reference frame and a key frame (the frame to be used for monocular depth estimation), and extract the reference frame I from the video sequence. t+1 I t-1 and keyframe I t ;

[0045] Step 2: Relative pose estimation. Use PoseNet to estimate the camera pose change from the reference frame to obtain the rotation matrix R and translation vector t.

[0046] Step 3: Keyframe Depth Map Estimation. Depth estimation is performed using DepthNet (a monocular depth estimation network) that includes a depth encoder and a depth decoder. Spatial features are extracted from keyframes using DepthAnythingv2, and semantic features are extracted using SigLip. Both types of features are normalized and subjected to multi-level feature aggregation before being fed into a cross-modal feature fusion unit to perform cross-modal depth aggregation with the photometric features (i.e., the initial depth features) extracted by the original backbone network (the depth encoder of the monocular depth estimation network). Based on this, the decoder uses a cascaded upsampling structure to progressively restore the depth map resolution, integrating features from different levels of encoder and decoder through skip connections to output the final multi-size depth map and improve local detail accuracy.

[0047] Step 4: Image reprojection. Using the pose estimated in Step 2 (rotation matrix R and translation vector t) and the multi-scale depth map estimated in Step 3, the reference frame is reprojected onto the keyframe viewpoint to generate the reprojected frame I. t±1→t ;

[0048] Step 5: Calculate the training loss function. Optimize the model by calculating the photometric loss and depth smoothing loss between the reprojected frames and keyframes to ensure accurate depth and pose estimations.

[0049] Step 6: Complete training, input the image to be estimated, and use the trained and improved depth estimation network to perform depth estimation on the image and output the result.

[0050] The characteristic of the method in this embodiment is that the deep fusion mechanism of the spatial-semantic basic model is based on the theoretical framework of multimodal representation complementarity and scene understanding hierarchy.

[0051] Addressing the performance bottleneck of self-supervised monocular depth estimation in feature degradation scenarios, the core motivation stems from two aspects:

[0052] First, feature degradation scenarios lead to photometric signal degradation in self-supervised learning, which further results in insufficient geometric structure modeling. The prior spatial features obtained by the visual basic model through training on large-scale geometric data can effectively compensate for this.

[0053] Secondly, traditional visual methods rely excessively on low-level texture features, while language alignment models can build semantic associations through cross-modal contrastive learning, providing high-level abstract clues for scene understanding.

[0054] The mechanism of this multi-level feature fusion (compressed fusion and cross-modal fusion) is that spatial features maintain the continuous representation of the scene through pixel-level geometric constraints, semantic features strengthen the perception of region boundaries through object-level concept division, and the attention mechanism achieves non-linear interaction between the two through dynamic weight allocation.

[0055] Theoretical analysis shows that this fusion strategy is equivalent to introducing regularization based on prior knowledge, which can correct gradient bias caused by interference factors such as illumination changes, motion blur, and sparse regional textures in photometric consistency-supervised training. Essentially, it implicitly constructs structured constraint terms in the loss function, ensuring that the model simultaneously satisfies the three convergence conditions of geometric consistency, semantic rationality, and photometric smoothness during optimization, as shown in the following equation:

[0056]

[0057] In the formula, L self Indicates the self-monitored loss item. Represents the regularization term for prior features in the space; This represents the semantic prior feature regularization term.

[0058] Self-supervised monocular depth estimation algorithms transform the depth estimation task into a cross-view problem by assuming a static scene. Figure 1 This addresses the consistency issue, thereby achieving self-supervised training. The algorithm's flowchart and overall framework are as follows: Figure 1 and Figure 2 As shown. Specifically, the algorithm generates a photometric consistency loss to optimize the model by calculating the reprojection difference between the keyframe and the reference frame. The overall framework includes a depth estimation network, DepthNet, consisting of an encoder and a decoder, used to obtain the predicted depth information of the keyframe; and a pose estimation network, PoseNet, is introduced to estimate the 6-DOF pose changes between the keyframe and the reference frame.

[0059] Under the assumption of a static scene, the self-supervised monocular depth estimation algorithm synthesizes the scene from the keyframe's perspective (used for loss calculation) by utilizing the predicted depth map of the keyframe, the relative pose transformation of the reference frame, the input image of the reference frame, and the structure for motion reconstruction (SFM). Compared to multi-frame self-supervised models, this algorithm only uses the two frames before and after the keyframe as reference frames. The mapping formula for projecting each pixel in the keyframe onto the previous and next frames is as follows:

[0060] p t±1 ~KT t→t±1 D t (p t )K -1 p t

[0061] I t±1→t [p t ] = I t±1 [p t±1 ]

[0062] Where [] represents the sampling operation, K is the camera intrinsic parameter matrix, and p t±1 isIt±1 Pixels in an image. D t (p t ) and T t→t±1 These are the prediction results from the depth estimation network and the pose estimation network, respectively. According to the above formula, the algorithm can sample I... t±1 Obtain the reconstructed view I of the keyframes t±1→t .

[0063] In the training process of the self-supervised monocular depth estimation algorithm, to improve the model's performance and robustness, a comprehensive loss function consisting of photometric loss and gradient-aware smoothing loss is preferred. The photometric loss used by the algorithm combines structural similarity (SSIM) loss and luminance loss (L1), and balances the importance of the two through the hyperparameter α to capture the overall illumination and structural information of the image. The calculation formula is as follows:

[0064]

[0065] Secondly, to smooth object edges in the depth map and normalize textureless regions, the algorithm introduces gradient-aware smoothing loss to enhance the continuity of depth prediction. The calculation formula is as follows:

[0066]

[0067] In the formula, I represents the inverse depth of the average normalized depth, used to avoid the shrinkage problem in depth estimation. t Indicates the input image. and Let represent the partial derivatives of the image in the x and y directions, respectively. and It is the absolute value of the gradient of the input image in the corresponding direction. This is expressed through an exponential decay term. and The above formula suppresses the contribution of regions with large gradients to the training loss, thus focusing more on the depth estimation of smooth regions.

[0068] Ultimately, the photometric loss and gradient-aware smoothing loss are combined to form the training loss function, providing an effective supervision signal for the model, as shown in the following equation:

[0069] L self =L p +λL smooth

[0070] In the formula, λ represents the hyperparameter used to control the weights of different loss functions.

[0071] The cross-modal feature fusion unit is a dual-type deep fusion structure based on spatial-semantic prior feature enhancement proposed in this embodiment. An exemplary structure is shown below. Figure 3 As shown.

[0072] DepthAnythingv2 is an advanced visual foundation model for monocular depth estimation. Its core advantage lies in its efficient modeling of prior spatial information of scenes through training on millions of mixed datasets. The model's training dataset integrates five types of high-precision synthetic datasets (covering various virtual scene renderings) and eight large-scale real image libraries (NYU, KITTI, etc.). This hybrid training approach provides pixel-level depth benchmarks through synthetic datasets and enhances the model's cross-domain adaptability with real data, building a powerful geometric perception capability covering dynamic / static scenes. This makes the model particularly adept at inferring 3D spatial structures from single 2D images. Furthermore, to address the generalization problem caused by synthetic images, the model employs a data-driven (large-scale pseudo-labeled real images) and model-driven (expanding the size of the teacher model) strategy. Therefore, DepthAnythingv2 performs significantly better than other general-purpose visual foundation models in extracting spatial prior information.

[0073] SigLIP is a foundational model based on visual-language alignment. This model is trained on large-scale image-text alignment data and employs a joint optimization of image and text embedding spaces to ensure higher similarity between semantically related image components and textual information in the high-dimensional feature space, thereby achieving semantic mapping between image and text features. Specifically, in the model pre-training stage, the algorithm extracts features using visual encoders (such as ResNet / ViT) and text encoders (such as Transformer), optimizing the similarity of positive sample pairs and suppressing negative sample pairs during training, thus learning generalized representations without manual annotation. In the model inference stage, the algorithm directly transforms downstream tasks (such as image classification) into cross-modal matching problems. By converting category labels into natural language descriptions (such as "a photo of {category}"), it calculates the similarity between image and text features to achieve zero-shot prediction, greatly simplifying the model deployment process. Compared to the traditional visual language model CLIP, SigLIP innovatively introduces the Sigmoid cross-entropy loss function. By independently calculating the matching probability of each sample pair, it significantly reduces false negative interference, enabling the model to focus more on the essential semantic associations when extracting visual features.

[0074] Based on this, the spatial prior features output by DepthAnythingv2 and the semantic prior features output by SigLIP are respectively normalized hierarchically and then fed into the multi-level feature fusion unit for multi-level feature aggregation.

[0075] Hierarchical normalization aims to eliminate inconsistencies in the distribution of features at different levels, thereby improving feature representation and fusion efficiency. The multi-level feature fusion module captures multi-scale information representations by dynamically integrating features at different levels, thus ensuring effective collaboration between deep and shallow features, as shown in the following equation:

[0076] z = w1LN(z1) + w2LN(z2) + ... + w N LN(z N )

[0077] in, These represent different levels of features extracted from the backbone network.

[0078] In the detailed design of the cross-modal feature fusion structure, 1×1 convolution and reshape operations are preferably introduced into the module. The main function of 1×1 convolution is to reduce the dimensionality of features and remap the channel distribution, thereby effectively improving the feature representation capability of the network while reducing computational complexity. The reshape operation is used to adjust the feature size to ensure that it is suitable for the subsequent fusion process, thereby achieving accurate alignment of multimodal features in spatial dimensions.

[0079] Meanwhile, to further improve the quality of feature representation, a lightweight convolutional attention module (CBAM) is introduced in the fusion part, for example. This module combines channel attention and spatial attention mechanisms, dynamically adjusting feature weights through these two attention mechanisms to enhance key information in the features.

[0080] First, the channel attention mechanism extracts global information from the input features through global average pooling and max pooling, and generates channel weights through a fully connected layer, compressing the spatial dimension while keeping the channel dimension unchanged. Second, the spatial attention mechanism generates a spatial attention map by performing max and average pooling operations on the input features along the channel dimension, and calculates spatial weights using convolution operations, compressing the channel dimension while keeping the spatial dimension unchanged, thus more effectively focusing on the location information of the target. Finally, the CBAM module connects the channel attention mechanism and the spatial attention mechanism, and sequentially adjusts the weights of the input features, ultimately outputting enhanced multi-attention features.

[0081] Furthermore, during feature fusion, photometric information (i.e., initial depth features) is embedded into the network through convolution and interpolation operations, interacting with spatial and semantic features to form more expressive depth features. Preferably, an independent branch is designed to retain prior features, which are then concatenated with the enhanced features in the final fusion stage as the encoder's output. These fused features are ultimately processed by the depth decoder to generate a high-precision dense depth map.

[0082] Regarding independent branches, the cross-modal feature fusion unit processes the shape and channels of each prior feature after compression and fusion. Before the addition operation, the prior feature after shape and channel processing is concatenated with the interactively fused feature corresponding to the prior feature after shape and channel processing, thereby preserving the effect of the original prior feature.

[0083] In other words, to ensure the effectiveness of prior features, this method also retains an independent branch to pass on prior features, and then concatenates them with the enhanced features in the final fusion stage, thereby better achieving spatial-semantic-photometric triple deep fusion.

[0084] Furthermore, to enhance the application capability of the design model of this invention, a preferred approach is to introduce a joint distillation strategy based on intermediate features and pseudo-label depth maps, with the framework as follows: Figure 4 As shown, the offline phase also includes: using the constructed monocular depth perception model as the teacher model, using the spatial-semantic fusion-enhanced depth features obtained from the teacher model as intermediate features, using the multi-size depth maps obtained from the teacher model as pseudo-labeled depth maps, constraining the training of the student model through knowledge distillation, and using the resulting student model for monocular depth estimation in the online phase. Specifically, other depth perception models with lighter network structures than the teacher model are selected as student models.

[0085] The intermediate feature distillation loss can be achieved by calculating the difference between the teacher model and the student model in the deep feature space. This invention uses the common L2 loss metric, as shown in the following formula:

[0086]

[0087] Depth pseudo-label distillation loss is calculated by normalizing the predicted depth. and deep pseudo tags The mean absolute error between them is used to achieve this, as shown in the following formula:

[0088]

[0089] in, and It is to predict the depth value d of the student model. pred And the pseudo-label depth d of the teacher model pseudo The result after scaling and offset normalization is shown in the normalization formula below:

[0090]

[0091] The purpose of normalization is to eliminate depth value offsets and scale differences between different models.

[0092] Finally, the comprehensive training loss function used for joint distillation of intermediate features and pseudo-label depth maps is shown in the following equation:

[0093] L total =L p +λ1L smooth +λ2L feat +λ3L pred

[0094] In the formula, λ1, λ2, and λ3 represent hyperparameters used to control the weights of different loss functions, and L p L represents the photometric consistency constraint loss function. smooth This represents the gradient-aware smoothing loss function.

[0095] To verify the effectiveness of this invention, the following examples are provided:

[0096] KITTI is used as the primary dataset. As a fundamental dataset in the field of computer vision, it is widely used in depth estimation tasks. The specific implementation details are as follows:

[0097] During training, the batch size was set to 12, the total number of training epochs was set to 20, and the AdamW optimizer was used to jointly train the teacher and student models. The initial learning rate was maintained at 1e-4 for the first 15 epochs and decreased to 1e-5 for the last 5 epochs. During training, the network parameters of both the visual base model and the visual-semantic alignment multimodal model were frozen. For hyperparameter settings, λ1 = 1, λ2 = 5e-2, and λ3 = 1e-3. In terms of network architecture design, the SQLdepth model was chosen as the baseline model, where the pose estimation network PoseNet adopted the ResNet18 architecture and loaded model weights pre-trained on ImageNet. The original encoder part of the deep encoder SSPEM adopted the ResNet50 architecture and also loaded model weights pre-trained on ImageNet. Furthermore, the visual base model and the visual-semantic alignment multimodal model used the DepthAnythingv2 and SigLIP DINOv2-Large architectures, respectively, and loaded the corresponding algorithm training weights.

[0098] The depth estimation experimental results are combined with multiple evaluation metrics to comprehensively evaluate the model according to specific needs. Typically, the evaluation metrics for depth estimation algorithms include: Absolute Relative Error (AbsRel), Squared Relative Error (SqRel), Root Mean Squared Error (RMSE), Root Mean Squared Logarithmic Error (RMSELog), and threshold accuracy δ. The specific calculation formulas for all evaluation metrics are as follows:

[0099]

[0100] Threshold accuracy is typically expressed in different threshold forms, such as δ1 (δ < 1.25), δ2 (δ < 1.25), etc. 2 ), δ3 (δ<1.25) 3 The numbers 1, 2, and 3 represent the proportions of the predicted depth to the true depth ratio or the reciprocal of the predicted depth within the range of 1.25, 1.5625, and 1.953125, respectively.

[0101] The error and accuracy comparison results between the present invention and the SQLdepth model are shown in Tables 1 and 2.

[0102] Table 1. Error comparison results between the present invention and the SQLdepth model.

[0103]

[0104] Table 2. Accuracy comparison results between the present invention and the SQLdepth model.

[0105]

[0106] In addition, such as Figure 5 The image shown is a depth estimation result of three different time frames in the same scene using the method of this embodiment. It can be seen that the method of this embodiment can reasonably perceive depth, and the detected contours are relatively clear and the spatial relationships are reasonable.

[0107] Example 2

[0108] This application also relates to an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0109] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.

[0110] The relevant technical solutions are the same as above, and will not be repeated here.

[0111] Example 3

[0112] This application also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0113] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0114] The relevant technical solutions are the same as above, and will not be repeated here.

[0115] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A self-supervised monocular depth estimation method based on spatial-semantic prior feature enhancement, characterized in that, include: (1) In the offline stage, a monocular depth perception model is obtained through photometric consistency self-supervised training. The model includes: a depth encoder of the monocular depth estimation network, which is used to obtain the initial depth features of a single frame image; an enhanced depth feature generation module, which is used to extract the scene spatial prior features corresponding to a single frame image through a pre-trained visual basic model, extract the scene semantic prior features corresponding to a single frame image through a pre-trained visual-semantic alignment multimodal model, compress and fuse the spatial prior features and semantic prior features respectively through a multi-level feature fusion unit, extract convolutional features of the two compressed and fused prior features respectively through a cross-modal feature fusion unit and align them with the initial depth feature channels, use an attention mechanism to perform feature interaction fusion between each aligned prior feature and the initial depth feature, process the shape and channels of the two interactively fused features respectively, and perform addition operations between the three features with the initial depth feature to obtain the spatial-semantic fusion enhanced depth features; and a depth decoder of the monocular depth estimation network, which is used to perform upsampling and cross-layer connection operations on the enhanced depth features step by step to reconstruct multi-size depth maps. During the photometric consistency self-supervised training process, the model training loss based on the self-supervised training objective function is calculated, and the model parameters are updated by backpropagating gradients until convergence, thus completing the construction of the perception model. (2) In the online stage, monocular depth estimation: input a single frame image into the constructed monocular depth perception model to estimate the monocular depth; The cross-modal feature fusion unit further processes the shape and channel of each prior feature after compression and fusion. Before the addition operation, the prior feature after shape and channel processing is spliced ​​with the interactively fused feature corresponding to the prior feature after shape and channel processing, thereby preserving the effect of the original prior feature.

2. The self-supervised monocular depth estimation method as described in claim 1, characterized in that, The visual base model uses the DepthAnythingv2 encoder, and the visual-semantic aligned multimodal model uses the SigLIP encoder. The DepthAnythingv2 encoder and the SigLIP encoder extract hierarchical features from the input image, which are used as spatial prior features and semantic prior features, respectively.

3. The self-supervised monocular depth estimation method as described in claim 1, characterized in that, The multi-level feature fusion unit compresses and fuses each prior feature in the following way: Layer normalization is performed on each level of the prior features. 1×1 convolution is used to align the channels of the standardized prior features between different levels to achieve single-modal feature channel dimension remapping. Spatial dimension alignment of single-modal features is completed through interpolation and reshape operations to achieve spatial resolution matching.

4. The self-supervised monocular depth estimation method as described in claim 1, characterized in that, The self-supervised training objective function includes a photometric consistency constraint loss function and a gradient-aware smoothing loss function. The model training loss is a weighted sum of the values ​​of the photometric consistency constraint loss function and the gradient-aware smoothing loss function.

5. The self-supervised monocular depth estimation method as described in claim 4, characterized in that, Photometric consistency constraint loss function for: In the formula, This indicates the consistency of luminosity between two images. This represents the keyframe image in the training samples that needs to be estimated using a monocular depth sensor. Indicates that during the training process, by The reference frame is projected onto The reprojected image; The hyperparameters representing the balance between the structural similarity index and the L1 norm; express Structural similarity index between them; express The L1 norm between them; Gradient-aware smoothing loss function for: In the formula, , These represent the multi-scale depth maps output by the perception model. Partial derivatives in the x and y directions; This represents the inverse depth of the average normalization; These represent multi-size depth maps. The absolute values ​​of the gradients in the x and y directions.

6. The self-supervised monocular depth estimation method as described in claim 1, characterized in that, In the offline phase, the following steps are also included: using the constructed monocular depth perception model as the teacher model, using the spatial-semantic fusion enhanced depth features obtained from the teacher model as intermediate features, using the multi-size depth map obtained from the teacher model as a pseudo-label depth map, constraining the training of the student model through knowledge distillation, and using the resulting student model for monocular depth estimation in the online phase.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein the computer program, when executed by a processor, controls the device on which the storage medium is located to perform the steps of the method as described in any one of claims 1 to 6.

Images