A Monocular Depth Estimation Method and System Based on Text-Guided and Multi-Scale Fusion

Abstract

This invention discloses a monocular depth estimation method and system based on text guidance and multi-scale fusion. It employs a cross-semantic attention module to dynamically fuse these features and combines this with a multi-scale residual fusion module to significantly improve the accuracy and robustness of depth estimation. Unlike traditional methods that rely solely on image features, this embodiment fully utilizes the rich semantic priors inherited in the pre-trained encoder, thereby enhancing the expressive power of image features. Furthermore, the corresponding text descriptions of the images are encoded and incorporated into the model as prior knowledge. The weights of image and text features are dynamically adjusted and fused together through a cross-attention mechanism. Subsequently, multi-scale residual calculation captures fine-grained changes in the image at different resolutions, enabling the model to simultaneously understand local information of details and global scene layout. This design not only enhances the model's ability to adapt to complex scenes but also significantly improves the recovery of depth boundaries and details.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology and relates to a monocular depth estimation method and system based on text guidance and multi-scale fusion. Background Technology

[0002] Depth estimation of monocular images has long been a key task in many practical applications, and is widely used in fields such as robotics, autonomous driving, and virtual reality. Due to these rich possibilities, many researchers have devoted a great deal of effort to solving the monocular depth estimation problem.

[0003] Since the advent of deep learning, deep neural networks have been successfully applied to various vision tasks, achieving remarkable results. Many excellent works based on deep neural networks have also emerged in the field of monocular depth estimation, demonstrating its enormous potential in solving such vision tasks. Although deep neural network-based methods have shown a powerful ability to reveal depth layouts without requiring domain knowledge, they still face many challenges, including scene complexity, diversity, and the variability of object shapes and sizes. Specifically, most existing methods utilize visual features that are primarily dependent on the image for depth prediction. These visual features are simply upsampled back to their original size through a decoding process in a symmetric architecture and ultimately converted into a depth map. Despite some progress, limitations remain when handling complex scenes and subtle details.

[0004] In recent years, multimodal learning methods have shown great potential in improving depth estimation performance. For example, VPD introduces the fusion of visual and textual information, achieving more accurate depth prediction by jointly learning visual features and textual descriptions. WorDepth utilizes textual descriptions as semantic priors and improves depth estimation accuracy through a conditional generative model. Furthermore, CLIPD innovatively combines the semantic knowledge of the CLIP model with the depth estimation task, transforming the depth value regression problem into a distance classification problem. This approach achieves zero-shot depth estimation, surpassing existing unsupervised methods without requiring additional training data. These studies demonstrate that fusing information from multiple modalities can significantly improve the performance of depth estimation tasks, especially when dealing with complex and diverse scenes.

[0005] With the rapid development of deep learning and computer vision, monocular depth estimation has made significant progress in the research of various encoder-decoder architectures. However, traditional depth estimation methods often rely solely on image features, frequently neglecting the potential value of textual information. This leads to limitations in handling complex scenes, diverse objects, and capturing global and local semantic information, making monocular depth estimation an inherently ill-posed problem. Summary of the Invention

[0006] The purpose of this invention is to address the problem that traditional depth estimation methods in the prior art often rely solely on image features, frequently neglecting the potential value of textual information. This leads to limitations in handling complex scenes, diverse objects, and capturing global and local semantic information. The invention provides a monocular depth estimation method and system based on text guidance and multi-scale fusion.

[0007] To achieve the above objectives, the present invention employs the following technical solution:

[0008] A monocular depth estimation method based on text guidance and multi-scale fusion includes the following steps:

[0009] Obtain the visual features of the original image and the textual features of the original text;

[0010] The text features are mapped to the same dimension as the visual features of the original image to obtain the processed text features. The processed text features are then introduced into the visual features of the original image to obtain image features with semantic guidance. The text features are then enhanced based on the visual features to obtain enhanced text features.

[0011] Enhanced text features are introduced into semantically guided image features through bidirectional cross-guidance to obtain the final enhanced image features and acquire the enhanced image;

[0012] Multi-scale Laplacian residual calculation is performed on the original image and the enhanced image to obtain residual information between the original image and the enhanced image. Based on the residual information and the enhanced image features, a fusion operation is performed to obtain the estimated depth image.

[0013] A further improvement of the present invention is that:

[0014] The text features are mapped to the same dimension as the image information, and the processed text features ti are obtained by the following formula:

[0015] t i =Proj i ·t

[0016] Among them, Proj i This represents the projection matrix corresponding to scale i, with dimensions 1024×Ci; Ci is the number of channels of the image features in the i-th layer.

[0017] The process involves introducing the processed text features into the visual features of the original image to obtain semantically guided image features. The text features are then enhanced based on the visual features of the original image to obtain enhanced text features, including:

[0018] Using text features ti as the query and image features fi as the key and value, an enhancement mechanism is used to generate image features with semantic guidance.

[0019] The query, key, and value are each subjected to multiple different linear transformations. Then, the attention of multiple heads is calculated separately, and the outputs of the multiple heads are concatenated to obtain the enhanced image features.

[0020] The enhanced image features This can be expressed by the following formula:

[0021]

[0022] Where H represents the number of heads, W o Represents the linear transformation matrix of the output; Ai represents the output result of the head;

[0023] The entire process is defined as cross-attention:

[0024]

[0025] Among them, T i Represents the features of the projected text; This represents the flattening of image features.

[0026] The enhancement of text features based on visual features of the original image yields enhanced text features, including:

[0027]

[0028] Among them, T i This represents the text features after projection; This represents the flattening of image features.

[0029] The process of introducing enhanced text features into semantically guided image features through bidirectional cross-guidance to obtain the final enhanced image features includes:

[0030] Enhanced text features Enhanced image features as query input As keys and values, further enhanced image features are obtained.

[0031]

[0032] in, Indicates enhanced image features; Indicates enhanced text features;

[0033] The enhanced image features and the original image features are concatenated, and then a 2D convolution operation is performed.

[0034] The feature representation is further optimized using weighted residual connections to obtain the final enhanced features.

[0035] The process of obtaining residual information between the original image and the enhanced image includes:

[0036] Calculate the Laplacian residual L of the original image k :

[0037] L k =I k -Up(I k+1 )

[0038] Multi-scale Laplacian residuals are calculated on the enhanced image to obtain the enhanced Laplacian residuals corresponding to the original image:

[0039]

[0040] Among them, en anced_image k This represents the enhanced image;

[0041] Calculate the difference residual between the original image and the enhanced image:

[0042]

[0043] Among them, L k Represents the Laplacian residual of the k-th layer of the original image; ΔL represents the Laplacian residual of the k-th layer of the enhanced image. k This represents the difference between the enhanced image and the original image at the k-th level Laplacian residual.

[0044] A fusion operation is performed based on residual information and enhanced image features to obtain an estimated depth image, including:

[0045] For each scale k, these residual features are weighted using channel attention and spatial attention mechanisms to obtain the weighted residual features.

[0046] The weighted residual features are concatenated, and the features are fused through a convolutional layer to obtain the fused residual feature representation.

[0047] The fused residual feature representation and enhanced image features are input into the decoder to generate an estimated depth image.

[0048] A monocular depth estimation system based on text guidance and multi-scale fusion includes:

[0049] The initial feature extraction module is used to obtain the visual features of the original image and the text features of the original text;

[0050] The feature enhancement module is used to map text features to the same dimension as the visual features of the original image to obtain processed text features. The processed text features are then introduced into the visual features of the original image to obtain semantically guided image features. Based on the visual features, the text features are enhanced to obtain enhanced text features. The enhanced text features are then introduced into the semantically guided image features through bidirectional cross-guidance to obtain the final enhanced image features and obtain the enhanced image.

[0051] The residual calculation module is used to perform multi-scale Laplacian residual calculation on the original image and the enhanced image to obtain residual information between the original image and the enhanced image. Based on the residual information and the enhanced image features, a fusion operation is performed to obtain the estimated depth image.

[0052] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described in this invention.

[0053] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in this invention.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] This invention discloses a monocular depth estimation method based on text guidance and multi-scale fusion. By dynamically fusing text features to enhance image features, richer semantic information is obtained, resulting in more accurate depth prediction. By capturing edge features of the image at different scales and aligning them with the enhanced image features in a multimodal manner, the model's ability to express local information and its robustness are further improved. This enables the model to understand both detailed local information and global scene layout simultaneously. This method can not only adapt to complex scenes but also improve the recovery effect of image depth boundaries and details. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0059] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0060] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0061] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0062] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0063] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0064] The present invention will now be described in further detail with reference to the accompanying drawings:

[0065] See Figure 1 This invention discloses a monocular depth estimation method based on text-guided and multi-scale fusion, aiming to achieve high-precision monocular depth estimation. A novel model framework—TFDepth (a text-guided multi-scale fusion network for monocular depth estimation)—is proposed. TFDepth utilizes text features to guide the representation and fusion of image features, thereby acquiring richer semantic information and helping the model better understand the details and global structure in the image. This framework fully leverages the complementary advantages of multimodal features, thus enhancing the accuracy and robustness of monocular depth estimation.

[0066] This paper utilizes a cross-semantic attention module (CSAM) and a multi-scale residual fusion module (MSRFM) to weightedly fuse image and text features, achieving accurate capture of image details and global structure. The encoded image and corresponding text descriptions are incorporated into the model as prior knowledge. The weights of image and text features are dynamically adjusted through the cross-semantic attention mechanism, enabling the model to effectively utilize this semantic information in different depth estimation tasks. To improve the model's depth estimation accuracy, this embodiment also employs a multi-scale residual fusion module, which captures fine-grained variations in the image at different resolutions to optimize the recovery of depth boundaries and details. Furthermore, the DINOv2 encoder used in the decoder further enhances the expressive power of image features, ensuring the accuracy of the depth estimation results.

[0067] The DINOv2 encoder described is the encoder disclosed in reference Oquab M, Darcet T, Moutakanni T, et al. Dinov2: Learning robust visual features without supervision[J]. arXiv preprint arXiv:2304.07193,2023.

[0068] The model structure of this method includes a pre-trained encoder, a cross-semantic attention module (CSAM), a multi-scale residual fusion module (MSRFM), and an image decoder.

[0069] The encoder section is used to extract rich feature information from the input image and text. For image feature extraction, inspired by *Depth_Anything* (from Yang L, Kang B, Huang Z, et al. *Depth anything: Unleashing the power of large-scale unlabeled data* [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 10371-10381), this embodiment transfers the powerful semantic capabilities of the pre-trained DINOv2 model to the depth estimation model of this embodiment, using both frozen and unfrozen DINOv2 encoders to fully utilize their powerful semantic capabilities and ensure efficient extraction of image features. For text information processing, this embodiment uses the text encoder of the pre-trained visual-language model CLIP. CLIP provides a shared latent space, which can effectively incorporate linguistic priors into image features. Specifically, given a text description t = {t1, t2, ...}, this embodiment uses CLIP's text encoder to transform it into a feature vector of size b × 1024, where b is the batch size. These text features are then dynamically weighted and fused with image features through a cross-semantic attention module (CSAM) to ensure that the information from the two modalities can effectively complement each other at different levels.

[0070] CSAM employs a multi-head cross-attention mechanism, dynamically adjusting the weighting of image and text features through their interaction to enhance semantic information at different scales. It establishes effective connections between image and text features at different levels, ensuring deep complementarity of information and thus improving the accuracy of depth estimation.

[0071] The decoder effectively integrates features and details from multiple scales through a multi-scale residual fusion module (MSRFM), utilizing the Laplacian residual of the image to help recover details and depth boundaries. The key to this module is to effectively fuse the detail information of the original and enhanced images by calculating residual information at different scales, thereby achieving accurate depth estimation in the decoder.

[0072] Finally, after processing by the multi-scale residual fusion module, all fused multi-scale features are input into the LapDepth decoder (from the paper Song M, Lim S, Kim W. Monocular depth estimation using laplacian pyramid-based depth residuals[J]. IEEE transactions on circuits and systems for video technology, 2021, 31(11):4381-4393.). The final depth map is gradually recovered by recursively synthesizing depth residuals. The final depth map is generated. In the following sections, this embodiment will detail the function of each module and its role in the entire network, as well as how to improve the accuracy of depth estimation through training and optimization of the loss function.

[0073] Specifically, the following steps are included:

[0074] Step 1: The RGB image is first augmented to improve the model's generalization ability. Then, the augmented image and its corresponding text description are processed by their respective encoders for feature extraction. Specifically, the image uses a pre-trained DINOv2 encoder to extract multi-level visual features, while the text description uses a CLIP text encoder to obtain semantic features.

[0075] Specifically, it includes:

[0076] Step 1.1, Text Encoder

[0077] To incorporate linguistic priors into monocular depth estimation, this embodiment uses the text encoder of the pre-trained vision-language model CLIP. CLIP provides a shared latent space by default for visual and text embeddings. Specifically, given a text description t = {t1, t2...}, this embodiment encodes it using the CLIP text encoder to obtain a feature vector of size b × 1024, where b is the batch size. The generated text embedding captures rich semantic information and acts as a bridge to align the text description with visual features, facilitating a more context-aware depth estimation process.

[0078] Step 1.2: Extracting image features using both frozen and non-frozen encoders.

[0079] Inspired by Depth_Anything, this embodiment argues that incorporating high-level semantic information into the deep estimation model can significantly improve performance. Based on this idea, this embodiment transfers the powerful semantic capabilities of the pre-trained DINOv2 model to its deep estimation model, extracting image features using both frozen and unfrozen DINOv2 encoders. Furthermore, this embodiment also employs the feature alignment loss proposed in Depth_Anything.

[0080]

[0081] In this loss function, cos represents the measurement of the cosine similarity between two feature vectors, f is the feature extracted by the deep model S, and f′ is the feature from the frozen DINOv2 encoder.

[0082] Step 2: Image and text features are weighted and fused in a cross-semantic attention module to generate a joint feature representation. This fusion module employs a text-image feature fusion module based on a cross-attention mechanism. By dynamically fusing text features to enhance image features, it achieves more accurate depth prediction. The cross-attention mechanism dynamically weights image and text features at different scales, effectively combining information from both modalities and improving the model's understanding of scene semantics and the accuracy of depth prediction.

[0083] Specifically, the following steps are included:

[0084] In depth estimation tasks, multimodal features such as images and text have been shown to significantly improve model performance. To effectively fuse these multimodal features and improve depth estimation accuracy, this study proposes a cross-semantic attention module based on cross-attention. This module leverages the cross-semantic attention module to enhance image and text features at multiple scales, focuses on key information through a multi-scale residual fusion module, and finally further optimizes the feature representation through weighted residual connections.

[0085] Step 2.1 Text Feature Projection:

[0086] To effectively integrate text information into image features, this embodiment maps the text features extracted by the text encoder to the same dimension as the image features. The given text feature dimension is B×1024, where B is the batch size and 1024 is the dimension of each text feature. To enable the text features to be fused with the image features, this embodiment projects the text features onto the number of channels Ci at each scale using a multi-layer linear mapping. Specifically, this embodiment uses a projection matrix Proj. i To map text features t to feature spaces at each scale, we obtain text features t i :

[0087] t i =Proj i ·t (2)

[0088] Among them, t i ∈R B×Ci These are the text features after projection. (Proj) i This is the projection matrix corresponding to scale i, and its dimension is 1024×C. i C i This represents the number of channels in the image features of the i-th layer. This mapping ensures that the dimensionality of the text features matches the dimensionality of the image features, facilitating subsequent feature fusion.

[0089] Step 2.2 Cross-attention operation:

[0090] At each scale, this embodiment employs a multi-head cross-attention mechanism to interact with text features and image features, thereby incorporating semantic information from the text into the image features. Specifically, this embodiment uses text features t i As a query, image features f i As keys and values, they are enhanced through a cross-attention mechanism to generate semantically guided image features. To further enhance the relationship between image and text features, this embodiment employs a multi-head attention mechanism. In each head, the query, key, and value are mapped to different spaces through different linear transformations, and the attention value for each head is calculated. For the h-th head, the output is:

[0091]

[0092] in, It is a query, representing the text features after a linear transformation;

[0093] The key represents the image features after a linear transformation; It is a value that represents the image features after a linear transformation.

[0094] in, These are linear transformation matrices for the query, key, and value of each head, with dimensions Ci×Ci, which map the input features to a space that adapts to the attention mechanism. A h This represents the output of the h-th head. Finally, the enhanced image features are obtained by concatenating the outputs of all heads and performing a linear transformation.

[0095]

[0096] Where H is the number of heads, W oThis is the output linear transformation matrix. In this embodiment, the entire process is defined as cross-attention:

[0097]

[0098] Among them, T i The projected text features have a shape of B×1×C. i ; This represents the flattening of image features; This represents the image features enhanced by the cross-attention mechanism;

[0099] Through cross-attention operations, image features are enhanced and integrated with semantic information from the text. To better fuse image and text features, this embodiment proposes a bidirectional cross-guidance mechanism. The original image features also participate in the enhancement process of text features, forming a bidirectional cross-guidance:

[0100]

[0101] Finally, this embodiment will enhance the text features. Enhanced image features as query input As keys and values, further enhanced image features are obtained, resulting in...

[0102]

[0103] This process is performed at each scale to better capture the semantic relationships between images and text. Simultaneously, to further improve the accuracy of feature representation, the information between the original image features and the enhanced image features is balanced. At each scale, this embodiment fuses the enhanced image features with the original image features. Specifically, this embodiment fuses the enhanced image features... Compared with the original image features I i The features are then concatenated, and then further optimized using weighted residual connections to obtain the enhanced features:

[0104]

[0105] where w1,w2=Softmax(weights) (9)

[0106] Here, `weights` is a learnable parameter vector, initially set to random values ​​and learned during training. This module performs similar operations at multiple scales, progressively enhancing image and text features at each scale, which are then used in downstream tasks. Through this multi-scale feature fusion strategy, this embodiment can fully utilize image and text information at different scales, thereby improving the accuracy and robustness of depth estimation.

[0107] This module performs similar operations at multiple scales, progressively enhancing image and text features at each scale. Through this multi-scale feature fusion strategy, this embodiment can fully utilize image and text information at different scales, thereby improving the accuracy and robustness of depth estimation.

[0108] By dynamically fusing text features to enhance image features, more accurate depth prediction is achieved.

[0109] A multi-scale residual fusion module was designed to capture edge features of the image at different scales and align them with the enhanced image features in a multimodal manner, thereby further improving the model's ability to express local information and its robustness.

[0110] Step 3: The multi-scale residual fusion module further improves the accuracy of depth estimation by capturing subtle changes in the image. Inspired by LapDepth, this module utilizes Laplacian pyramid residuals to perform multi-scale processing on the image and the text-enhanced image, generating rich detail information. This detail information is then input into the decoder. The decoder employs a layer-by-layer reconstruction strategy similar to LapDepth, recursively synthesizing depth residuals to recover the image's depth information. Specifically, the Laplacian pyramid residuals capture high-frequency information and local structural changes in the image, helping the decoder better recover the details in the depth map.

[0111] Includes the following steps:

[0112] Original image I K Downsampling to different scales k, k∈{1,2,3,4}, yields multi-scale image features. At each scale, this embodiment calculates the Laplacian residual L. k This represents the change of image features at different resolutions. The formula is as follows:

[0113] L k =I k -Up(I k+1 (10)

[0114] Where k represents the level index of the Laplace pyramid; I K It is achieved by downsampling the original input image to 1 / 2. k -1The image obtained at the scale; Up(·) indicates an upsampling operation (using bilinear interpolation).

[0115] This embodiment also applies the enhanced image (processed via an image augmentation network) to this method to obtain richer detail information. The enhanced image is then processed using a similar multi-scale Laplacian residual calculation to obtain the enhanced Laplacian residual corresponding to the original image:

[0116]

[0117] To effectively combine the detail differences between the original and enhanced images, the difference residuals between the original and enhanced images were calculated:

[0118]

[0119] Where, ΔL k The difference residual captures the changes introduced during image enhancement, providing crucial information for subsequent feature fusion.

[0120] To further improve the accuracy of depth estimation, this embodiment proposes a multi-scale feature fusion method, which combines image features from multiple scales to further recover local details of the image. This embodiment uses the Laplacian residual L of the original image... k Enhanced Laplacian residuals of images The difference between the two residuals ΔL k The definition is as follows:

[0121] f1 k L represents the Laplacian residual of the k-th layer of the original image. k ;

[0122] f2 k Represents the Laplacian residual of the k-th layer of the enhanced image.

[0123] f3 k : Represents the residual ΔL of the difference between the two at the k-th layer. k .

[0124] For each scale k, these features are processed using a weighted approach of channel attention (CA) and spatial attention (SA):

[0125] Fi k = fi k ·CA(fi k )·SA(fi k ·CA(fi k )) i∈{1,2,3},k∈{1,2,3,4} (13)

[0126] CA(·) calculates the weight of each channel and weights the features by multiplying them channel by channel to focus on important channel information. The spatial attention mechanism SA(·) generates a spatial weight map by calculating the average and maximum values ​​at each location and applies it to the spatial dimension of the feature map, thus enabling the model to pay more attention to important regions in the image.

[0127] After applying channel and spatial attention mechanisms, this embodiment concatenates the three residual features weighted at each scale and fuses them through a convolutional layer to obtain the fuse. k The purpose of this operation is to integrate feature information from different sources and extract a more compact high-dimensional representation through convolutional layers. Building upon this, to further improve the accuracy of multi-scale feature fusion, this embodiment also performs a weighted average of the original image features, enhanced image features, and difference residual features at each scale. The weighted averaging operation ensures a reasonable balance among the three features during the fusion process, thereby preventing any single feature from dominating the fusion process. After obtaining the weighted averaged features, this embodiment processes them using a 1x1 convolution to obtain a more compact and refined feature representation f. res k This reduces the dimensionality of the features and extracts a more compact representation. Finally, this embodiment combines the two types of features using a weighted fusion method to obtain the final fused feature f for each scale. final k :

[0128] fuse k =Conv2d(concat(F1) k F2 k F3 k (14)

[0129] f res k ==Conv2d((f1 k +f2 k +f3 k ) / 3) (15)

[0130] f final k =αfuse k +β·f res k (16)

[0131] Here, α and β are learnable fusion weights, initially set to random values, used to balance the contributions of fused features and residual features. Through this weighted fusion strategy, this embodiment can effectively fuse original image features, enhanced image features, and differential residual features at multiple scales, thereby improving the accuracy and robustness of depth estimation. The module's effectiveness was verified in ablation experiments. Experimental results show that utilizing a multi-scale residual fusion strategy significantly improves the model's adaptability to complex scenes, particularly in depth boundary recovery and detail preservation.

[0132] Step 4: The decoder effectively recovers the overall depth structure and detail information of the image by synthesizing depth residuals layer by layer. This process not only improves the accuracy of depth estimation but also enhances the robustness of the model in complex and diverse scenes. By integrating multimodal information from images and text, and combining a cross-attention mechanism and a detail extraction module, the monocular depth estimation method based on text guidance and multi-scale fusion achieves significant performance improvements in monocular depth estimation tasks, demonstrating its broad potential and superiority in practical applications.

[0133] This embodiment also discloses the model training process:

[0134] Scale-invariant loss: This embodiment uses true values To minimize a supervised loss function, this embodiment employs scale-invariant depth loss to improve training stability across diverse scenarios. This loss function promotes scale invariance of the prediction results by calculating the logarithmic difference of the depth images.

[0135]

[0136] in Ω represents the image space, N e y is the number of effective pixels, y is the predicted depth, and γ is a scaling factor that controls the sensitivity to loss.

[0137] Feature alignment loss: This embodiment also introduces the feature alignment loss proposed in Depth_Anything to transfer powerful semantic capabilities to the depth estimation model of this embodiment:

[0138]

[0139] In this loss function, cos represents the measurement of cosine similarity between two feature vectors, f is the feature extracted by the deep model S, and f′ is the feature from the frozen DINOv2 encoder. This alignment loss encourages the deep estimation model to maintain semantic consistency with the pre-trained DINOv2 features, enhancing the model's ability to capture meaningful representations for accurate deep predictions.

[0140] Final loss function: The final loss function consists of a weighted combination of scale-invariant loss and feature alignment loss. Its specific form is:

[0141] L=αL SI +βL FA (19)

[0142] Here, α and β are the weighting coefficients for balancing L1 and L2 losses, respectively. After extensive experiments, their values ​​were set to 10 and 1, respectively.

[0143] This method also discloses an embodiment.

[0144] Step 1: Process the dataset (NYU, KITTI)

[0145] This embodiment evaluates the method on two different datasets: NYU Depth V2 for indoor scenes and KITTI for outdoor environments.

[0146] The NYU Depth V2 dataset contains 120K pairs of RGB and depth images, captured using a Microsoft Kinect sensor in 464 indoor scenes at a resolution of 640×480 pixels. This embodiment follows the dataset partitioning method proposed in existing literature (Lee J H, Han MK, Ko DW, et al. From big to small: Multi-scale local planar guidance for monocular depth estimation[J]. arXiv preprint arXiv:1907.10326,2019.), containing 24,231 training images and 654 test images.

[0147] The existing literature mentioned is David Eigen, Christian Puhrsch, and Rob Fergus. Depth mapprediction from a single image using a multi-scale deep network. Advances in neural information processing systems, 27, 2014, 4, 5, 6, 1. The KITTI dataset contains various road environment images collected from autonomous driving scenarios, with a resolution of 1242×375 pixels. For performance comparison, this embodiment adopts the partitioning strategy proposed by Eigen et al. in the literature David Eigen, Christian Puhrsch, and Rob Fergus. Depth mapprediction from a single image using a multi-scale deep network. Advances in neural information processing systems, 27, 2014, 4, 5, 6, 1. The test set includes 697 images selected from 29 scenarios, while the training set consists of 23,488 images from the remaining 32 scenarios. Based on the methods in existing literature (Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 4009–4018, 2021. 2, 3, 5, 6, 7, 8, 1. and Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and PingTan. Neural window fully-connected crfs for monocular depth estimation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 3916-3925, 2022. 2, 5, 6, 8), after removing invalid ground truth samples, this embodiment finally obtained 652 valid test images.During the testing phase, in accordance with the guidelines of the KITTI dataset, this embodiment limits the maximum value of the predicted output to the order of 80 meters.

[0148] Step 2, Experimental setup (evaluation metrics, hyperparameters)

[0149] Step 2.1 Training

[0150] The proposed method is implemented using the PyTorch framework. In the experiments, the text encoder part adopts the same settings as in existing literature (Zeng Z, Wang D, Yang F, et al. Wordepth: Variational language prior for monocular depth estimation[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2024:9708-9719). Specifically, this embodiment uses the CLIP ResNet-50 version as the text encoder, responsible for extracting text features. To improve computational efficiency, the image title generation part uses ExpansionNet-v2. This architecture effectively extracts text information and generates accurate descriptions for the image, thus contributing to the depth estimation task in this embodiment.

[0151] Hyperparameters: In the optimization process, this embodiment uses the AdamW optimizer and does not use weight decay. This embodiment sets the same learning rate of 1×10⁻⁴ for both the encoder and decoder, and sets the weight decay to 0. The optimizer's eps parameter is set to 1×10⁻³ to ensure numerical computation stability. Under this scheduler, the model is trained for 50 epochs on the KITTI and NYU Depth V2 datasets. The weight α of the scale-invariant loss is set to 10, and the weight of the feature loss is set to 1. During the training phase, to reduce overfitting and improve the model's generalization ability, this embodiment employs online data augmentation techniques, with experimental settings following the practices in WorDepth. Specifically, this embodiment implements various random image transformations on the NYU Depth V2 and KITTI datasets, including adjusting image brightness, gamma values, and color intensity. Simultaneously, this embodiment randomly performs horizontal flipping and rotation of images to enhance image diversity. These data augmentation methods effectively increase the diversity of the training set and improve the model's adaptability to different scenes.

[0152] Step 2.2 Evaluation Indicators

[0153] This embodiment employs the evaluation metrics proposed by Eigen et al. (specifically from the literature D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” in Proc. Adv. Neural Inf. Process. Syst., Dec. 2014, pp. 2366–2374.), which are widely used in the performance evaluation of monocular depth estimation. Specifically, this embodiment uses the mean absolute relative error (Abs Rel), root mean square error (RMSE), absolute error in log space (log10), root mean square logarithmic error (RMSElog), and threshold accuracy (δi) to quantitatively evaluate the method of this study.

[0154] Step 2.3 Quantitative Results - Visualization

[0155] Experimental results on the NYU Depth V2 dataset in this embodiment demonstrate that, compared to existing depth estimation techniques, the model in this embodiment shows significant improvements across all evaluation metrics. In particular, the TFDepth method excels in threshold accuracy δ<1.25, a metric that measures the proportion of deviation between the predicted result and the true value within a specific range. Specifically, when using vit-l as the backbone, the TFDepth method achieves an accuracy of 0.969 for δ<1.25, significantly outperforming other state-of-the-art methods. Even when using vit-s as the backbone, the TFDepth method outperforms many existing methods, further validating the superiority of the method in this embodiment. The experimental results of this embodiment demonstrate that the TFDepth method improves across all evaluation metrics, proving its significant progress in depth estimation accuracy, especially in handling complex scale estimation tasks. By leveraging text modality to enhance image features, this embodiment can more accurately capture scene details, thus making the depth estimation results closer to the true value. This improvement further validates the advantages of the model in various depth estimation tasks.

[0156] The experimental results of the method in this embodiment on the KITTI dataset were obtained using Eigen Split (from the literature David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems, 27, 2014, 4, 5, 6, 1). Compared with existing depth estimation techniques, the TFDepth method also achieves state-of-the-art performance. Similar to the NYU Depth V2 dataset, the TFDepth method improves accuracy at a threshold of δ < 1.25, but the relative performance improvement on this metric is not as significant as on NYU Depth V2. This difference may stem from the wider range of object sizes and shapes in outdoor scenes, where objects of the same category may have different sizes and shapes. For example, the term "car" may refer to a sedan, coupe, or hatchback—they differ in size (e.g., a coupe is generally smaller than a sedan) and shape (a hatchback has a higher roof and a connected trunk). While textual descriptions offer flexibility in balancing generality and specificity when used as prior information, over-reliance on textual features can be counterproductive when descriptions are vague. The TFDepth method addresses these issues by introducing a cross-attention mechanism to enhance image features with textual features. Specifically, textual features dynamically adjust the weights of image features through the CSAM module, enabling the model to more accurately identify and distinguish objects of different shapes. This multimodal feature enhancement strategy not only improves the accuracy of depth estimation but also allows the model to maintain high performance in complex outdoor scenes. Experimental results demonstrate that the positive effects of prior information significantly outweigh the potential negative impacts when dealing with highly diverse and complex outdoor objects, further validating the advantages of the model in various depth estimation tasks.

[0157] Step 2.4 Ablation

[0158] To evaluate the contribution of the fusion module and detail extraction module to the model performance, this embodiment conducted ablation experiments on the NYU Depth V2 dataset, using vit-s as the backbone. Specifically, this embodiment set up the following three experimental configurations: (1) removing the cross-semantic attention module and the multi-scale residual fusion module; (2) removing the multi-scale residual fusion module; and (3) using the complete model. Experimental results show that the complete model performs best on all evaluation metrics, significantly outperforming the configuration with some modules removed. This indicates that the cross-semantic attention module effectively combines text and image features through a cross-attention mechanism, significantly improving the accuracy of depth estimation. Meanwhile, the multi-scale residual fusion module plays a key role in capturing subtle changes in images, and its presence further enhances the model's adaptability to complex scenes. The removal of the cross-semantic attention module and the multi-scale residual fusion module led to a significant performance decrease, indicating that these two modules play an indispensable role in the overall model. While removing only the multi-scale residual fusion module also resulted in a performance decrease, its impact was slightly less than when all modules were removed. This further verifies the importance of the multi-scale residual fusion module in improving the accuracy of depth estimation. In summary, the synergistic effect of the cross-semantic attention module and the multi-scale residual fusion module significantly improves the model's depth estimation performance in complex and diverse scenarios, verifying the effectiveness and superiority of the model design in this embodiment.

[0159] This invention also discloses a monocular depth estimation system based on text guidance and multi-scale fusion, comprising:

[0160] The initial feature extraction module is used to obtain the visual features of the original image and the text features of the original text;

[0161] The feature enhancement module is used to map text features to the same dimension as the visual features of the original image to obtain processed text features. The processed text features are then introduced into the visual features of the original image to obtain semantically guided image features. Based on the visual features, the text features are enhanced to obtain enhanced text features. The enhanced text features are then introduced into the semantically guided image features through bidirectional cross-guidance to obtain the final enhanced image features and obtain the enhanced image.

[0162] The residual calculation module is used to perform multi-scale Laplacian residual calculation on the original image and the enhanced image to obtain residual information between the original image and the enhanced image. Based on the residual information and the enhanced image features, a fusion operation is performed to obtain the estimated depth image.

[0163] A schematic diagram of a terminal device according to an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the various device embodiments described above.

[0164] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0165] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory.

[0166] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0167] The memory can be used to store the computer program and / or module. The processor implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.

[0168] If the modules / units integrated into the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0169] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A monocular depth estimation method based on text guidance and multi-scale fusion, characterized in that, Includes the following steps: Obtain the visual features of the original image and the textual features of the original text; The text features are mapped to the same dimension as the visual features of the original image to obtain the processed text features. The processed text features are then introduced into the visual features of the original image to obtain image features with semantic guidance. The text features are then enhanced based on the visual features to obtain enhanced text features. Enhanced text features are introduced into semantically guided image features through bidirectional cross-guidance to obtain the final enhanced image features and acquire the enhanced image; Multi-scale Laplacian residual calculation is performed on the original image and the enhanced image to obtain the residual information between the original image and the enhanced image. Based on the residual information and the enhanced image features, a fusion operation is performed to obtain the estimated depth image. The text features are mapped to the same dimension as the image information, and the processed text features ti are obtained by the following formula: in, Let represent the projection matrix corresponding to scale i, with dimensions 1024×Ci; Ci is the number of channels of the image features in the i-th layer. The process involves introducing the processed text features into the visual features of the original image to obtain semantically guided image features. The text features are then enhanced based on the visual features of the original image to obtain enhanced text features, including: Using text features ti as the query and image features fi as the key and value, an enhancement mechanism is used to generate semantically guided image features; The query, key, and value are subjected to multiple different linear transformations, and then the attention of multiple heads is calculated separately. The outputs of the multiple heads are then concatenated to obtain the enhanced image feature Ii enhanced; The enhancement of text features based on visual features of the original image yields enhanced text features, including: in, This represents the text features after projection; This represents the flattening of image features; The process of introducing enhanced text features into semantically guided image features through bidirectional cross-guidance to obtain the final enhanced image features includes: Using the enhanced text feature Tii as the query input and the enhanced image feature Ii as the key and value, we obtain the further enhanced image feature Ii final. in, Indicates enhanced image features; Indicates enhanced text features; The enhanced image features and the original image features are concatenated, and then a 2D convolution operation is performed. The feature representation is further optimized using weighted residual connections to obtain the final enhanced features.

2. The monocular depth estimation method based on text guidance and multi-scale fusion according to claim 1, characterized in that, The enhanced image feature Ii enhanced is represented by the following formula: Where H represents the number of heads, W o Represents the linear transformation matrix of the output; Ai represents the output result of the head; The entire process is defined as cross-attention: in, Represents the features of the projected text; This represents the flattening of image features.

3. The monocular depth estimation method based on text guidance and multi-scale fusion according to claim 1, characterized in that, The process of obtaining residual information between the original image and the enhanced image includes: Calculate the Laplacian residual of the original image : Multi-scale Laplacian residuals are calculated on the enhanced image to obtain the enhanced Laplacian residuals corresponding to the original image: in, This represents the enhanced image; Calculate the difference residual between the original image and the enhanced image: in, Represents the Laplacian residual of the k-th layer of the original image; ΔL represents the Laplacian residual of the k-th layer of the enhanced image. k This represents the difference between the Laplacian residuals of the enhanced image and the original image at the k-th layer scale.

4. The monocular depth estimation method based on text guidance and multi-scale fusion according to claim 3, characterized in that, A fusion operation is performed based on residual information and enhanced image features to obtain an estimated depth image, including: For each scale k, these residual features are weighted using channel attention and spatial attention mechanisms to obtain the weighted residual features. The weighted residual features are concatenated, and the features are fused through a convolutional layer to obtain the fused residual feature representation. The fused residual feature representation and enhanced image features are input into the decoder to generate an estimated depth image.

5. A monocular depth estimation system based on text guidance and multi-scale fusion, used to implement the method of claim 1, characterized in that, include: The initial feature extraction module is used to obtain the visual features of the original image and the text features of the original text; The feature enhancement module is used to map text features to the same dimension as the visual features of the original image to obtain processed text features. The processed text features are then introduced into the visual features of the original image to obtain semantically guided image features. Based on the visual features, the text features are enhanced to obtain enhanced text features. The enhanced text features are then introduced into the semantically guided image features through bidirectional cross-guidance to obtain the final enhanced image features and obtain the enhanced image. The residual calculation module is used to perform multi-scale Laplacian residual calculation on the original image and the enhanced image to obtain residual information between the original image and the enhanced image. Based on the residual information and the enhanced image features, a fusion operation is performed to obtain the estimated depth image.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-4.

Images