A monocular depth estimation method and device fusing images rely on dynamic prompt tokens and CLIP

Abstract

This invention belongs to the field of monocular depth estimation technology, specifically disclosing a method and apparatus for monocular depth estimation that fuses image-dependent dynamic cue tokens and CLIP (Text-Based Image Coding). The method includes: extracting multi-scale visual features of the input image through a CLIP image encoder and aggregating them to obtain global image features; generating dynamic cue tokens based on the global image features and static learnable tokens through multi-head cross-attention interaction and nonlinear mapping; concatenating the static learnable tokens and dynamic cue tokens and performing fusion encoding through an extended CLIP text encoder to generate cross-modal text embeddings; and outputting a high-precision depth map through depth binning classification based on the similarity matching between the cross-modal text embeddings and the input image features. This invention achieves high-precision monocular depth estimation with strong generalization ability and robustness, significantly improving the depth prediction accuracy for distant objects, and accurately distinguishing the distance between vehicles in adjacent lanes, meeting the obstacle avoidance requirements of autonomous driving scenarios.

Description

Technical Field

[0001] This invention relates to the field of monocular depth estimation technology, and more particularly to a monocular depth estimation method and apparatus that fuses image-dependent dynamic cue tokens and CLIP. Background Technology

[0002] With the rapid development of computer vision technology, monocular depth estimation (MDE) has broad application prospects in fields such as autonomous driving (obstacle avoidance), robot navigation (path planning), and VR / AR (immersive scene construction). Monocular depth estimation aims to recover the three-dimensional structural information of a scene from a single RGB image. Compared with binocular or multi-view methods, monocular methods have advantages such as low hardware cost and simple deployment, but they also face greater technical challenges.

[0003] Existing monocular depth estimation techniques are mainly divided into three categories:

[0004] Traditional geometric cue methods rely on manually designed rules (such as "objects appear larger when closer and smaller when farther away" and "occluded objects appear closer") to infer depth through visual cues in images. However, they have poor generalization ability and exhibit significant errors in scenes with weak textures, transparent materials, or complex lighting and shadows, making them unsuitable for practical applications.

[0005] Supervised deep learning methods train networks (such as MonoDepth and AdaBins) on large-scale labeled deep datasets to directly learn the mapping relationship between images and depth. Although they have high accuracy, they rely on massive amounts of labeled data and have insufficient generalization ability in open scenes outside the training set (such as indoor-outdoor switching and special weather conditions).

[0006] Cross-modal transfer method: Utilizing the cross-modal alignment capability of the visual-language pre-trained model CLIP (Contrastive Language-Image Pre-training) (learning semantic associations between images and text through large-scale image-text pairing data), zero-shot transfer is achieved. CLIP does not require retraining for deep tasks; it can adapt to depth estimation using only text prompts, providing a new path for open-world scenarios.

[0007] In recent years, with the emergence of contrastive learning language-image pre-trained models (CLIP), researchers have begun to explore how to leverage the rich semantic prior knowledge inherent in CLIP models to improve the performance of monocular depth estimation. CLIP models, pre-trained on large-scale image-text pairs, establish powerful cross-modal understanding capabilities, providing a new approach to monocular depth estimation tasks.

[0008] In existing technologies, monocular depth estimation methods based on CLIP are mainly divided into two categories: one is to use CLIP's text encoder and hand-designed text cues for depth estimation; the other is to introduce learnable text tokens to replace hand-designed cues and optimize token parameters through end-to-end training.

[0009] However, existing CLIP-based monocular depth estimation methods still have the following technical problems:

[0010] First, in existing CLIP-based monocular depth estimation schemes, manual text prompts introduce subjective bias, leading to "quantization artifacts" in depth estimation and preventing the formation of smooth, continuous depth maps, thus affecting estimation accuracy. While introducing learnable tokens can alleviate this problem to some extent, these tokens have fixed parameters after training, becoming "static tokens" that cannot be dynamically adjusted based on the specific content of the input image, resulting in insufficient robustness of depth estimation across different scenes.

[0011] Secondly, existing methods only establish a fixed mapping between tokens and image features through loss function optimization during the training phase. The inference phase lacks a dynamic interaction mechanism between image features and text tokens, failing to capture the deep semantics of local details in the image. This static association method limits the model's adaptability to complex scenes, especially performing poorly when dealing with images outside the training set distribution.

[0012] Finally, existing methods rely on simple cross-modal similarity calculations to correlate tokens with depth ranges, failing to consider the hierarchical interaction between multi-scale image features and tokens. This makes it difficult to accurately model the complex nonlinear relationship between images and depth, limiting further refinement of depth estimation. Especially when dealing with scenes with rich textures and complex geometries, existing methods often fail to accurately recover detailed information, leading to blurred depth map edges or structural distortion. Summary of the Invention

[0013] The present invention aims to at least partially solve one of the aforementioned technical problems in the prior art.

[0014] A first aspect of an exemplary embodiment of the present invention provides a monocular depth estimation method that fuses image-dependent dynamic cue tokens and CLIPs, the method comprising:

[0015] Step S100: Extract multi-scale visual features from the input image using the CLIP image encoder and aggregate them to obtain global image features;

[0016] Step S200: Based on the global image features and the static learnable token, a dynamic cue token is generated through multi-head cross-attention interaction and nonlinear mapping. The dynamic cue token is used to adapt to the input image content.

[0017] Step S300: After concatenating the static learnable token and the dynamic cue token, the static learnable token is fused and encoded by the extended CLIP text encoder to generate a cross-modal text embedding. The extended CLIP text encoder fuses the depth prior of the static learnable token and the image adaptation information of the dynamic cue token.

[0018] Step S400: Based on the similarity matching between the cross-modal text embedding and the input image features, output a high-precision depth map through depth binning classification.

[0019] According to an embodiment of the present invention, in step S100, the extraction of multi-scale visual features of the input image by the CLIP image encoder and the aggregation to obtain global image features include:

[0020] Input image preprocessing involves standardizing a single RGB image whose depth is to be estimated, so that the pixel values ​​of the image are normalized to the range [0, 1] to meet the input requirements of the CLIP image encoder;

[0021] Multi-scale feature extraction involves inputting the preprocessed image into a pre-trained CLIP image encoder, and outputting multi-scale image features through multi-layer neural networks or convolution operations.

[0022] Global feature aggregation involves performing spatial average pooling on multi-scale image features to calculate the mean of each feature channel, thereby obtaining the global image features.

[0023] According to an embodiment of the present invention, in step S200, the generation of a dynamic cue token based on the global image features and the static learnable token, through multi-head cross-attention interaction and nonlinear mapping, wherein the dynamic cue token is used to adapt to the input image content, includes:

[0024] Construct a set of original static tokens for static learnable text, wherein the original static tokens are initialized as normal distribution vectors with a mean of zero and a standard deviation of 0.02;

[0025] All the image features are mapped to the Key and Value matrices of the attention mechanism through a learnable weight matrix, respectively, wherein the token is mapped to the Query matrix through the learnable weight matrix;

[0026] The similarity between the Query matrix and the Key matrix is ​​calculated using the scaled dot product attention formula to obtain the attention weight matrix. ;

[0027] The attention weight matrix With the Value matrix Weighted summation is performed to obtain attention-enhanced features;

[0028] The attention enhancement feature is residually connected to the original static token, and then the residually connected attention enhancement feature is input into the KAN network for nonlinear mapping to generate the dynamic cue token.

[0029] According to an embodiment of the present invention, in step S300, after concatenating the static learnable token and the dynamic cue token, the two are fused and encoded using an extended CLIP text encoder to generate a cross-modal text embedding. The extended CLIP text encoder fuses the depth prior of the static learnable token and the image adaptation information of the dynamic cue token, including:

[0030] The static learnable token and the dynamic cue token are concatenated along the token dimension to form an extended input;

[0031] Add a linear projection layer to map the extended input to a dimension compatible with the CLIP text encoder;

[0032] The projected features are input into the multi-layer Transformer structure of the CLIP text encoder, and the semantic associations between tokens are captured through a self-attention mechanism to generate the cross-modal text embedding.

[0033] According to an embodiment of the present invention, in step S400, the step of outputting a high-precision depth map through depth binning classification based on the similarity matching between the cross-modal text embedding and the input image features includes:

[0034] The continuous depth range is evenly divided into M depth bins, each bin corresponding to a specific depth interval, wherein the depth range is 0 to 100 meters.

[0035] Calculate the cosine similarity between the input image features and the cross-modal text embedding, and obtain the probability distribution of each bin through Softmax normalization;

[0036] The median of the depth interval corresponding to the bin with the highest probability is taken as the depth estimate of the pixel.

[0037] A second aspect of an exemplary embodiment of the present invention provides a monocular depth estimation apparatus that fuses image-dependent dynamic cue tokens and CLIP, the apparatus comprising:

[0038] The image feature extraction module is used to extract multi-scale visual features of the input image through the CLIP image encoder and aggregate them to obtain global image features.

[0039] A dynamic token generation module is used to generate dynamic prompt tokens based on the global image features and static learnable tokens through multi-head cross-attention interaction and nonlinear mapping. The dynamic prompt tokens are used to adapt to the input image content.

[0040] A cross-modal coding module is used to concatenate a static learnable token and a dynamic cue token, and then perform fusion encoding through an extended CLIP text encoder to generate a cross-modal text embedding. The extended CLIP text encoder fuses the depth prior of the static learnable token and the image adaptation information of the dynamic cue token.

[0041] The depth estimation module is used to output a high-precision depth map by depth binning classification based on the similarity matching between the cross-modal text embedding and the input image features.

[0042] According to one embodiment of the present invention, the image feature extraction module is used for:

[0043] Input image preprocessing involves standardizing a single RGB image whose depth is to be estimated, so that the pixel values ​​of the image are normalized to the range [0, 1] to meet the input requirements of the CLIP image encoder;

[0044] Multi-scale feature extraction involves inputting the preprocessed image into a pre-trained CLIP image encoder, and outputting multi-scale image features through multi-layer neural networks or convolution operations.

[0045] Global feature aggregation involves performing spatial average pooling on multi-scale image features to calculate the mean of each feature channel, thereby obtaining the global image features.

[0046] According to one embodiment of the present invention, the dynamic token generation module is used for:

[0047] Construct a set of original static tokens for static learnable text, wherein the original static tokens are initialized as normal distribution vectors with a mean of zero and a standard deviation of 0.02;

[0048] All the image features are mapped to the Key and Value matrices of the attention mechanism through a learnable weight matrix, respectively, wherein the token is mapped to the Query matrix through the learnable weight matrix;

[0049] The similarity between the Query matrix and the Key matrix is ​​calculated using the scaled dot product attention formula to obtain the attention weight matrix. ;

[0050] The attention weight matrix With the Value matrix Weighted summation is performed to obtain attention-enhanced features;

[0051] The attention enhancement feature is residually connected to the original static token, and then the residually connected attention enhancement feature is input into the KAN network for nonlinear mapping to generate the dynamic cue token.

[0052] According to one embodiment of the present invention, the cross-modal coding module is used for:

[0053] The static learnable token and the dynamic cue token are concatenated along the token dimension to form an extended input;

[0054] Add a linear projection layer to map the extended input to a dimension compatible with the CLIP text encoder;

[0055] The projected features are input into the multi-layer Transformer structure of the CLIP text encoder, and the semantic associations between tokens are captured through a self-attention mechanism to generate the cross-modal text embedding.

[0056] According to one embodiment of the present invention, the depth estimation module is used for:

[0057] The continuous depth range is evenly divided into M depth bins, each bin corresponding to a specific depth interval, wherein the depth range is 0 to 100 meters.

[0058] Calculate the cosine similarity between the input image features and the cross-modal text embedding, and obtain the probability distribution of each bin through Softmax normalization;

[0059] The median of the depth interval corresponding to the bin with the highest probability is taken as the depth estimate of the pixel.

[0060] The technical advantages of this invention are as follows:

[0061] The present invention discloses a monocular depth estimation method and apparatus that fuses image-dependent dynamic cue tokens and CLIP. The method includes: extracting multi-scale visual features of the input image using a CLIP image encoder and aggregating them to obtain global image features; generating dynamic cue tokens based on the global image features and static learnable tokens through multi-head cross-attention interaction and nonlinear mapping, the dynamic cue tokens being used to adapt to the content of the input image; concatenating the static learnable tokens and dynamic cue tokens, and then performing fusion encoding through an extended CLIP text encoder to generate cross-modal text embeddings; the extended CLIP text encoder then fuses the depth prior of the static learnable tokens with the image adaptation information of the dynamic cue tokens; and outputting a high-precision depth map based on the similarity matching between the cross-modal text embeddings and the input image features through depth binning classification. This invention utilizes the cross-modal capability of the CLIP model, combined with image-dependent dynamic cue tokens, to achieve high-precision monocular depth estimation with strong generalization ability and robustness. In outdoor road scenes, the depth estimation errors for vehicles, pedestrians, and buildings are significantly reduced, especially the depth prediction accuracy for distant objects is significantly improved, accurately distinguishing the distance between vehicles in adjacent lanes, meeting the obstacle avoidance requirements of autonomous driving scenarios. Attached Figure Description

[0062] Figure 1 This is a flowchart of the steps of a monocular depth estimation method that integrates image-dependent dynamic cue tokens and CLIP according to the present invention.

[0063] Figure 2 This is a schematic diagram of the structure of each module in a monocular depth estimation method that integrates image-dependent dynamic cue tokens and CLIP according to the present invention.

[0064] Figure 3 This is a general framework diagram of a monocular depth estimation method that integrates image-dependent dynamic cue tokens and CLIP according to the present invention. Detailed Implementation

[0065] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0066] Example 1

[0067] like Figures 1 to 3 As shown, an exemplary embodiment of the present invention provides a monocular depth estimation method that fuses image-dependent dynamic cue tokens and CLIPs, including the following steps:

[0068] Step S100: Extract multi-scale visual features from the input image using the CLIP image encoder and aggregate them to obtain global image features.

[0069] Specifically, the input image is first preprocessed by standardizing the single RGB image (with a resolution of H×W×3, where H and W are the image height and width) to be estimated so that the pixel values ​​of the image are normalized to the range [0, 1] to meet the input requirements of the CLIP image encoder.

[0070] Then, multi-scale feature extraction is performed. The preprocessed image is input into a pre-trained CLIP image encoder, and multi-layer neural networks or convolutional operations are used to output multi-scale image features. The pre-trained CLIP image encoder can be a ResNet-50 or ViT-B / 32-enabled image encoder, with ViT-B / 32 being preferred to improve global feature capture capabilities. The image encoder outputs multi-scale image features through multi-layer Transformer or convolutional operations. Where B is the batch size (B=1 during inference), HW is the total number of pixels after the image is flattened, and C is the feature dimension (CLIP defaults to C=512).

[0071] Finally, global feature aggregation is performed on multi-scale image features. Spatial average pooling is performed to calculate the mean of each feature channel, thus obtaining the global image features. This feature can comprehensively reflect the overall scene semantics of the image (such as indoor / outdoor, object distribution, etc.), providing image content guidance for dynamic token generation.

[0072] Step S200: Based on global image features and static learnable tokens, a dynamic cue token is generated through multi-head cross-attention interaction and nonlinear mapping. The dynamic cue token is used to adapt to the input image content.

[0073] Specifically, the first step is to construct a set of original static tokens from statically learnable text. Where N is the number of learnable context tokens (the number N is determined by the text prompt template; the 1o2d template (3 learnable context tokens p0-p2) corresponds to N=3, and the 4o4d template (8 learnable context tokens p0-p7) corresponds to N=8); 1o2d has a structure of '1 learnable context token + 1 auxiliary semantic word object_tkn + 2 learnable context tokens + 1 fixed depth_tkn', and 4o4d has a structure of '4 learnable context tokens + 1 auxiliary semantic word object_tkn + 4 learnable context tokens + 1 fixed depth_tkn'. The original static tokens are initialized as a normally distributed vector with a mean of zero and a standard deviation of 0.02 to ensure the stability of the training process.

[0074] Then all image features Through learnable weight matrix and Mapped to the Key matrix of the attention mechanism and Value Matrix Among them, tokens Through learnable weight matrix Mapped to Query Matrix .

[0075] Next, the similarity between the Query matrix and the Key matrix is ​​calculated using the scaled dot product attention formula, resulting in the attention weight matrix A. The specific calculation formula is as follows: ,in, K is a scaling factor used to avoid gradient vanishing due to excessively high feature dimensions. T This represents the transpose of matrix K. The Softmax function is used to normalize the weights to the interval [0, 1].

[0076] Then, the attention weight matrix A and the value matrix V are weighted and summed to obtain the attention-enhanced features. The specific calculation formula is as follows: .

[0077] Finally, attention-enhancing features will be implemented. With the original static token Perform residual join (element-by-element addition), the formula is as follows: To preserve the underlying deep semantic information of the static token, the residual concatenation features are then... The input is a Kolmogorov-Arnold network (KAN), configured with 8 B-spline basis functions, a hidden layer width of 64, and approximately 0.525M trainable parameters. This network precisely transforms image features into vectors in the text token space through nonlinear mapping, generating the final image-dependent dynamic cue tokens. .

[0078] Compared to traditional MLP networks, KAN networks have a stronger ability to approximate nonlinear functions and can more accurately capture the complex relationship between images and text under the same parameter scale. Experiments have shown that on the NYU dataset, it can reduce the Abs Rel error by 0.002 and the RMSE error by 0.005.

[0079] KAN network, or Kolmogorov-Arnold network, is a nonlinear mapping network based on B-spline basis functions, which has a stronger function approximation ability than traditional MLP.

[0080] It should be noted that in some embodiments, multi-head cross-attention can be replaced with single-head attention or self-attention mechanism to reduce the amount of computation (the single-head attention parameter scale is reduced by about 1 / M, where M is the number of attention heads). Although the robustness in complex scenarios is slightly reduced (RMSE improvement ≤ 0.03), it can still achieve the adaptation of dynamic tokens to image content and meet the deployment requirements of low computing power devices.

[0081] In addition, in some embodiments, the KAN network can be replaced with a 2-layer MLP (512-512-512, ReLU activation), with the parameter size controlled at around 0.525M, consistent with KAN. Although the AbsRel error on the NYU dataset increases from 0.242 to 0.244 and the RMSE increases from 0.862 to 0.867, it is still significantly better than the existing technology and is suitable for scenarios with requirements for deployment complexity.

[0082] Step S300: After concatenating the static learnable token and the dynamic cue token, perform fusion encoding through the extended CLIP text encoder to generate a cross-modal text embedding. The extended CLIP text encoder fuses the depth prior of the static learnable token and the image adaptation information of the dynamic cue token.

[0083] Specifically, firstly, statically learnable tokens... With dynamic prompt tokens Concatenate along the token dimension to form an expanded input. , where 2N is the total number of tokens after concatenation, ensuring that the encoder receives both static basic semantics and dynamic image adaptation semantics simultaneously.

[0084] Then add a linear projection layer to expand the input. Mapping to a dimension compatible with the CLIP text encoder (keeping C=512 constant), the projection formula is as follows:

[0085] in For learnable weight matrix, As a learnable bias vector, this projection layer ensures that the concatenated input can be successfully integrated into subsequent Transformer structures.

[0086] Finally, the projected features are... The CLIP text encoder's multi-layered Transformer structure (12 layers by default) is used to capture semantic relationships between tokens through a self-attention mechanism, generating the final cross-modal text embedding. This embedding integrates the deep prior of static tokens and the image adaptation information of dynamic tokens, significantly improving its correlation with image features.

[0087] Step S400: Based on the similarity matching between cross-modal text embedding and input image features, output a high-precision depth map through depth binning classification.

[0088] Specifically, the continuous depth range is first uniformly divided into M depth bins, each bin corresponding to a specific depth interval, where the depth range is from 0 to 100 meters. That is, the continuous depth range (e.g., 0-100 meters) is uniformly divided into M depth bins, each bin corresponding to a specific depth interval (e.g., bin 1 corresponds to 0-2.5 meters, bin 2 corresponds to 2.5-5 meters, and so on).

[0089] It should be noted that depth binning divides a continuous depth range into multiple discrete intervals, determines the depth interval to which a pixel belongs through cross-modal matching, and thus achieves depth estimation.

[0090] Then, the multi-scale features output by the CLIP image encoder are... Text embedding with extended text encoder output Similarity is calculated using cosine similarity to measure the degree of association between the two features, with the similarity between each pixel feature and each text embedding. The formula is as follows:

[0091]

[0092] in Let be the feature vector of the i-th pixel. Let be the j-th text embedding vector, and let the subscript T denote the transpose. This represents the L2 norm.

[0093] The cosine similarity between the input image features and the cross-modal text embedding is calculated, and the probability distribution of each bin is obtained through Softmax normalization. Finally, the median of the depth interval corresponding to the bin with the highest probability is taken as the depth estimate of the pixel. That is, for each pixel's similarity vector... Perform Softmax normalization to obtain the probability distribution of each depth bin, and take the median of the depth interval corresponding to the bin with the highest probability as the depth estimate of that pixel.

[0094] In some embodiments, uniform binning can be replaced by adaptive binning (which dynamically adjusts the binning intervals according to the image depth distribution) to further improve the precision of depth estimation, especially in scenarios with large differences in depth range (such as those with both near and far objects).

[0095] In this example, the monocular depth estimation method fusing image-dependent dynamic cue tokens and CLIP effectively leverages the cross-modal characteristics of the CLIP model. Through an image-dependent dynamic cue token generator and extended CLIP text encoding / decoding, combined with an optimized depth binning classification process, dynamic and high-precision monocular depth estimation is achieved. The dynamic cue tokens adaptively adjust based on the input image content, enhancing the accuracy and robustness of the depth estimation. The depth binning classification method avoids the smoothing effect found in regression methods, preserving more depth details and edge information.

[0096] In a specific example, the specific parameters of the model training and the training process are as follows:

[0097] 1. Training dataset: The NYU-Depth V2 (indoor scene) and KITTI (outdoor scene) datasets are used, which include RGB images and corresponding true depth maps.

[0098] 2. Optimizer configuration: The AdamW optimizer is used, with a learning rate of 3.57 × 10⁻⁶. -4 The weight decay coefficient is 1×10 -4 The batch size is 16.

[0099] 3. Learning rate scheduling: The OneCycleLR scheduling strategy is adopted, with a maximum learning rate of 3.57 × 10⁻⁶. -4 The basic momentum is 0.85, the maximum momentum is 0.95, the div factor is 25, and the final div factor is 100.

[0100] 4. Loss Function: The L1 loss function is used to calculate the mean absolute error between the depth map output by the model and the true depth map. The formula is as follows:

[0101]

[0102] Where HW is the total number of pixels after the image is flattened. Let be the depth value of the i-th pixel in the b-th sample predicted by the model. This corresponds to the actual depth value.

[0103] 5. Training Environment and Rounds: Based on the PyTorch framework, a single NVIDIA V100 GPU was used for 25 rounds of training. During training, data augmentation techniques such as random cropping (NYU dataset cropped to 416×544, KITTI dataset cropped to 352×704) and horizontal flipping were employed to avoid overfitting.

[0104] To verify the effectiveness of the above technical solution, two public datasets, NYU-Depth V2 (indoor) and KITTI (outdoor), were used for testing to compare the performance differences of existing technologies (DepthCLIP, learnable text token scheme).

[0105] Specifically, for the NYU-Depth V2 dataset test (indoor scene):

[0106] Test conditions: Input image resolution 480×640, CLIP backbone is ViT-B / 32, dynamic token template "4o4d", depth bin number M=7, after 25 rounds of training optimization (freezing pre-trained CLIP parameters).

[0107] Test results:

[0108]

[0109] Key results: In indoor scenes, this solution significantly improves the depth estimation accuracy of objects such as bookshelves and furniture, reduces edge blurring, and the continuity of depth values ​​for weakly textured walls is significantly better than existing technologies.

[0110] For the KITTI dataset test (outdoor scene):

[0111] Test conditions: Input image resolution 375×1242, CLIP backbone is ViT-B / 32, dynamic token template "4o4d", depth bin number M=7, after 25 rounds of training optimization (pre-trained CLIP parameters are frozen).

[0112] Test results:

[0113]

[0114] Key effects:

[0115] In this example, the attention interaction mechanism between dynamic tokens and image features enables text prompts to adapt to the deep semantics of different scenarios (indoor / outdoor, complex / simple), overcoming the scenario limitations of static tokens. The introduction of the KAN network achieves stronger nonlinear mapping capabilities than MLP under the same parameter scale, making the association between image features and text tokens more accurate. The extended text encoder's fusion of static and dynamic tokens not only retains the deep prior of static tokens but also adds dynamic semantics for image adaptation, improving the accuracy of cross-modal matching.

[0116] Example 2

[0117] This embodiment provides a monocular depth estimation device that integrates image-dependent dynamic cue tokens and CLIP, including an image feature extraction module, a dynamic token generation module, a cross-modal coding module, and a depth estimation module.

[0118] The image feature extraction module extracts multi-scale visual features from the input image using the CLIP image encoder and aggregates them to obtain global image features. This module first preprocesses the input image by standardizing the single RGB image to be estimated, normalizing the pixel values ​​to the range [0, 1] to meet the input requirements of the CLIP image encoder. Then, multi-scale feature extraction is performed by inputting the preprocessed image into the pre-trained CLIP image encoder and outputting multi-scale image features through a multi-layer neural network or convolutional operations. Finally, global feature aggregation is performed by applying spatial average pooling to the multi-scale image features, calculating the mean of each feature channel to obtain the global image features.

[0119] The dynamic token generation module generates dynamic cue tokens based on global image features and static learnable tokens. These tokens are adapted to the input image content through multi-head cross-attention interaction and nonlinear mapping. The module first constructs a set of original static tokens from static learnable text, initialized as normally distributed vectors with a mean of zero and a standard deviation of 0.02. Next, all image features are mapped to the Key and Value matrices of the attention mechanism using learnable weight matrices. The tokens are mapped to the Query matrix using the learnable weight matrix. The similarity between the Query and Key matrices is then calculated using the scaled dot product attention formula to obtain the attention weight matrix A. The attention weight matrix A and the Value matrix V are then weighted and summed to obtain the attention-enhanced features. Finally, the attention-enhanced features are residually concatenated with the original static tokens. The residually concatenated attention-enhanced features are then input into a KAN network for nonlinear mapping to generate the dynamic cue tokens.

[0120] The cross-modal encoding module concatenates static learnable tokens and dynamic cue tokens, then fuses them through an extended CLIP text encoder to generate cross-modal text embeddings. The extended CLIP text encoder integrates the depth prior of the static learnable tokens with the image adaptation information of the dynamic cue tokens. This module first concatenates the static learnable tokens and dynamic cue tokens along the token dimension to form an extended input. Then, a linear projection layer is added to map the extended input to a dimension compatible with the CLIP text encoder. Finally, the projected features are input into a multi-layer Transformer structure of the CLIP text encoder, where a self-attention mechanism captures the semantic relationships between tokens to generate cross-modal text embeddings.

[0121] The depth estimation module is based on similarity matching between cross-modal text embeddings and input image features, and outputs high-precision depth maps through depth binning classification. This module first uniformly divides a continuous depth range into M depth bins, each bin corresponding to a specific depth interval, where the depth range is 0–100 meters. Then, it calculates the cosine similarity between the input image features and the cross-modal text embeddings, and obtains the probability distribution of each bin through Softmax normalization. Finally, it takes the median value of the depth interval corresponding to the bin with the highest probability as the depth estimate for the pixel.

[0122] This monocular depth estimation device, which integrates image-dependent dynamic cue tokens and CLIP, dynamically adjusts the cue tokens based on the input image content, thereby improving the accuracy and adaptability of depth estimation. By combining static learnable tokens with dynamic cue tokens, it retains prior depth knowledge while enhancing adaptability to different scenes. The depth binning classification method makes depth estimation more accurate, especially in the 0–100 meter depth range, providing high-quality depth map output.

[0123] In this example, multi-head cross-attention enables dynamic interaction between global image features and static learnable tokens. Combined with the nonlinear mapping of the KAN network, dynamic cue tokens adapted to the input image content are generated, addressing the core deficiency of static tokens in existing technologies. Furthermore, by concatenating "static tokens + dynamic tokens" as input, using linear projection adaptation and multi-layer Transformer encoding, deep fusion of cross-modal features is achieved, while preserving the depth prior of static tokens and the image adaptability of dynamic tokens, thus improving the correlation between text embeddings and image features. In the token generation module, a KAN network is introduced to replace the traditional MLP, enhancing the nonlinear function approximation capability under the same parameter scale, accurately modeling the complex relationship between image features and text tokens, and further improving depth estimation accuracy.

[0124] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A monocular depth estimation method fusing images relying on dynamic prompt tokens and CLIP, characterized in that, The estimation method includes: Step S100: Extract multi-scale visual features from the input image using the CLIP image encoder and aggregate them to obtain global image features; Step S200: Based on the global image features and the static learnable token, a dynamic cue token is generated through multi-head cross-attention interaction and nonlinear mapping. The dynamic cue token is used to adapt to the input image content. Step S300: After concatenating the static learnable token and the dynamic cue token, the static learnable token is fused and encoded by the extended CLIP text encoder to generate a cross-modal text embedding. The extended CLIP text encoder fuses the depth prior of the static learnable token and the image adaptation information of the dynamic cue token. Step S400: Based on the similarity matching between the cross-modal text embedding and the input image features, output a high-precision depth map through depth binning classification.

2. The monocular depth estimation method based on the fusion of image-dependent dynamic cue tokens and CLIP as described in claim 1, characterized in that, In step S100, the extraction of multi-scale visual features from the input image using the CLIP image encoder and the aggregation to obtain global image features includes: Input image preprocessing involves standardizing a single RGB image whose depth is to be estimated, so that the pixel values ​​of the image are normalized to the range [0, 1] to meet the input requirements of the CLIP image encoder; Multi-scale feature extraction involves inputting the preprocessed image into a pre-trained CLIP image encoder, and outputting multi-scale image features through multi-layer neural networks or convolution operations. Global feature aggregation involves performing spatial average pooling on multi-scale image features to calculate the mean of each feature channel, thereby obtaining the global image features.

3. The monocular depth estimation method based on the fusion of image-dependent dynamic cue tokens and CLIP as described in claim 1, characterized in that, In step S200, based on the global image features and the static learnable token, a dynamic cue token is generated through multi-head cross-attention interaction and nonlinear mapping. This dynamic cue token is used to adapt to the input image content and includes: Construct a set of original static tokens for static learnable text, wherein the original static tokens are initialized as normal distribution vectors with a mean of zero and a standard deviation of 0.02; All the image features are mapped to the Key and Value matrices of the attention mechanism through a learnable weight matrix, respectively, wherein the token is mapped to the Query matrix through the learnable weight matrix; The similarity between the Query matrix and the Key matrix is ​​calculated using the scaled dot product attention formula to obtain the attention weight matrix. ; The attention weight matrix With the Value matrix Weighted summation is performed to obtain attention-enhanced features; The attention enhancement feature is residually connected to the original static token, and then the residually connected attention enhancement feature is input into the KAN network for nonlinear mapping to generate the dynamic cue token.

4. The monocular depth estimation method for fusing image-dependent dynamic cue tokens and CLIPs according to claim 1, characterized in that, In step S300, after concatenating the static learnable token and the dynamic cue token, the two are fused and encoded using an extended CLIP text encoder to generate a cross-modal text embedding. The extended CLIP text encoder fuses the depth prior of the static learnable token with the image adaptation information of the dynamic cue token, including: The static learnable token and the dynamic cue token are concatenated along the token dimension to form an extended input; Add a linear projection layer to map the extended input to a dimension compatible with the CLIP text encoder; The projected features are input into the multi-layer Transformer structure of the CLIP text encoder, and the semantic associations between tokens are captured through a self-attention mechanism to generate the cross-modal text embedding.

5. The monocular depth estimation method based on the fusion of image-dependent dynamic cue tokens and CLIP as described in claim 1, characterized in that, In step S400, the step of outputting a high-precision depth map through depth binning classification based on the similarity matching between the cross-modal text embedding and the input image features includes: The continuous depth range is evenly divided into M depth bins, each bin corresponding to a specific depth interval, wherein the depth range is 0 to 100 meters. Calculate the cosine similarity between the input image features and the cross-modal text embedding, and obtain the probability distribution of each bin through Softmax normalization; The median of the depth interval corresponding to the bin with the highest probability is taken as the depth estimate of the pixel.

6. A monocular depth estimation device that integrates image-dependent dynamic cue tokens and CLIP, characterized in that, include: The image feature extraction module is used to extract multi-scale visual features of the input image through the CLIP image encoder and aggregate them to obtain global image features. A dynamic token generation module is used to generate dynamic prompt tokens based on the global image features and static learnable tokens through multi-head cross-attention interaction and nonlinear mapping. The dynamic prompt tokens are used to adapt to the input image content. A cross-modal coding module is used to concatenate a static learnable token and a dynamic cue token, and then perform fusion encoding through an extended CLIP text encoder to generate a cross-modal text embedding. The extended CLIP text encoder fuses the depth prior of the static learnable token and the image adaptation information of the dynamic cue token. The depth estimation module is used to output a high-precision depth map by depth binning classification based on the similarity matching between the cross-modal text embedding and the input image features.

7. The monocular depth estimation apparatus for fusing image-dependent dynamic cue tokens and CLIP according to claim 6, characterized in that, The image feature extraction module is used for: Input image preprocessing involves standardizing a single RGB image whose depth is to be estimated, so that the pixel values ​​of the image are normalized to the range [0, 1] to meet the input requirements of the CLIP image encoder; Multi-scale feature extraction involves inputting the preprocessed image into a pre-trained CLIP image encoder, and outputting multi-scale image features through multi-layer neural networks or convolution operations. Global feature aggregation involves performing spatial average pooling on multi-scale image features to calculate the mean of each feature channel, thereby obtaining the global image features.

8. The monocular depth estimation apparatus for fusing image-dependent dynamic cue tokens and CLIP according to claim 6, characterized in that, The dynamic token generation module is used for: Construct a set of original static tokens for static learnable text, wherein the original static tokens are initialized as normal distribution vectors with a mean of zero and a standard deviation of 0.02; All the image features are mapped to the Key and Value matrices of the attention mechanism through a learnable weight matrix, respectively, wherein the token is mapped to the Query matrix through the learnable weight matrix; The similarity between the Query matrix and the Key matrix is ​​calculated using the scaled dot product attention formula to obtain the attention weight matrix. ; The attention weight matrix With the Value matrix Weighted summation is performed to obtain attention-enhanced features; The attention enhancement feature is residually connected to the original static token, and then the residually connected attention enhancement feature is input into the KAN network for nonlinear mapping to generate the dynamic cue token.

9. The monocular depth estimation apparatus for fusing image-dependent dynamic cue tokens and CLIP according to claim 6, characterized in that, The cross-modal coding module is used for: The static learnable token and the dynamic cue token are concatenated along the token dimension to form an extended input; Add a linear projection layer to map the extended input to a dimension compatible with the CLIP text encoder; The projected features are input into the multi-layer Transformer structure of the CLIP text encoder, and the semantic associations between tokens are captured through a self-attention mechanism to generate the cross-modal text embedding.

10. The monocular depth estimation apparatus for fusing image-dependent dynamic cue tokens and CLIP according to claim 6, characterized in that, The depth estimation module is used for: The continuous depth range is evenly divided into M depth bins, each bin corresponding to a specific depth interval, wherein the depth range is 0 to 100 meters. Calculate the cosine similarity between the input image features and the cross-modal text embedding, and obtain the probability distribution of each bin through Softmax normalization; The median of the depth interval corresponding to the bin with the highest probability is taken as the depth estimate of the pixel.

Images