A Multimodal Monocular Depth Estimation Method Based on High-Order Features and Attention Mechanism

Abstract

This invention discloses a multimodal monocular depth estimation method based on high-order features and an attention mechanism, comprising: acquiring monocular RGB images and corresponding ground truth depth maps, and performing data augmentation processing on the monocular RGB images in the training set; constructing a multimodal monocular depth estimation network model based on high-order features and an attention mechanism, constructing a loss function for the multimodal monocular depth estimation network model, training the multimodal monocular depth estimation network model using a pre-defined training set, wherein gradient values ​​are propagated through a chain rule during backpropagation in the training process, and the parameters of the network model are updated using an optimization algorithm, and through multiple iterations of training, the loss function is gradually converged to minimize its value on the training data to obtain the optimal weights; and inputting images from the test set into the multimodal monocular depth estimation network model with the optimal weights to obtain the predicted depth map corresponding to the RGB monocular image.

Description

Technical Field

[0001] This invention relates to the field of image depth estimation technology, and in particular to a multimodal monocular depth estimation method based on high-order features and attention mechanisms. Background Technology

[0002] Depth estimation, which obtains depth information by determining the spatial distance from a visual sensor to the observed scene, is a fundamental problem in scene perception and has been widely applied in fields such as 3D reconstruction, visual navigation, and obstacle detection. While binocular depth estimation methods can achieve good accuracy, they often require high-end hardware. Furthermore, processing with two cameras simultaneously not only introduces positional discrepancies but also demands significant computation. Therefore, monocular depth estimation, which obtains accurate depth information from a single RGB image, has significant application value. Early work relied primarily on manual feature extraction or geometric prior knowledge. However, these methods are time-consuming and labor-intensive, and cannot capture low-texture scene information, resulting in low efficiency and capability in depth feature extraction. In recent years, convolutional neural networks (CNNs) have been increasingly applied in computer vision, leading researchers to attempt to implement monocular depth estimation using CNNs. However, these techniques largely ignore small-scale objects and vertical information, and cannot fully capture the complex relationships and structures between image features. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention discloses a multimodal monocular depth estimation method based on high-order features and an attention mechanism, specifically including:

[0004] Obtain monocular RGB images and their corresponding ground truth depth maps. Use the monocular RGB images and their corresponding ground truth depth maps as a dataset, and divide the dataset into a training set and a test set. Perform data augmentation processing on the monocular RGB images in the training set.

[0005] A multimodal monocular depth estimation network model based on high-order features and attention mechanism is constructed. The model includes a contrastive language-image pre-trained CLIP architecture, an EMA decoder unit, a high-order image feature enhancement unit, and a vertical discriminator unit.

[0006] A loss function for a multimodal monocular depth estimation network model is constructed. The multimodal monocular depth estimation network model is trained using a pre-defined training set. During backpropagation in the training process, gradient values ​​are passed through the chain rule, and an optimization algorithm is used to update the parameters of the network model. Through multiple iterations of training, the loss function is gradually converged to minimize its value on the training data, thereby obtaining the optimal weights.

[0007] The images in the test set are input into a multimodal monocular depth estimation network model with optimal weights to obtain the predicted depth map corresponding to the RGB monocular image.

[0008] Furthermore, the loss function of the multimodal monocular depth estimation network model includes an improved scale-invariant logarithmic loss function L. s and discriminator loss function L d Improved scale-invariant logarithmic loss function L s The calculation method is as follows:

[0009]

[0010] in, d i For depth truth value, Let L represent the network's predicted depth value, P represent the number of pixels with valid ground truth values, and L represent the discriminator loss function. d Represented as:

[0011]

[0012] Here, G and D represent the functions corresponding to the generator and discriminator, respectively. The generator includes a CLIP model with higher-order modules and an EMA decoder. x represents the input RGB image, and y is the corresponding depth map. G(z) represents the predicted depth map generated by the generator based on the input RGB image. D(x,y) represents the discriminator's judgment of the true depth map, and D(x,G(z)) represents the discriminator's judgment of the generated predicted depth map. The total loss function L is L s and L d The sum of weights:

[0013] L=γL s +βL d

[0014] Here, γ and β represent different weights.

[0015] Furthermore, the CLIP model includes an image encoder and a text encoder. The image encoder is used to extract features from monocular RGB images, and the text encoder is used to extract text features from depth cue words. The CLIP model is pre-trained using a dataset consisting of 400 million images and corresponding English language captions. The trained CLIP model is then fine-tuned to serve as the main architecture of the multimodal monocular depth estimation network model.

[0016] Furthermore, the text encoder includes a modified tokenizer, a learnable parameter lookup table, and a CLIPTransformer structure, wherein learnable deep cue words without human subjective bias are input into the text encoder to obtain text cue features.

[0017] Furthermore, the EMA decoder unit includes an EMA attention mechanism, a convolutional layer, a ReLU activation function, and a bilinear layer. The EMA attention mechanism divides the input feature X into G sub-features in the channel dimension through a grouping operation, and inputs the reshaped feature map Y into three parallel paths for different operations. A cross-spatial information aggregation method is used to integrate the results of different paths.

[0018] Furthermore, the higher-order image feature enhancement unit uses covariance normalization to perform higher-order modeling on the feature map output by the image encoder, thereby capturing higher-order statistical information and using the Newton-Schulz iterative method to accelerate the training process of covariance normalization.

[0019] Furthermore, the vertical discriminator unit includes an integrated sigmoid activation function and a vertical attention module. The vertical attention module performs horizontal pooling on the input image x to obtain different vertical feature maps X. q and X k The reshaped feature map X q After transpose and X k Multiply the products and then pass the product through the softmax function to generate an attention map A. vsa The initial input X is mapped to X using a convolutional layer. v During training, an increasingly larger attention coefficient γ is used to multiply the mapping feature X. v Then the product is multiplied by the attention graph A. vsa Perform matrix multiplication and add the result to the original input x to obtain the enhanced feature F. vsa .

[0020] By employing the above technical solution, the multimodal monocular depth estimation method based on high-order features and attention mechanism provided by this invention has the following advantages:

[0021] The multimodal monocular depth estimation system employs a fusion approach combining text and image modalities to capture more comprehensive depth information. The entire system uses the CLIP model as its backbone and leverages matrix power normalization (MPN-COV) for high-order statistical modeling, extracting image features with more complex statistical information and spatial structure. These features, combined with learnable depth cues, further enhance the fusion of text and image. This multimodal architecture, incorporating higher-order information, possesses richer structural information representation and more discriminative features with refined depth, achieving efficient and high-precision depth map extraction.

[0022] Furthermore, utilizing an efficient multi-scale attention (EMA) decoder for depth map reconstruction captures contextual information at different scales, establishes long-term dependencies between features, and meticulously preserves spatial locations. The vertical attention mechanism can keenly capture dramatic depth changes in the vertical direction, significantly improving the efficiency of depth estimation by leveraging vertical prior knowledge. Simultaneously, the discriminator allows the model to progressively correct and refine the accuracy of depth reconstruction during training. Therefore, the use of a multi-scale attention decoder and a vertical discriminator enhances the feature extraction and depth value acquisition capabilities of the multimodal model, achieving superior results. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the overall process of the system in the embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the architecture of the monocular depth estimation network in this invention;

[0026] Figure 3 This is a schematic diagram of the text encoder unit in CLIP of the present invention;

[0027] Figure 4 This is a schematic diagram of the high-order image feature enhancement unit in this invention;

[0028] Figure 5 This is a schematic diagram of the EMA decoder unit in this invention;

[0029] Figure 6 This is a schematic diagram of the vertical discriminator unit in this invention;

[0030] Figure 7 This is a comparison chart of the experimental results on the NYU DepthV2 dataset in this invention;

[0031] Figure 8 This is a comparison chart of experimental results on the KITTI dataset in this invention. Detailed Implementation

[0032] To make the technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention:

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] like Figure 1 The multimodal monocular depth estimation method shown includes the following steps: (The method is based on high-order features and an attention mechanism.)

[0036] S1. Obtain the dataset, divide it into training and test sets. The dataset consists of the original RGB monocular images and their corresponding real depth images, and perform data augmentation on the RGB monocular images in the training set.

[0037] In practice, training and test sets were obtained from the NYU Depth v2 and KITTI datasets. The NYU Depth v2 dataset contains 464 indoor scenes, totaling 120,000 image-depth pairs. The original images are 480×640 pixels in size, and are randomly cropped to 416×640 pixels during training. During implementation, 24,000 images were randomly selected as the training set, and 654 images were used as the test set for model validation. The KITTI dataset is an outdoor driving dataset consisting of 61 scenes and 93,000 image-depth pairs, acquired using a LiDAR sensor. The original KITTI images are 1240×374 pixels in size, and are randomly cropped to 704×352 pixels during training. 23,000 images from the KITTI dataset were used for training, while 697 test images were used to evaluate the model's performance in outdoor scenes.

[0038] S2. Construct a multimodal monocular depth estimation network model, which includes a contrastive language-image pre-trained CLIP architecture, an EMA decoder unit, a high-order image feature enhancement unit, and a vertical discriminator unit. Figure 2 The overall structure of a multimodal monocular depth estimation network based on high-order features and attention mechanisms is presented.

[0039] The entire model uses the CLIP architecture as its backbone. The CLIP model was pre-trained on a dataset consisting of 400 million images and corresponding English language captions. The encoding part comprises an image encoder and a text encoder. In this embodiment, both the image encoder and the text encoder use a pre-trained ResNet-50 architecture. Specifically, firstly, a monocular RGB image is input into the image encoder (with the end pooling layers removed) to extract image features. Then, these output features are fed into the MPN-COV higher-order image feature enhancement unit for second-order modeling, thereby capturing more interactive higher-order statistical information and richer feature representations. Next, the extracted second-order image features are converted into 7×N vectors, which are concatenated with N deep cue words (one for each depth bin) and fed into the text encoder for text feature extraction. Notably, to eliminate biases in human language, learnable deep cue words are used instead of human-defined cue words. Figure 3 This demonstrates the overall structure of the text encoder in CLIP.

[0040] After obtaining text features with high-order information, these features are inner-producted with the first-order image features extracted by the image encoder, followed by normalization. Furthermore, an EMA decoder is constructed to reconstruct the multi-scale depth information of the normalized result. The EMA attention mechanism in the EMA decoder unit uses grouping operations to learn attention weight descriptors, which enhances the feature representation of the region of interest in each sub-feature. The logarithm of the EMA decoder output after a softmax operation is multiplied by 2N equally spaced depth center vectors within the dataset depth range, and then addition is performed along the 2N dimension to obtain a low-dimensional depth map. The depth map is then upsampled to the full input resolution to obtain the depth prediction result. Finally, the predicted depth map and the ground truth depth are compared... Figure 1 The input is fed into a vertical discriminator unit with vertical feature extraction capability to correct the depth information restoration result of the model.

[0041] During implementation, the higher-order image feature enhancement unit extracts second-order statistical information from the image, thereby obtaining richer image feature representation and spatial structure. Figure 4 This paper showcases the structural details of the high-order model MPN-COV within the high-order image feature enhancement unit. The core idea of ​​MPN-COV is the covariance normalization operation; specifically, for feature mappings X∈R...W×H×C First, it is transformed into a high-dimensional feature matrix through a covariance pooling layer. The calculation method is as follows:

[0042]

[0043] in, F represents the feature function consisting of convolutional layers, nonlinear ReLU units, and max pooling layers, where W and b are the weight matrix and bias in the function; F is obtained through... The calculated statistical matrix contains N d-dimensional convolutional features. Then, the covariance matrix is ​​calculated through the latent interactions between convolutions and embedded into the deep model. The calculation method is as follows:

[0044]

[0045] in, Let C represent the identity matrix of size N×N, with the superscript T indicating matrix transpose. After obtaining the symmetric positive semi-definite matrix C, we further perform singular value decomposition on it.

[0046] C→(U,Λ),C=UΛU T

[0047] Among them, Λ=diag(λ1,λ2,…,λ d Let denot be a diagonal matrix consisting of decreasing eigenvalues ​​λi, whose corresponding eigenvectors form an orthogonal matrix U, U = [u1, u2, ..., ui]. d Here, λ i (i = 1, 2, ..., d) is a sequence of eigenvalues, u i These are the eigenvectors corresponding to the eigenvalues. Converting the covariance matrix C to the exponentiation of eigenvalues ​​can be expressed as:

[0048] (U,Λ)→A,A=C α =UΦ(Λ)U T

[0049] in, The exponent α is a positive real number in the range (0,1). Empirically, α = 0.5.

[0050] During implementation, the core unit of the EMA decoder unit is the EMA attention mechanism, the structure of which is as follows: Figure 5 As shown. Specifically, the input feature X is first grouped into G sub-features along the channel dimension. Then, these groups of features are reshaped and arranged in a batch dimension to obtain a reshaped input tensor Y with the shape (B*G, C / / G, H, W). Grouping and reshaping help preserve information on each channel and reduce computational overhead.

[0051] Subsequently, the reshaped input tensor Y is fed into three parallel paths for different operations. Two of these paths include global average pooling layers along the horizontal and vertical dimensions, respectively, which are used to capture long-range dependencies in both dimensions while preserving precise location information. The average pooling operations along the horizontal and vertical dimensions can be represented as:

[0052]

[0053] Among them, Y c Let represent the input feature of the c-th channel. Then, the output is concatenated into a 1×1 convolutional layer, where a two-dimensional binomial distribution on the linear convolution is fitted using two non-linear sigmoid functions. The outputs of the two activation functions are then reweighted and fed into a group normalization layer to improve the feature distribution.

[0054] To capture multi-scale spatial information, a 3×3 convolutional layer is used to convolve the reconstructed feature Y on the third path. A cross-spatial information aggregation method across different spatial dimensions is employed to integrate the group normalization results of the first two paths with the convolutional output of the third path. Specifically, the output of the 3×3 convolution operation is sequentially fed into an average pooling layer and a softmax layer for processing. The resulting output is then multiplied by the group normalization outputs of the first two paths (dot product operation) to obtain the first spatial attention map. Similarly, the group normalization output is sequentially fed into an average pooling layer and a softmax layer, and the result is multiplied by the output of the 3×3 convolutional layer in the third path to obtain the second spatial attention map. Finally, the two spatial attention maps are added together and multiplied by Y to obtain an output depth map with weight redistribution.

[0055] In implementation, a six-layer convolutional network integrating a sigmoid activation function and a vertical attention module is used as the vertical discriminator unit to distinguish the output depth map of the decoder. The structural details of the vertical attention module are as follows: Figure 6 As shown.

[0056] First, horizontal pooling is performed on the input image x to obtain vertical features, and f in the equation... softmax This represents the average pooling layer in the horizontal direction. Next, two 1×1 convolutional layers are used to process the extracted vertical features, resulting in different feature maps X. q and X k Then the reshaped feature map X q transpose multiplied by X k The product result is used to generate an attention map A through a softmax activation function. vsa As shown in the following formula:

[0057]

[0058] in, In addition, the initial input x is mapped to X using a 1×1 convolutional layer. v During training, the attention coefficient γ is gradually increased and multiplied by the mapping feature X. v Then the product is multiplied by the attention graph A. vsa Perform matrix multiplication and add the result to the original input x to obtain the enhanced feature F. vsa .

[0059]

[0060] S3. Construct the loss function for the network, train the multimodal monocular depth estimation network on the predefined training set, and update the parameters of the proposed network using the loss function. Specifically, the parameter update involves using the backpropagation algorithm to propagate gradients, iterating repeatedly to reduce errors, making the predicted depth approximate the true depth, and ultimately obtaining the optimal weight model.

[0061] In this embodiment, the loss function of the multimodal monocular depth estimation network consists of two parts: an improved scale-invariant logarithmic loss function L. s and discriminator loss function L d L s The calculation method is as follows:

[0062]

[0063] in, d i For depth truth value, Let L represent the network's predicted depth value, and P represent the number of pixels with valid ground truth values. Additionally, the discriminator's loss function L... d Represented as:

[0064]

[0065] Here, G and D represent the functions corresponding to the generator and discriminator, respectively. The generator includes a CLIP model with higher-order modules and an EMA decoder; x represents the input RGB image, and y is the corresponding depth map; G(z) represents the predicted depth map generated by the generator based on the input RGB image, D(x,y) represents the discriminator's judgment of the true depth map, and D(x,G(z)) represents the discriminator's judgment of the generated predicted depth map. The total loss function is L. s and L d The sum of weights.

[0066] L=γL s +βL d

[0067] Here, γ and β represent different weights, which were set to 1 and 0.5 respectively in the experiment.

[0068] S4. Load the optimal weight model obtained in step S3, and use the multimodal monocular depth estimation network constructed in S2 based on high-order features and attention mechanism to extract depth information of the test set.

[0069] In the specific implementation process, experiments were conducted on the monocular depth estimation model designed within the system on the NYU DepthV2 and KITTI datasets. For example... Figure 7 and Figure 8 As shown, the proposed design method can capture more edge information and details on both indoor and outdoor datasets, resulting in richer depth information maps.

[0070] To facilitate implementation by those skilled in the art, the hardware platform used in this invention is an NVIDIA GeForce RTX 4090 GPU, and the software platform is the PyTorch deep learning framework. During training, the batch size is set to 16, and 25 generations of training are performed using a fixed random seed. The Adam optimizer and the One CycleLR learning rate scheduler are employed. The learning rates for the generator (a CLIP model with higher-order modules and an EMA decoder) and the vertical discriminator are set to 0.000357 and 0.0000357, respectively. Furthermore, cyclic momentum is set to true, and the base momentum is 0.85. The maximum momentum, div factor, and final div factor are set to 0.95, 25, and 100, respectively. Notably, the pre-trained CLIP model used in the model is provided by the OpenAI CLIP repository, which uses ResNet-50 as its backbone architecture.

[0071] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-modal monocular depth estimation method based on high-order features and attention mechanism, characterized in that The application relates to a method for monocular depth estimation based on a multi-modal network model. The method comprises the following steps: acquiring monocular RGB images and corresponding real depth maps, taking the monocular RGB images and the corresponding real depth maps as a dataset, dividing the dataset into a training set and a test set, and performing data enhancement processing on the monocular RGB images in the training set; A multi-modal monocular depth estimation network model based on high-order features and attention mechanisms is constructed, which comprises a contrastive language-image pre-training CLIP architecture, an EMA decoder unit, a high-order image feature enhancement unit and a vertical discriminator unit; A loss function of the multi-modal monocular depth estimation network model is constructed, the multi-modal monocular depth estimation network model is trained by using the divided training set, in the back propagation of the training process, the gradient value is transmitted through the chain rule, and an optimization algorithm is used to update the parameters of the network model, through multiple iterative training, the value of the loss function on the training data is gradually converged to be minimum, and the best weight is obtained; The images in the test set are input into the multi-modal monocular depth estimation network model with the best weight, and the predicted depth map corresponding to the RGB monocular image is obtained. The loss function of the multi-modal monocular depth estimation network model comprises an improved scale invariant logarithmic loss function L s and a discriminator loss function L d The calculation method of the improved scale invariant logarithmic loss function L s is as follows: wherein, d i is the depth ground truth, represents the network predicted depth value, P represents the number of pixels with valid ground truth, the discriminator loss function L d is represented as: where G and D represent the functions corresponding to the generator and discriminator respectively, wherein the generator comprises a CLIP model with a high-order module and an EMA decoder, x represents an input RGB image, y is a corresponding depth map; G(z) represents a predicted depth map generated by the generator based on the input RGB image, D(x,y) represents the judgment of the discriminator on the real depth map, D(x,G(z)) represents the judgment of the discriminator on the generated predicted depth map, and the total loss function L is the sum of the weights of L s and L d . L = γL s + βL d Wherein, gamma and beta represent different weights.

2. The multi-modal monocular depth estimation method based on high-order features and attention mechanism according to claim 1, characterized in that: The CLIP model comprises an image encoder and a text encoder, the image encoder is used for feature extraction of the monocular RGB image, and the text encoder is used for text feature extraction of the depth prompt word, wherein a dataset composed of 400 million images and corresponding English language subtitles is used to pre-train the CLIP model, and the trained CLIP model is fine-tuned as the main architecture of the multi-modal monocular depth estimation network model.

3. The method of claim 2, wherein the method is based on high-level features and attention mechanism. The text encoder comprises a modified marker, a learnable parameter lookup table and a CLIPTransformer structure, wherein the learnable depth prompt word without human subjective bias is input into the text encoder to obtain the text prompt feature.

4. The multi-modal monocular depth estimation method based on high-order features and attention mechanism according to claim 1, characterized in that: The EMA decoder unit comprises an EMA attention mechanism, a convolution layer, a ReLU activation function and a bilinear layer, wherein the EMA attention mechanism divides the input feature X into G sub-features in the channel dimension through grouping operation, and maps the reshaped feature Y to three parallel paths for different operations, and adopts a cross-space information aggregation method to integrate the results of different paths.

5. The multi-modal monocular depth estimation method based on high-order features and attention mechanism according to claim 1, characterized in that: The high-order image feature enhancement unit adopts covariance normalization to model the feature map output by the image encoder, so as to capture high-order statistical information, and uses the Newton-Schulz iteration method to accelerate the training process of the covariance normalization.

6. The method of claim 1, wherein: The vertical discriminator unit includes an integrated sigmoid activation function and a vertical attention module. The vertical attention module performs horizontal pooling on the input image x to obtain different vertical feature maps X. q and X k The reshaped feature map X q After transpose and X k Multiply the products and then pass the product through the softmax function to generate an attention map A. vsa The initial input X is mapped to X using a convolutional layer. v During training, an increasingly larger attention coefficient γ is used to multiply the mapping feature X. v Then the product is multiplied by the attention graph A. vsa Perform matrix multiplication and add the result to the original input x to obtain the enhanced feature F. vsa .

Images