Multi-modal monocular depth estimation method based on high-order features and attention mechanism

Abstract

The invention discloses a multi-modal monocular depth estimation method based on high-order features and an attention mechanism, and the method comprises the steps: obtaining a monocular RGB image and a corresponding real depth image, and carrying out the data enhancement processing of the monocular RGB image in a training set; a multi-modal monocular depth estimation network model based on high-order features and an attention mechanism is constructed, a loss function of the multi-modal monocular depth estimation network model is constructed, the multi-modal monocular depth estimation network model is trained by using a divided training set, and in back propagation of a training process, gradient values are transmitted through a chained rule, and a multi-modal monocular depth estimation network model is obtained. Updating parameters of the network model by adopting an optimization algorithm, and gradually converging and controlling the value of a loss function on training data to be minimum through multiple iterative training to obtain an optimal weight; and inputting the images in the test set into the multi-modal monocular depth estimation network model with the optimal weight to obtain a predicted depth map corresponding to the RGB monocular map.

Description

Technical Field

[0001] The present invention relates to the technical field of image depth estimation, and in particular to a multimodal monocular depth estimation method based on high-order features and an attention mechanism. Background Art

[0002] Depth estimation is a basic problem in scene perception. It obtains depth information by determining the spatial distance from the visual sensor to the observed scene. This technology has been widely used in many fields such as 3D reconstruction, visual navigation and obstacle detection. Although the binocular depth estimation method can achieve good accuracy, it often requires high hardware configuration. In addition, the simultaneous processing of two cameras not only has the problem of position deviation, but also requires a lot of calculation. Therefore, the monocular depth estimation task of obtaining accurate depth information through an RGB image has obvious application value. Early related work mainly relied on manual feature extraction or the use of geometric prior knowledge. However, these methods are time-consuming and labor-intensive, and cannot capture low-texture scene information, resulting in low efficiency and ability of depth feature extraction. In recent years, convolutional neural networks have been gradually applied to the field of computer vision, so researchers have tried to use convolutional neural networks to achieve monocular depth estimation tasks. However, most of these techniques ignore small-scale objects and vertical direction information, and cannot fully capture the complex relationship and structure between image features. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention discloses a multimodal monocular depth estimation method based on high-order features and attention mechanism, which specifically includes:

[0004] Obtain a monocular RGB image and a corresponding true depth image, use the monocular RGB image and the corresponding true depth image as a data set, divide the data set into a training set and a test set, and perform data enhancement processing on the monocular RGB image in the training set;

[0005] Construct a multimodal monocular depth estimation network model based on high-order features and attention mechanism, which includes a contrastive language-image pre-trained CLIP architecture, an EMA decoder unit, a high-order image feature enhancement unit, and a vertical discriminator unit;

[0006] Constructing a loss function of a multimodal monocular depth estimation network model, training the multimodal monocular depth estimation network model using a divided training set, wherein in the back propagation of the training process, the gradient value is transferred through the chain rule, and the parameters of the network model are updated using an optimization algorithm, and through multiple iterative training, the loss function is gradually converged to minimize the value on the training data, and the optimal weight is obtained;

[0007] The images in the test set are input into the multimodal monocular depth estimation network model with the optimal weights to obtain the predicted depth map corresponding to the RGB monocular image.

[0008] Furthermore, the loss function of the multimodal monocular depth estimation network model includes an improved scale-invariant logarithmic loss function L s and the discriminator loss function L d , improved scale-invariant logarithmic loss function L s The calculation method is as follows:

[0009]

[0010] in, d i is the true depth value, represents the network predicted depth value, P represents the number of pixels with valid true values, and the discriminator loss function L d It is expressed as:

[0011]

[0012] Among them, G and D represent the functions corresponding to the generator and discriminator respectively, where the generator includes a CLIP model with a high-order module and an EMA decoder, x represents the input RGB image, and y represents the corresponding depth map; G(z) represents the predicted depth map generated by the generator based on the input RGB image, D(x,y) represents the judgment of the discriminator on the true depth map, and D(x,G(z)) represents the judgment of the discriminator on the generated predicted depth map. The total loss function L is L s and L d The sum of the weights of :

[0013] L=γL s +βL d

[0014] Among them, γ and β represent different weights.

[0015] Furthermore, the CLIP model includes an image encoder and a text encoder, the image encoder is used to extract features from a monocular RGB image, and the text encoder is used to extract text features from depth cue words, wherein a dataset consisting of 400 million images and corresponding English subtitles is used to pre-train the CLIP model, and the trained CLIP model is fine-tuned as the main architecture of a multimodal monocular depth estimation network model.

[0016] Furthermore, the text encoder includes a modified marker, a learnable parameter lookup table and a CLIPTransformer structure, wherein learnable deep cue words without human subjective bias are input into the text encoder to obtain text cue features.

[0017] Furthermore, the EMA decoder unit includes an EMA attention mechanism, a convolutional layer, a ReLU activation function and a bilinear layer, wherein the EMA attention mechanism divides the input feature X into G sub-features in the channel dimension through a grouping operation, and inputs the reshaped feature map Y into three parallel paths for different operations, and integrates the results of different paths using a cross-spatial information aggregation method.

[0018] Furthermore, the high-order image feature enhancement unit uses covariance normalization to perform high-order modeling on the feature map output by the image encoder, thereby capturing high-order statistical information, and uses the Newton-Schultz iterative method to accelerate the training process of covariance normalization.

[0019] Furthermore, the vertical discriminator unit includes an integrated sigmoid activation function and a vertical attention module, and the vertical attention module performs horizontal pooling on the input image x to obtain different vertical feature maps X q and X k , the reshaped feature map X q After transposition and X k Multiply them, and then pass the product through the softmax function to generate the attention map A vsa , the initial input X is mapped to X using a convolutional layer v During the training process, a gradually increasing attention coefficient γ is used to multiply the mapping feature X v , and then multiply the product with the attention map A vsa Perform matrix multiplication and add it to the original input x to finally obtain the enhanced feature F vsa .

[0020] Due to the adoption of the above technical solution, the multimodal monocular depth estimation method based on high-order features and attention mechanism provided by the present invention has the following advantages:

[0021] The multimodal monocular depth estimation system uses the fusion of text and image modalities to capture more comprehensive depth information. The entire system uses the CLIP model as the backbone and uses matrix power normalization (MPN-COV) technology for high-order statistical modeling to extract image features with more complex statistical information and spatial structure. These features combined with learnable depth cues further promote the enhanced fusion of text and images. The multimodal architecture that introduces high-order information has richer structural information expression and deeply refined discernible features, achieving efficient and high-precision depth map extraction.

[0022] In addition, the reconstruction of the depth map using an effective multi-scale attention (EMA) decoder can capture contextual information of different scales, establish long-term dependencies between features, and carefully preserve spatial positions. The vertical attention mechanism can keenly capture the drastic depth changes in the vertical direction, and significantly improve the efficiency of depth estimation by using vertical prior knowledge. At the same time, the discriminator enables the model to gradually correct and refine the accuracy of depth restoration during training. Therefore, the use of the multi-scale attention decoder and the vertical discriminator strengthens the feature extraction ability and depth value acquisition ability of the multimodal model, achieving better results. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a schematic diagram of the overall process of the embodiment system in the present invention;

[0025] Figure 2 Schematic diagram of the architecture of the monocular depth estimation network in the present invention;

[0026] Figure 3 Schematic diagram of a text encoder unit in CLIP of the present invention;

[0027] Figure 4 Schematic diagram of a high-order image feature enhancement unit in the present invention;

[0028] Figure 5 It is a schematic diagram of the EMA decoder unit in the present invention;

[0029] Figure 6 Schematic diagram of a vertical discriminator unit in the present invention;

[0030] Figure 7 This is a comparison chart of the experimental results on the NYU DepthV2 dataset in the present invention;

[0031] Figure 8 This is a comparison chart of the experimental results on the KITTI dataset in the present invention. DETAILED DESCRIPTION

[0032] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] like Figure 1 A multimodal monocular depth estimation method based on high-order features and attention mechanism is shown, which specifically includes the following steps:

[0036] S1. Obtain a data set and divide it into a training set and a test set. The data set consists of an original RGB monocular image and its corresponding real depth image, and perform data enhancement on the RGB monocular image in the training set.

[0037] In the specific implementation, the training set and test set are obtained from the NYU Depth v2 and KITTI datasets. The NYU Depth V2 dataset contains 464 indoor scenes and a total of 120,000 image-depth pairs. The size of the original image is 480×640 pixels, which is randomly cropped to 416×640 pixels during the training process. During the implementation, 24,000 images are randomly selected as the training set, and 654 images are used as the test set for model verification. The KITTI dataset is an outdoor driving dataset consisting of 93,000 image-depth pairs in 61 scenes, acquired using a lidar sensor. The size of the original KITTI image is 1240×374 pixels, which is randomly cropped to 704×352 pixels during the training process. 23,000 images in the KITTI dataset are used for training, and 697 test images are used to evaluate the performance of the model in outdoor scenes.

[0038] S2. Construct a multimodal monocular depth estimation network model, wherein the multimodal monocular depth estimation network model includes a contrastive language-image pre-trained CLIP architecture, an EMA decoder unit, a high-order image feature enhancement unit, and a vertical discriminator unit. Figure 2 The overall structure of the multimodal monocular depth estimation network based on high-order features and attention mechanism is demonstrated.

[0039] The entire model uses the CLIP architecture as the backbone. The CLIP model is pre-trained on a dataset consisting of 400 million images and corresponding English language subtitles. The encoding part consists of an image encoder and a text encoder. In this embodiment, both the image encoder and the text encoder use a pre-trained ResNet-50 architecture. Specifically, first, the monocular RGB image is input into the image encoder with the terminal pooling layer removed to extract image features. Then, these output features are sent to the MPN-COV high-order image feature enhancement unit for second-order modeling, thereby capturing more interactive high-order statistical information and richer feature representations. Next, the extracted second-order image features are converted into 7×N vectors, which are connected with N deep prompt words (one for each depth box) and sent to the text encoder for text feature extraction. It is worth noting that in order to eliminate bias in human language, learnable deep prompt words are used to replace human-defined prompts. Figure 3 Shows the overall structure of the text encoder in CLIP.

[0040] After obtaining the text features with high-order information, these features are inner-producted with the first-order image features extracted by the image encoder, and then normalized. In addition, an EMA decoder is constructed to restore the multi-scale depth information of the normalized result. The EMA attention mechanism in the EMA decoder unit uses a grouping operation to learn the attention weight descriptor, which enhances the feature representation of the region of interest in each sub-feature. The logarithm obtained by performing a Softmax operation on the output of the EMA decoder is multiplied by 2N equidistant depth center vectors within the depth range of the dataset, and then an addition operation is performed along the 2N dimension to obtain a low-dimensional depth map. The depth map is then upsampled to the full input resolution to obtain the depth prediction result. Finally, the predicted depth map and the true depth value are compared. Figure 1 The depth information restoration result of the correction model is input into the vertical discriminator unit with vertical feature extraction capability.

[0041] During the implementation, the high-order image feature enhancement unit extracts second-order statistical information from the image to obtain richer image feature representation and spatial structure. Figure 4 The structural details of the high-order model MPN-COV in the high-order image feature enhancement unit are shown. The core idea of ​​MPN-COV is the covariance normalization operation. Specifically, for the feature map X∈RW×H×C , first convert it into a high-dimensional feature matrix through the covariance pooling layer. The calculation method is as follows:

[0042]

[0043] in, represents the feature function composed of convolutional layer, nonlinear ReLU unit and maximum pooling layer, W and b are the weight matrix and bias in the function; F is obtained by The calculated statistical matrix contains N d-dimensional convolution features. Then the covariance matrix is ​​calculated through the potential interaction between convolutions and embedded in the deep model. The calculation method is as follows:

[0044]

[0045] in, represents the identity matrix of size N×N, and the superscript T is the matrix transposition operation. After obtaining the symmetric positive semidefinite matrix C, it is further subjected to singular value decomposition.

[0046] C→(U,Λ),C=UΛU T

[0047] Where Λ=diag(λ 1 ,λ 2 ,…,λ d ) represents a diagonal matrix composed of decreasing eigenvalues ​​λi, whose corresponding eigenvectors form an orthogonal matrix U, U = [u 1 ,u 2 ,…,u d Here, λ i (i=1,2,…,d) is the sequence of eigenvalues, u i is the eigenvector corresponding to the eigenvalue. The power operation to convert the covariance matrix C to the eigenvalue can be expressed as:

[0048] (U,Λ)→A,A=C α =UΦ(Λ)U T

[0049] in, The exponent α is a positive real number in the range (0,1]. As a rule of thumb, α=0.5.

[0050] During implementation, the core unit of the EMA decoder unit is the EMA attention mechanism, and the structure of the EMA attention mechanism is as follows: Figure 5As shown in the figure. Specifically, the input feature X is first grouped into G sub-features in the channel dimension, and then these group features are reshaped and arranged into the batch dimension, resulting in a reshaped input tensor Y with a shape of (B*G, C / / G, H, W). Grouping and reshaping help preserve information on each channel and reduce computational overhead.

[0051] Subsequently, the reshaped input tensor Y is fed into three parallel paths for different operations. Two of the paths include global average pooling layers along the horizontal and vertical dimensions, respectively, which are used to capture long-range dependencies in both dimensions while retaining precise location information. The average pooling operations along the horizontal and vertical dimensions can be expressed as:

[0052]

[0053] Among them, Y c represents the input feature of the cth channel. Then, the output is connected to a 1×1 convolution layer, and two nonlinear sigmoid functions are used to fit the two-dimensional binomial distribution on the linear convolution. The output of the two activation functions is re-weighted with the original input and sent to the group normalization layer to improve the feature distribution.

[0054] In order to capture multi-scale spatial information, a convolutional layer with a convolution kernel of 3×3 is used on the third path to convolve the reorganized feature Y, and a cross-spatial information aggregation method with different spatial dimensions is used to integrate the group normalization results of the first two paths and the convolution output of the third path. Specifically, the output of the 3×3 convolution operation is fed into the average pooling layer and the softmax layer for sequential processing. The obtained output is then multiplied (dot product operation) with the group normalization output of the first two paths mentioned above to obtain the first spatial attention map. Similarly, the group normalization output is fed into the average pooling layer and the softmax layer in sequence, and the result is then multiplied by the output of the 3×3 convolution layer in the third path to obtain the second spatial attention map. Finally, the two spatial attention maps are added and then multiplied with Y to obtain the output depth map with weight redistribution.

[0055] In the implementation, a six-layer convolutional network integrating a sigmoid activation function and a vertical attention module is used as the vertical discriminator unit to identify the output depth map of the decoder. The structural details of the vertical attention module are as follows Figure 6 shown.

[0056] First, the input image x is pooled horizontally to obtain vertical features. The f in the equation softmax represents the average pooling layer in the horizontal direction. Next, two 1×1 convolutional layers are used to process the extracted vertical features to obtain different feature maps X q and X k. Then the reshaped feature map X q The transpose of X k The product result generates the attention map A through the softmax activation function vsa , as shown below:

[0057]

[0058] in, In addition, the initial input x is mapped to X using a 1×1 convolutional layer v During the training process, a gradually increasing attention coefficient γ is used to multiply the mapping feature X v , and then multiply the product with the attention map A vsa Perform matrix multiplication and add it to the original input x to finally obtain the enhanced feature F vsa .

[0059]

[0060] S3. Construct a loss function of the network, train a multimodal monocular depth estimation network on the divided training set, and update the parameters of the proposed network through the loss function. The parameter update specifically refers to using the back propagation algorithm to transfer the gradient, iterating repeatedly to reduce the error, so that the predicted depth is close to the actual depth, and finally obtaining the optimal weight model.

[0061] In this embodiment, the loss function of the multimodal monocular depth estimation network consists of two parts: an improved scale-invariant logarithmic loss function L s and the discriminator loss function L d .L s The calculation method is as follows:

[0062]

[0063] in, d i is the true depth value, represents the network predicted depth value, and P represents the number of pixels with valid true values. In addition, the loss function L of the discriminator d It is expressed as:

[0064]

[0065] Among them, G and D represent the functions corresponding to the generator and discriminator respectively, where the generator includes a CLIP model with a high-order module and an EMA decoder; x represents the input RGB image, and y is the corresponding depth map; G(z) represents the predicted depth map generated by the generator based on the input RGB image, D(x,y) represents the judgment of the discriminator on the real depth map, and D(x,G(z)) represents the judgment of the discriminator on the generated predicted depth map. The total loss function is L s and L d The sum of the weights of .

[0066] L=γL s +βL d

[0067] Among them, γ and β represent different weights, which are set to 1 and 0.5 respectively in the experiment.

[0068] S4. Load the best weight model obtained in step S3, and use the multimodal monocular depth estimation network based on high-order features and attention mechanism constructed in S2 to extract the depth information of the test set.

[0069] During the specific implementation, the monocular depth estimation model designed in the system was experimented on the NYU DepthV2 and KITTI datasets. Figure 7 and Figure 8 As shown in the figure, the proposed design method can capture more edge information and details on both indoor and outdoor datasets, and obtain richer depth information maps.

[0070] For the convenience of specific implementation by those skilled in the art, the hardware platform used in the present invention is NVIDIA GeForceRTX 4090GPU, and the software platform is PyTorch deep learning framework. During training, the batch size is set to 16, and 25 generations of training are performed using a fixed random seed. The Adam optimizer and One CycleLR learning rate scheduler are used. The learning rates of the generator (CLIP model with high-order modules and EMA decoder) and the vertical discriminator are set to 0.000357 and 0.0000357, respectively. In addition, the cyclic momentum is set to true, and the base momentum is 0.85. The maximum momentum, div factor, and final div factor are set to 0.95, 25, and 100, respectively. It is worth noting that the pre-trained CLIP model used in the model is provided by the OpenAI CLIP repository, which uses ResNet-50 as its backbone architecture.

[0071] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multimodal monocular depth estimation method based on high-order features and attention mechanism, characterized in that include: Obtain a monocular RGB image and a corresponding true depth image, use the monocular RGB image and the corresponding true depth image as a data set, divide the data set into a training set and a test set, and perform data enhancement processing on the monocular RGB image in the training set; Construct a multimodal monocular depth estimation network model based on high-order features and attention mechanism, which includes a contrastive language-image pre-trained CLIP architecture, an EMA decoder unit, a high-order image feature enhancement unit, and a vertical discriminator unit; Constructing a loss function of a multimodal monocular depth estimation network model, training the multimodal monocular depth estimation network model using a divided training set, wherein in the back propagation of the training process, the gradient value is transferred through the chain rule, and the parameters of the network model are updated using an optimization algorithm, and through multiple iterative training, the loss function is gradually converged to minimize the value on the training data, and the optimal weight is obtained; The images in the test set are input into the multimodal monocular depth estimation network model with the optimal weights to obtain the predicted depth map corresponding to the RGB monocular image.

2. The multimodal monocular depth estimation method based on high-order features and attention mechanism according to claim 1, characterized in that: The loss function of the multimodal monocular depth estimation network model includes an improved scale-invariant logarithmic loss function L s And the discriminator loss function L d , improved scale-invariant logarithmic loss function L s The calculation method is as follows: in, d i is the true depth value, represents the network predicted depth value, P represents the number of pixels with valid true values, and the discriminator loss function L d It is expressed as: Among them, G and D represent the functions corresponding to the generator and discriminator respectively, where the generator includes a CLIP model with a high-order module and an EMA decoder, x represents the input RGB image, and y represents the corresponding depth map; G(z) represents the predicted depth map generated by the generator based on the input RGB image, D(x,y) represents the judgment of the discriminator on the true depth map, and D(x,G(z)) represents the judgment of the discriminator on the generated predicted depth map. The total loss function L is L s and L d The sum of the weights of : L=γL s +βL d Among them, γ and β represent different weights.

3. The multimodal monocular depth estimation method based on high-order features and attention mechanism according to claim 2, characterized in that: The CLIP model includes an image encoder and a text encoder. The image encoder is used to extract features from a monocular RGB image, and the text encoder is used to extract text features from depth prompt words. A dataset consisting of 400 million images and corresponding English subtitles is used to pre-train the CLIP model, and the trained CLIP model is fine-tuned as the main architecture of a multimodal monocular depth estimation network model.

4. The multimodal monocular depth estimation method based on high-order features and attention mechanism according to claim 3, characterized in that: The text encoder includes a modified marker, a learnable parameter lookup table and a CLIPTransformer structure, wherein learnable deep prompt words without human subjective bias are input into the text encoder to obtain text prompt features.

5. The multimodal monocular depth estimation method based on high-order features and attention mechanism according to claim 1, characterized in that: The EMA decoder unit includes an EMA attention mechanism, a convolutional layer, a ReLU activation function and a bilinear layer, wherein the EMA attention mechanism divides the input feature X into G sub-features in the channel dimension through a grouping operation, and inputs the reshaped feature map Y into three parallel paths for different operations, and integrates the results of different paths using a cross-spatial information aggregation method.

6. The multimodal monocular depth estimation method based on high-order features and attention mechanism according to claim 1, characterized in that: The high-order image feature enhancement unit uses covariance normalization to perform high-order modeling on the feature map output by the image encoder, thereby capturing high-order statistical information, and uses the Newton-Schultz iterative method to accelerate the training process of covariance normalization.

7. The multimodal monocular depth estimation method based on high-order features and attention mechanism according to claim 1, characterized in that: The vertical discriminator unit includes an integrated sigmoid activation function and a vertical attention module, which performs horizontal pooling on the input image x to obtain different vertical feature maps X q and X k , the reshaped feature map X q After transposition and X k Multiply them, and then pass the product through the softmax function to generate the attention map A vsa , the initial input X is mapped to X using a convolutional layer v During the training process, a gradually increasing attention coefficient γ is used to multiply the mapping feature X v , and then multiply the product with the attention map A vsa Perform matrix multiplication and add it to the original input x to finally obtain the enhanced feature F vsa .

Images