A semantically guided monocular depth estimation method

Abstract

The present invention discloses a semantic-guided monocular depth estimation method, which comprises the following steps: acquiring a monocular image and extracting image features; performing depth decoding and semantic segmentation on the image features respectively to obtain depth features and semantic features; using an attention mechanism to calculate attention weight coefficients of the depth features and the semantic features, and optimizing the depth features to obtain optimized depth features; calculating a reconstructed image based on the source image and the optimized depth features; calculating a loss function based on the reconstructed image and an input monocular image, and training a model using the loss function; predicting the depth information of the monocular image using the trained model; the present invention uses the attention mechanism to complete the guidance of the depth information by semantic segmentation, thereby generating a semantically consistent intermediate depth representation, realizing feature fusion across tasks, and overcoming the limitations of photometric reconstruction loss.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more particularly to a semantically guided monocular depth estimation method. Background Art

[0002] Currently, most self-supervised depth estimation methods rely on monocular image sequences for training and use photometric reconstruction loss (photometric error) to constrain the model. However, photometric reconstruction loss has limitations: the model's ability to accurately predict depth information assumes that the objects in the scene are static and the camera is dynamic. If there are moving objects in the scene, and the object's movement speed is the same as the camera's, it will cause obvious "black holes" in the final prediction results. This limitation seriously affects the model's scope of real-world application, especially in the field of autonomous driving, where the ability to detect and correctly model dynamic objects is crucial.

[0003] Therefore, how to overcome the limitations of photometric reconstruction is an urgent problem that those skilled in the art need to solve. Summary of the Invention

[0004] In view of this, the present invention provides a semantically guided monocular depth estimation method, which adopts the attention mechanism to guide the depth information of semantic segmentation, thereby generating a semantically consistent intermediate depth representation and realizing cross-task feature fusion, thereby overcoming the limitations of photometric reconstruction loss.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A semantically guided monocular depth estimation method comprises the following steps:

[0007] Acquire monocular images and extract image features;

[0008] Perform deep decoding and semantic segmentation on image features to obtain deep features and semantic features;

[0009] The attention mechanism is used to calculate the attention weight coefficients of deep features and semantic features, and the deep features are optimized to obtain optimized deep features;

[0010] Calculating a reconstructed image based on the source image and the optimized depth features;

[0011] Calculate the loss function based on the reconstructed image and the input monocular image, and train the model using the loss function;

[0012] Predict the depth information of monocular images through the trained model.

[0013] Furthermore, a monocular image is obtained and image features are extracted, the steps including:

[0014] Input the monocular image sequence into the shared encoder and use Conv-stem to divide the image into image blocks of the same size;

[0015] Multi-path Transformer is used to extract local features and global features of image blocks and mix them to obtain image features.

[0016] Furthermore, the image features are subjected to depth decoding and semantic segmentation respectively, and the steps include:

[0017] performing depth decoding by a depth decoder, the depth decoder comprising a plurality of first decoder blocks;

[0018] Perform semantic segmentation by using a semantic segmentation decoder, the semantic segmentation decoder is composed of a plurality of second decoder blocks; the first decoder and the second decoder correspond to each other in pairs;

[0019] The first decoder and the second decoder are both composed of a bilinear sampling layer, a cascade layer and a convolution layer connected in sequence.

[0020] Furthermore, the attention mechanism is used to calculate the attention weight coefficients of the deep features and semantic features, and the deep features are optimized. The steps include:

[0021] Get the feature map output by each pair of the first decoder and the second decoder as the deep feature F to be fused d ∈R H ×W×C and semantic features F s ∈R H×W×C ; Where H and W represent the width and height of the feature map, and C represents the dimension of the feature map;

[0022] The depth feature F is transformed into d and semantic features F s Unify the dimensions and concatenate them through cascade layers to obtain the mixed feature F ds ;

[0023] According to the mixed feature F ds , calculate the attention score, and perform deep feature F d and semantic features F s Weighted optimization;

[0024] The optimized depth features and semantic features are input into the next pair of first decoders and second decoders for depth decoding and semantic segmentation respectively, and so on, to output the final optimized feature map; wherein the optimized feature map includes an optimized depth feature map and an optimized semantic feature map.

[0025] Furthermore, the attention score calculation formula is:

[0026]

[0027] Among them, φ k and φ q is the attention calculation parameter, φ k is the key, φ q is the query; C′ is the unified dimension.

[0028] Furthermore, the reconstructed image is calculated based on the source image and the optimized depth feature, and the steps include:

[0029] Input the source image and the monocular image into the pose estimation network, and use the pose estimation network to predict the relative pose change;

[0030] The monocular image is reconstructed based on the relative pose change and optimized depth features. The reconstruction formula is:

[0031] I t′′→t =I′ t [proj(reproj(I t ,D t ,T t→t ))]

[0032] Among them, D t To optimize the deep features, I t is the input monocular image, I′ t is the source image; T t→t′ is the relative posture change; I t′→t To reconstruct the image, D_t is the predicted depth image, K is the known camera intrinsic parameter, [] is the sampling operator, reproj returns the 3D point cloud of the camera, and proj output projects the point cloud to I t′ The 2D coordinates of .

[0033] Furthermore, the loss function includes one or more of reconstruction loss, smoothing loss, semantic segmentation loss and semantic guidance loss; when there are multiple types, the multiple losses are weighted and summed to obtain the final loss function.

[0034] Furthermore, the reconstruction loss and smoothing loss are calculated:

[0035]

[0036] Among them, SSIM is the structural similarity paradigm, α represents the weight parameter, and Represent the depth gradient in the x-direction and y-direction respectively, D t Optimizing deep features for prediction.

[0037] Furthermore, the calculation steps of semantic segmentation loss are:

[0038] Predict semantic segmentation information S through the semantic segmentation decoder t With known label S' t Calculate the semantic segmentation loss L CE , the calculation method is as follows:

[0039]

[0040] Where N is the number of pixels and C is the number of categories. For each pixel i and each category c, S t (i,c) represents the model’s predicted probability of assigning pixel i to category c, S′ t (i,c) indicates whether the true label of pixel i is category c.

[0041] Furthermore, the calculation steps of semantic guidance loss are:

[0042] The depth feature D will be optimized t Input to the deep-semantic conversion network Θ D2S , and get its output S D2S =Θ(D t ), and the semantic guidance loss is calculated using the following formula:

[0043]

[0044] Among them, N represents the number of pixels, y i represents the true value, p i Represents Θ D2S Predict the predicted value of the i-th pixel in the segmentation result.

[0045] Beneficial effects of the present invention:

[0046] It can be seen from the above technical solution that compared with the existing technology, the present invention discloses a semantically guided monocular depth estimation method, which uses an attention mechanism to guide semantic segmentation on depth information, thereby generating a semantically consistent intermediate depth representation and realizing cross-task feature fusion, thereby overcoming the limitations of photometric reconstruction loss. It can also use semantic information to improve depth features, thereby improving the prediction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0048] Figure 1A flowchart of a monocular depth estimation method based on semantic guidance provided by an embodiment of the present invention;

[0049] Figure 2 A network architecture diagram of a semantically guided monocular depth estimation method provided by one embodiment of the present invention;

[0050] Figure 3 An architectural diagram of a deep-semantic conversion network provided by one embodiment of the present invention;

[0051] Figure 4 A decoder feature visualization diagram provided by an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of depth information prediction comparison provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] like Figure 1 and Figure 2 , an embodiment of the present invention discloses a semantically guided monocular depth estimation method, comprising the following steps:

[0055] S1: Acquire a monocular image and extract image features.

[0056] In one embodiment, S1 specifically includes:

[0057] S11: The monocular image sequence is input to the shared encoder and the resolution is restored through bilinear sampling. The Conv-stem in the shared encoder divides the image into image patches of the same size. The Conv-stem divides the monocular image into multiple image patches of the same size. The Conv-stem can better capture prominent low-level features and retain significant information. The Conv-stem consists of two convolutions with a kernel size of 3×3, the first convolution has a stride of 2, and the second convolution has a stride of 1.

[0058] S12: Use multi-path Transformer to extract local features and global features of image blocks, and obtain image features by splicing and mixing through cascade layers.

[0059] S2: Perform depth decoding and semantic segmentation on the image features respectively, and use the depth decoder and semantic segmentation decoder to perform bilinear sampling on the extracted features to restore the resolution, thereby obtaining depth features and semantic features.

[0060] In one embodiment, S2 specifically includes:

[0061] S21: performing depth decoding by a depth decoder, where the depth decoder is composed of a plurality of first decoder blocks;

[0062] S22: performing semantic segmentation by a semantic segmentation decoder, where the semantic segmentation decoder is composed of a plurality of second decoder blocks;

[0063] S23: The first decoder and the second decoder correspond to each other and perform depth decoding and semantic segmentation decoding respectively. The first decoder and the second decoder are both composed of a bilinear sampling layer, a cascade layer and a convolution layer connected in sequence;

[0064] The decoding process is as follows:

[0065] The output result of the previous decoder and the optimized result are respectively output to the bilinear sampling layer for upsampling. The sampling result and the feature output of the shared encoder are input to the cascade layer for cascade, and then the output of the current decoder is obtained after 3×3 convolution.

[0066] While executing S2, execute S3: The cross-task feature enhancement module (CFEM) uses the attention mechanism to calculate the attention weight coefficients of the deep features and semantic features, and optimizes the deep features to obtain the optimized deep features.

[0067] In one embodiment, S3 specifically includes:

[0068] S31: Obtain the feature map output by each pair of the first decoder and the second decoder as the deep feature F to be fused d ∈R H×W×C and semantic features F s ∈R H×W×C ; Where H and W represent the width and height of the feature map, and C represents the dimension of the feature map;

[0069] S32: Depth feature F is transformed into d and semantic features F s Unify the dimensions and concatenate them through cascade layers to obtain the mixed feature F ds ;

[0070] S33: According to the mixed feature F ds and deep features F d, calculate the attention score, and perform deep feature F d Weighted optimization; the calculation formula for the willpower score is:

[0071]

[0072] Among them, φ k and φ q is the attention calculation parameter, φ k is the key, φ q is the query; C′ is the unified dimension;

[0073] The enhanced deep features are obtained by multiplying the attention score with the deep features:

[0074] F′ d =Softmax(A×φ v (F d ))

[0075] Among them, φ v Representative value function;

[0076] S34: Input the optimized depth features and semantic features into the next pair of first decoders and second decoders, perform depth decoding and semantic segmentation respectively, and so on, to output the final optimized depth feature map; wherein, during the optimization process, the semantic features also use the same method as the depth features, calculate the attention score and then perform weighted optimization, and finally output the optimized semantic feature map.

[0077] The feature map can intuitively demonstrate the feature extraction capability of the model. In order to observe the structure of the feature map more clearly, all feature maps are projected onto the original input image for clearer visualization, such as Figure 4 As shown in the figure, the first column shows the feature maps of the present invention, the second column shows the feature maps of the present invention superimposed on the input map, and the third column shows the feature maps of other methods, and the feature maps of other methods superimposed on the input map. The model uses a cross-task feature enhancement module to enhance the depth representation with semantic information, which significantly improves the model's ability to capture features in low-texture areas of scene structure (such as the sky). At the same time, compared with other methods, the present invention can clearly separate object boundaries from background pixels (such as traffic signs and cars).

[0078] S4: Calculate and obtain a reconstructed image based on the source image and the optimized depth features.

[0079] In one embodiment, S4 specifically includes:

[0080] S41: Input the source image and the monocular image into the pose estimation network, and predict the relative pose change through the pose estimation network; the pose estimation network predicts the input image I t With the source image I t' and the relative motion of 6 degrees of freedom between t→t’ To complete the pose estimation. These 6 degrees of freedom include the rotational freedom and translational freedom on the X, Y, and Z coordinate axes;

[0081] S42: Reconstruct the monocular image based on the relative pose change and optimized depth features. The reconstruction formula is:

[0082] I t′→t =I′ t [proj(reproj(I t ,D t ,T t→t ))]

[0083] Among them, D t To optimize the deep features, I t is the input monocular image, I′ t is the source image; T t→t′ is the relative posture change; I t′→t To reconstruct the image, D_t is the predicted depth image, K is the known camera intrinsic parameter, [] is the sampling operator, reproj returns the 3D point cloud of the camera, and proj output projects the point cloud to I t′ The 2D coordinates of .

[0084] S5: Calculate a loss function based on the reconstructed image and the input monocular image, and train the model through the loss function; wherein the loss function includes one or more of reconstruction loss, smoothing loss, semantic segmentation loss, and semantic guidance loss; when there are multiple losses, perform weighted summation on the multiple losses to obtain the final loss function.

[0085] In one embodiment, S5 specifically includes:

[0086] S51: Calculate the reconstruction loss L by reconstructing the image and the input monocular image p and smoothing loss L smooth , the calculation method is as follows:

[0087]

[0088]

[0089] Among them, SSIM is the structural similarity paradigm, α represents the weight parameter, and Represent the depth gradient in the x-direction and y-direction respectively, D t is the predicted depth information;

[0090] S52: Predict semantic segmentation information S through semantic segmentation decoder tWith known label S' t Calculate the semantic segmentation loss L seg In order to train the semantic segmentation network and reduce the training cost, a pseudo-labeling method is adopted. The public semantic segmentation model is used to predict the semantic segmentation label for the input image. For each pixel, the loss function compares its predicted value with its corresponding pseudo label and calculates the cross entropy loss. The calculation method is as follows:

[0091]

[0092] Where N is the number of pixels and C is the number of categories. For each pixel i and each category c, S t (i,c) represents the model’s predicted probability of assigning pixel i to category c, S′ t (i,c) indicates whether the true label of pixel i is category c;

[0093] S53: Using only reconstruction loss and smoothing loss will result in large errors in low-texture prediction results. Therefore, a new semantic-guided loss is proposed. Considering that the deep decoder and semantic segmentation decoder are used for two different tasks, their outputs cannot be directly compared. Therefore, a deep-semantic conversion network Θ is designed. D2S , converting the depth image into a semantic segmentation image to achieve knowledge transfer between the depth decoder and the semantic segmentation decoder. Specifically, Θ D2S It consists of two 3×3 convolutional layers, each followed by a BatchNorm layer and a ReLu layer, and a 1x1 convolutional layer to output the segmented image, such as Figure 3 Shown from left to right are depth map, depth-semantic conversion network, semantic segmentation map; depth-semantic conversion network Θ D2S It consists of two 3×3 convolutional layers, each followed by a BatchNorm layer and a ReLu layer, and a 1x1 convolutional layer to output the segmented image. D2S The segmentation image generated from the predicted depth image is introduced into the depth decoder to extract the semantic knowledge in the semantic segmentation decoder; the predicted depth information D t Input to the deep-semantic conversion network Θ D2S , and get its output S D2S =Θ(D t ), and the semantic guidance loss is calculated using the following formula:

[0094]

[0095] Among them, N represents the number of pixels, y i represents the true value, p i Represents Θ D2S Predict the predicted value of the i-th pixel in the segmentation result;

[0096] S54: Calculate the total loss using the reconstruction loss, smoothing loss, semantic segmentation loss, and semantic guidance loss:

[0097] L f =L p +βL smooth +γL seg +ωL D2S

[0098] Among them, β, γ, and ω are weighting coefficients used to balance the importance of different losses.

[0099] In this example, the network is trained on a single NVIDIA RTX A6000 GPU using public authoritative data and the KITTI training model. During training, the resolution of the original image is adjusted to 192×640, the batch size is set to 12, and the Adam optimizer is used. The initial learning rate is set to 1.5×10 -4 The entire learning process lasted for 20 epochs. At the 10th and 15th epochs, the learning rate was reduced to 1 / 10 of the original rate. As in Chapter 3, the hyperparameters α for the reconstruction loss and β for the smoothing loss were set to 0.85 and 0.001, respectively. The hyperparameter γ for the semantic segmentation cross-entropy loss was set to 0.5, and the hyperparameter ω for the semantic guidance loss was set to 0.75. After training, the best performing model was selected as the final result.

[0100] S6: Predict the depth information of the monocular image through the trained model.

[0101] In one embodiment, an image outside the training set is selected as the input of the model, and the output of the model is: depth information of each pixel of the corresponding image.

[0102] In order to demonstrate the superiority of the prediction effect of the present invention, the following is an explanation of the specific image prediction results:

[0103] like Figure 5 The top image shows the input image, the bottom image shows the depth information predicted based on the input image, the bottom row shows the prediction results of our method, and the middle row shows the prediction results of other methods. These three images contain different types of objects, such as pedestrians, vehicles, and streets. In the areas marked with yellow boxes, it can be seen that our method performs better in low-texture areas and at object edges.

[0104] In another embodiment, the shared encoder is divided into 5 modules, consisting of five encoder modules, and the size of the feature map is reduced to half of the input after each encoder module.

[0105] The first decoder uses bilinear sampling to restore the size, and the second decoder uses upsampling to restore the size. The output of each decoder is twice the input. The input of the decoder consists of three parts. The first part comes from the output of the decoder in the previous stage, and the second part corresponds to the output of the encoder. The details of the decoder output feature map are enhanced by fusing feature maps of different scales. The third part comes from the cross-task feature enhancement module, which enhances the deep representation through semantic information.

[0106] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0107] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semantically guided monocular depth estimation method, characterized in that: The following steps are involved: Acquire monocular images and extract image features; Performing depth decoding and semantic segmentation on the image features respectively to obtain depth features and semantic features; wherein the depth decoding is performed by a depth decoder, and the depth decoder is composed of a plurality of first decoder blocks; and semantic segmentation is performed by a semantic segmentation decoder, and the semantic segmentation decoder is composed of a plurality of second decoder blocks; The attention mechanism is used to calculate the attention weight coefficients of the deep features and semantic features, and the deep features are optimized to obtain the optimized deep features. The steps include: Get the feature map output by each pair of the first decoder and the second decoder as the deep feature F to be fused d ∈R H×W×C and semantic features F s ∈R H×W×C ; Where H and W represent the width and height of the feature map, and C represents the dimension of the feature map; The depth feature F is transformed into d and semantic features F s Unify the dimensions and concatenate them through cascade layers to obtain the mixed feature F ds ; According to the mixed feature F ds , calculate the attention score, and perform deep feature F d and semantic features F s Weighted optimization; The optimized depth features and semantic features are input to the next pair of first decoder and second decoder for depth decoding and semantic segmentation respectively, and so on, to output the final optimized feature map; Calculating a reconstructed image based on the source image and the optimized depth features; Calculate the loss function based on the reconstructed image and the input monocular image, and train the model using the loss function; Predict the depth information of monocular images through the trained model.

2. The semantically guided monocular depth estimation method according to claim 1, characterized in that: Acquire a monocular image and extract image features. The steps include: Input the monocular image sequence into the shared encoder and use Conv-stem to divide the image into image blocks of the same size; Multi-path Transformer is used to extract local features and global features of image blocks and mix them to obtain image features.

3. The semantically guided monocular depth estimation method according to claim 1, characterized in that The first decoder and the second decoder are both composed of a bilinear sampling layer, a cascade layer and a convolution layer connected in sequence.

4. The semantically guided monocular depth estimation method according to claim 1, characterized in that: The attention score calculation formula is: Among them, φ k and φ q is the attention calculation parameter, φ k is the key, φ q is the query; C′ is the unified dimension.

5. The semantically guided monocular depth estimation method according to claim 1, characterized in that: Calculating a reconstructed image based on the source image and the optimized depth feature includes the following steps: Input the source image and the monocular image into the pose estimation network, and use the pose estimation network to predict the relative pose change; The monocular image is reconstructed based on the relative pose change and optimized depth features. The reconstruction formula is: Among them, D t To optimize the deep features, I t is the input monocular image, I′ t is the source image; T t→t′ is the relative posture change; I t′→t To reconstruct the image, D_t is the predicted depth image, K is the known camera intrinsic parameter, [] is the sampling operator, reproj returns the 3D point cloud of the camera, and proj output projects the point cloud to I t′ The 2D coordinates of .

6. The semantically guided monocular depth estimation method according to claim 1, characterized in that: The loss function includes one or more of reconstruction loss, smoothing loss, semantic segmentation loss and semantic guidance loss; when there are multiple losses, the multiple losses are weighted summed to obtain the final loss function.

7. The semantically guided monocular depth estimation method according to claim 6, characterized in that: Calculate the reconstruction loss and smoothing loss: Among them, SSIM is the structural similarity paradigm, α represents the weight parameter, and Represent the depth gradient in the x-direction and y-direction respectively, D t Optimizing deep features for prediction.

8. The semantically guided monocular depth estimation method according to claim 6, characterized in that: The calculation steps of semantic segmentation loss are: Predict semantic segmentation information S through the semantic segmentation decoder t With known label S' t Calculate the semantic segmentation loss L CE , the calculation method is as follows: Where N is the number of pixels, C is the number of categories, and for each pixel i and each category c, S t (i,c) represents the model’s predicted probability of assigning pixel i to category c, S′ t (i,c) indicates whether the true label of pixel i is category c.

9. The semantically guided monocular depth estimation method according to claim 6, wherein: The calculation steps of semantic guidance loss are: The depth feature D will be optimized t Input to the deep-semantic conversion network Θ D2S , and get its output S D2S =Θ(D t ), and the semantic guidance loss is calculated using the following formula: Among them, N represents the number of pixels, y i represents the true value, p i Represents Θ D2S Predict the predicted value of the i-th pixel in the segmentation result.

Images