A monocular depth estimation method based on two-stage semantic geometry enhancement

Abstract

The application provides a monocular depth estimation method based on two-stage semantic geometry enhancement. In the first stage, a fusion and refinement module is constructed, a deformable neighborhood attention is mixed as the core, and a hierarchical deformable neighborhood modeling and a gate fusion are used to strengthen the local structure expression. In the second stage, a geometry injection module is introduced, the geometry prior of a depth base model is explicitly injected into the fusion feature, the global relationship is geometrically recalibrated, the local bias in the first stage is inhibited, and the depth consistency across scenes is improved. The global geometry constraint is strengthened while the local detail quality is ensured, and the depth prediction precision and the depth map quality in a complex scene are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a monocular depth estimation method based on two-stage semantic geometry enhancement. Background Technology

[0002] Monocular depth estimation is a technique for recovering the depth information of each pixel from a single RGB image, and it is widely used in fields such as autonomous driving, robot navigation, and augmented reality. Existing monocular depth estimation methods are prone to ambiguity in multi-resolution scenes; simultaneously, fine-grained structures are easily weakened during feature compression, leading to blurred boundaries and missing thin structures. For example, DepthAnything V2 improves depth prediction accuracy through a data-driven approach, but feature compression also results in poor performance of the final prediction in complex texture regions. Existing methods lack a clear, phased collaborative strategy between semantic enhancement and geometric constraints, making it difficult to simultaneously consider local details and global consistency. Relying solely on local structure enhancement can improve boundary and texture representation, but it is difficult to stably constrain global geometric relationships across scenes; relying solely on global geometric constraints may weaken detail representation, leading to insufficient recovery of local structures. Therefore, a single stage or mechanism is insufficient to obtain stable and high-quality depth prediction results in complex scenes. While Bridging Geometric and Semantic Foundation Models for Generalized Monocular Depth Estimation and Semantic-Enhanced Monocular Depth Estimation via Fusion and Distillation of Foundation Models enhance the fine-grained structure of detailed objects by introducing semantic information, they cannot fully capture semantic information simply through cross-attention and self-attention. Furthermore, the global consistency of depth maps predicted based solely on local structural information still differs somewhat from that of ground truth depth maps. Summary of the Invention

[0003] To address the aforementioned issues, this invention discloses a monocular depth estimation method based on two-stage semantic geometry enhancement. The first stage constructs a fusion and refinement module (FRM), with Hybrid Deformable Neighborhood Attention (HDNA) as its core. Through hierarchical deformable neighborhood modeling and gated fusion, it enhances the expression of local structures and improves the depth discrimination capability for complex boundaries and textured regions. The second stage introduces a geometry injection module (GIM), which explicitly injects the geometric priors of the depth base model into the fusion features and performs geometric recalibration of global relationships. This suppresses the local bias that may occur in the first stage and improves the stability of cross-scene depth prediction.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows:

[0005] A monocular depth estimation method based on two-stage semantic geometry enhancement includes the following steps:

[0006] Step 1: Input the RGB image into the first stage of the network for feature fusion. The first stage of the network includes an encoding part and a decoding part. The encoding part includes a Transformer encoder. Semantic encoder The Fusion and Refinement Module (FRM) is used in the decoding section, which employs a decoder. ;

[0007] Step 2: Send the RGB images to the Transformer encoder respectively. and the semantic encoder to obtain deep features and semantic features ;

[0008] Step 3: Process the depth features obtained in Step 2 and semantic features The input is fed into the fusion and refinement module (FRM) to obtain richly detailed fusion and refinement features. The fusion and refinement module (FRM) includes a cross-attention block. A hybrid deformable neighborhood attention block HDNA, and a self-attention block The hybrid deformable neighborhood attention block HDNA obtains fine-grained and coarse-grained offsets by using 3×3 small convolution kernels and 7×7 large convolution kernels respectively. Then, the offsets are mixed and fused to obtain deformable sampling offsets. Finally, local neighborhood attention is constructed by unfolding to reduce computational overhead.

[0009] Step 4: Refine the fused features obtained in Step 3 Input to the decoder By using skip connections to perform bottom-up feature fusion, image details are gradually reconstructed, ultimately obtaining a depth map with the same resolution as the input RGB image, rich in fine details, and high accuracy. ;

[0010] Step 5: Label the ground truth depth map corresponding to the RGB image with the depth map predicted in Step 4. The loss value is calculated, and the network hyperparameters are iteratively optimized to obtain the trained first-stage network model;

[0011] Step 6: Input the RGB image into the second stage of the network for geometric prior injection. The second stage of the network includes an encoding part and a decoding part. The encoding part includes a Transformer encoder. Semantic encoder The module consists of a fusion and refinement module (FRM) and a geometry injection module (GIM). The decoding part uses a decoder. ;

[0012] Step 7: Import the network model weights obtained in Step 5 and freeze them, then train only for the Geometry Injection Module (GIM).

[0013] Step 8: Send the RGB images to the Transformer encoder respectively. and the semantic encoder to obtain deep features and semantic features Then, depth features and semantic features The input is fed into the fusion and refinement module (FRM) to obtain richly detailed fusion and refinement features. ;

[0014] Step 9: Apply the depth features obtained in Step 8 Input to the decoder Obtain depth map Then the depth map The fusion and refinement features obtained in step 8 The input is fed into the geometry injection module (GIM) to inject geometric priors into the fused and refined features. Obtain the final output features ;

[0015] Step 10: Calculate the final output features obtained in Step 9. Input to the decoder This yields a depth map with the same resolution as the input RGB image, richer details, and higher accuracy. ;

[0016] Step 11: Label the ground truth depth map corresponding to the RGB image with the depth map predicted in Step 10. The loss value is calculated, and the network hyperparameters are iteratively optimized to obtain the final trained network model.

[0017] As a supplement to the present invention, in step 1, the first-stage model uses a weighted encoder-decoder framework and is fine-tuned. During the fine-tuning process, the encoder... , and decoder The parameters are frozen and no training is performed; only the proposed fusion and refinement module FRM is randomly initialized and trained.

[0018] As a supplement to the present invention, in step 3, the depth features are... and semantic features The input is fed into the fusion and refinement module FRM, and the FRM algorithm is shown in equation (1). Specifically, the deep features are first... and semantic features Through cross attention blocks Obtain fusion features , The algorithm is shown in equation (2), where the query ( ) from deep features semantic features As a key ( ) and value ( Then, the data is fed into an MLP (Multilayer Perceptron) to obtain fused features. Subsequently, features were merged. The sampled offsets are fed into the hybrid deformable neighborhood attention block HDNA and used for local neighborhood attention by small convolutions (3×3) and large convolutions (7×7), resulting in local inductive bias features that preserve the convolutions. The HDNA algorithm is shown in equation (3), and finally passes through a self-attention block. Achieve rich and detailed fusion refinement features ,at this time , , Same origin The algorithm is shown in equation (4). Since all network layers in the fusion and refinement module FRM do not change the size of the feature map dimension, the final output fused and refined features are... With input deep features The dimensions are the same. Furthermore, due to the Transformer's advantage in global information processing, the final fused features achieve excellent global preservation, while the injection of semantic features enables the network to better handle fine-detail predictions in complex regions.

[0019] (1);

[0020] in Indicates cross attention blocks, Represents a hybrid deformable neighborhood attention block. This represents a self-attention block.

[0021]

[0022] (2);

[0023]

[0024] , , The weight matrix maps the features to query vectors respectively. Key vector Value vector The query vector ( ) from deep features Key vector ( ) and value vector ( (From semantic features) . Indicates matrix transpose. This represents the transposed key vector. This is a scaling factor to avoid the vanishing gradient problem during training.

[0025]

[0026]

[0027] (3);

[0028]

[0029]

[0030] in , , The weight matrix is ​​used to convert the features into query vectors. Key vector Value vector ; i is the current center position index, MLP is a multilayer perceptron, For offset The induced deformation local neighborhood, where j is the index of the j-th sampling point in the local neighborhood of the center position i; Let represent the key vector of the j-th sampling point in the local neighborhood centered at position i after transposition. This is a bias term. express The small offset obtained through a 3×3 convolution kernel express Large offset obtained through 7×7 convolution kernel express The feature map obtained after deformable sampling.

[0031]

[0032] , (4;

[0033]

[0034] , , The weight matrix maps the features to query vectors respectively. Key vector Value vector The query vector ( ), key vector ( ) and value vector ( (From the local inductive bias features of further preserved convolution) . Indicates matrix transpose. This represents the transposed key vector. This is a scaling factor to avoid the vanishing gradient problem during training.

[0035] As a supplement to the present invention, in step 5, the ground truth depth map and predicted depth map of the input image are finally processed. Total loss function used The definition is shown in equation (5):

[0036] (5);

[0037] in, For scale-shift invariant loss, For multi-scale gradient matching loss; The weights for the scale-invariant loss are set to 1 during model training; The weights for the multi-scale gradient matching loss are set to 2 during model training. , The calculation methods are shown in equations (6) and (7) respectively:

[0038]

[0039] (6);

[0040] in, Represents the set of all valid pixels. The set of effective pixels representing the gradient in the x-direction. This represents the effective set of pixels representing the gradient in the y-direction. Represents the gradient in the horizontal direction. Represents the gradient in the vertical direction. This represents the predicted depth value at (i, j). This represents the true depth value at (i, j), where k is the scale factor, representing the interval k pixels.

[0041] (7);

[0042] in, Represents the depth of prediction. H represents the true depth, H represents the height of the input image, and W represents the width of the input image. The mean absolute error loss with constant scale offset is calculated as shown in equation (8):

[0043] (8);

[0044] in, and These are the prediction depths. and true depth The scaling and offset results are calculated as shown in equation (9):

[0045] (9);

[0046] in, and Used to align predicted and true values ​​to satisfy zero translation and unit scale; and The calculation methods are shown in equations (10) and (11) respectively:

[0047] (10);

[0048] (11);

[0049] (12);

[0050] In model training, k has three scales: 1, 2, and 4.

[0051] As a supplement to the present invention, in step 9, the depth map is... With fusion and refinement features The data is input into the geometry injection module GIM, and the GIM algorithm is shown in equation (13). Specifically, the depth map is... The geometric prior G is generated by the geometric prior generator GPG. The GPG algorithm is shown in Equation (14). In the visual Transformer, the two-dimensional input image of size h×w is uniformly divided into HW blocks, which are the number of blocks per row and per column, respectively. Each block P ijWithin the spatial domain, there are unique two-dimensional coordinates, namely row and column indices. Given a corresponding depth map, the block at the corresponding location in the depth map reflects its distance from the camera plane. Based on these two types of priors (spatial prior and positional prior), a simple weighted learning is performed to obtain the geometric prior G, which is then attenuated (the specific process is as follows: The attenuation rate, This represents the new matrix obtained by raising each element of G to the power of β, since And since the elements in G are non-negative, the generated It is a matrix with 1s on the diagonal; the smaller the element value, the greater the geometric distance. Then... (Injecting explicit geometric priors into the feature map via multiplication) Injecting geometric prior G into the fused and refined features. The final output features are obtained from this process. At this point, Q, K, and V are homologous, all originating from fused and refined features. Since none of the network layers in the Geometric Injection Module (GIM) change the dimensionality of the feature map, the final output features... Fusion and refinement features with input The dimensions are the same.

[0052] ,

[0053] (13);

[0054] in , , The weight matrix is ​​used to convert the features into query vectors. Key vector Value vector The query vector ( ), key vector (K) and value vector ( All of these features are derived from the fusion and refinement features obtained from the first stage through the fusion and refinement module FRM. , The attenuation rate, Let G be the new matrix obtained by raising each element of G to the power of β. G represents the geometric prior obtained by the geometric prior generator GPG from the depth map Z, as shown in equation (14):

[0055] ,

[0056] , (14;

[0057] = GPG(Z) =

[0058] Indicate the position (i, j) and ( The depth distance between blocks j') Indicate the position (i, j) and ( Manhattan distance between blocks, j') and These are learnable weighted coefficients that sum to 1.

[0059] As a supplement to the present invention, in step 11, the ground truth depth map and predicted depth map of the input image are finally processed. Total loss function used The definition is the same as in the first stage of network training, the difference being the total loss function at this stage. middle The ratio is 1:1.

[0060] Since the goal of the first stage of training is to obtain a more refined, fine-grained structure, multi-scale gradient matching loss is used at this stage. The proportion of depth is relatively large, so a weight ratio of 1:2 was set in the first stage. The goal of the second stage training is to enable the model to predict global consistency. If the weight ratio remains unchanged, the model will focus more on learning local features of object edges and neglect global depth consistency. Therefore, a weight ratio of 1:1 was set in the second stage.

[0061] The beneficial effects of this invention are as follows:

[0062] 1. This invention fine-tunes the depth estimation discrimination model, which can not only utilize its rich priors, but also improve performance on specific tasks. By using the fusion and refinement module FRM, it overcomes the misjudgment of depth and loss of fine details in the discrimination model in areas with blurred boundaries, thin structures and complex textures.

[0063] 2. This invention employs a two-stage network structure, leveraging the rich priors of the discriminative model while significantly improving its generalization ability. The first stage uses the Fusion and Refinement (FRM) module to combine the advantages of deep and semantic features. First, the Transformer encoder processes RGB image input, improving the accuracy of depth estimation and enabling the model to better understand the overall layout of different scenes and the relative positions of distant objects. Second, the semantic encoder excels at capturing spatial features when processing RGB image input, further enhancing the representation of edge details and texture information. The second stage uses the Geometric Injection (GIM) module to inject geometric priors into the first-stage network, keeping the weights frozen, further improving the depth consistency of the prediction results without compromising the ability to capture fine object edge details learned in the first stage.

[0064] By proposing the Hybrid Deformable Neighborhood Attention (HDNA), an end-to-end collaborative architecture is constructed that integrates dual-scale convolutional feature enhancement and dynamic offset grid sampling. This strictly limits the network sampling offset, enabling the model to adaptively meet the depth prediction requirements of different regions. This not only compensates for the loss of fine-grained structure in discriminative models but also improves the accuracy of depth prediction. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of the two-stage framework of the model of the present invention.

[0066] Figure 2 This is a schematic diagram of the integration and refinement modules of the present invention.

[0067] Figure 3 This is a schematic diagram of the geometry injection module of the present invention.

[0068] Figure 4 Comparison chart of the present invention and the qualitative results of representative depth estimation.

[0069] Figure 5 Comparison of hybrid deformable neighborhood attention ablation experiments. Detailed Implementation

[0070] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0071] like Figure 1 As shown, the monocular depth estimation method based on two-stage semantic geometry enhancement described in this invention includes the following steps:

[0072] Step 1: Input the RGB image into the first stage of the network for feature fusion. The first stage of the network includes an encoding part and a decoding part. The encoding part includes a Transformer encoder. Semantic encoder The Fusion and Refinement Module (FRM) is used in the decoding section, which employs a decoder. ;

[0073] The first-stage model uses a fixed-weight encoder-decoder framework and fine-tunes it. During the fine-tuning process, the encoder... , and decoder The parameters are frozen and no training is performed; only the proposed fusion and refinement module FRM is randomly initialized and trained.

[0074] Step 2: Send the RGB images to the Transformer encoder respectively. and the semantic encoder to obtain deep features and semantic features ;

[0075] Step 3: Process the depth features obtained in Step 2 and semantic features The input is fed into the fusion and refinement module (FRM) to obtain richly detailed fusion and refinement features. The fusion and refinement module (FRM) includes a cross-attention block. A hybrid deformable neighborhood attention block HDNA, and a self-attention block ;

[0076] Specifically, the following applies: Depth features and semantic features The input is given to the fusion and refinement module FRM, and the FRM algorithm is shown in equation (1). Specifically, as follows... Figure 2 As shown. First, the depth features... and semantic features Through cross attention blocks Obtain fusion features , The algorithm is shown in equation (2), where the query ( ) from deep features semantic features As a key ( ) and value ( Then, the data is fed into an MLP (Multilayer Perceptron) to obtain fused features. Subsequently, features were merged. The sampled offsets are fed into the hybrid deformable neighborhood attention block HDNA and used for local neighborhood attention by small convolutions (3×3) and large convolutions (7×7), resulting in local inductive bias features that preserve the convolutions. The HDNA algorithm is shown in equation (3), and finally passes through a self-attention block. Achieve rich and detailed fusion refinement features ,at this time , , Same origin The algorithm is shown in equation (4). Since all network layers in the fusion and refinement module FRM do not change the size of the feature map dimension, the final output fused and refined features are... With input deep features The dimensions are the same. Furthermore, due to the Transformer's advantage in global information processing, the final fused features achieve excellent global preservation, while the injection of semantic features enables the network to better handle fine-detail predictions in complex regions.

[0077] (1);

[0078] in Indicates cross attention blocks, Represents a hybrid deformable neighborhood attention block. This represents a self-attention block.

[0079]

[0080] (2);

[0081]

[0082] , , The weight matrix maps the features to query vectors respectively. Key vector Value vector The query vector ( ) from deep features Key vector ( ) and value vector ( (From semantic features) . Indicates matrix transpose. This represents the transposed key vector. This is a scaling factor to avoid the vanishing gradient problem during training.

[0083]

[0084]

[0085] (3);

[0086]

[0087]

[0088] in , , The weight matrix is ​​used to convert the features into query vectors. Key vector Value vector ; i is the current center position index, MLP is a multilayer perceptron, For offset The induced deformation local neighborhood, where j is the index of the j-th sampling point in the local neighborhood of the center position i; Let represent the key vector of the j-th sampling point in the local neighborhood centered at position i after transposition. This is a bias term. express The small offset obtained through a 3×3 convolution kernel express Large offset obtained through 7×7 convolution kernel express The feature map obtained after deformable sampling.

[0089]

[0090] , (4;

[0091]

[0092] , , The weight matrix maps the features to query vectors respectively. Key vector Value vector The query vector ( ), key vector ( ) and value vector ( (From the local inductive bias features of further preserved convolution) . Indicates matrix transpose. This represents the transposed key vector. This is a scaling factor to avoid the vanishing gradient problem during training.

[0093] Step 4: Refine the fused features obtained in Step 3 Input to the decoder By using skip connections to perform bottom-up feature fusion, image details are gradually reconstructed, ultimately obtaining a depth map with the same resolution as the input RGB image, rich in fine details, and high accuracy. ;

[0094] Step 5: Label the ground truth depth map corresponding to the RGB image with the depth map predicted in Step 4. The loss value is calculated, and the network hyperparameters are iteratively optimized to obtain the trained first-stage network model;

[0095] Ground truth depth map and predicted depth map of the input image Total loss function used The definition is shown in equation (5):

[0096] (5);

[0097] in, For scale-shift invariant loss, For multi-scale gradient matching loss; The weights for the scale-invariant loss are set to 1 during model training; The weights for the multi-scale gradient matching loss are set to 2 during model training. , The calculation methods are shown in equations (6) and (7) respectively:

[0098]

[0099] (6);

[0100] in, Represents the set of all valid pixels. The set of effective pixels representing the gradient in the x-direction. This represents the effective set of pixels representing the gradient in the y-direction. Represents the gradient in the horizontal direction. Represents the gradient in the vertical direction. This represents the predicted depth value at (i, j). This represents the true depth value at (i, j), where k is the scale factor, representing the interval k pixels.

[0101] (7);

[0102] in, Represents the depth of prediction. H represents the true depth, H represents the height of the input image, and W represents the width of the input image. The mean absolute error loss with constant scale offset is calculated as shown in equation (8):

[0103] (8);

[0104] in, and These are the prediction depths. and true depth The scaling and offset results are calculated as shown in equation (9):

[0105] (9);

[0106] in, and Used to align predicted and true values ​​to satisfy zero translation and unit scale; and The calculation methods are shown in equations (10) and (11) respectively:

[0107] (10);

[0108] (11);

[0109] (12);

[0110] In model training, k has three scales: 1, 2, and 4.

[0111] Step 6: Input the RGB image into the second stage of the network for geometric prior injection. The second stage of the network includes an encoding part and a decoding part. The encoding part includes a Transformer encoder. Semantic encoder The module consists of a fusion and refinement module (FRM) and a geometry injection module (GIM). The decoding part uses a decoder. ;

[0112] Step 7: Import the network model weights obtained in Step 5 and freeze them, then train only for the Geometry Injection Module (GIM).

[0113] Step 8: Send the RGB images to the Transformer encoder respectively. and the semantic encoder to obtain deep features and semantic features Then, depth features and semantic features The input is fed into the fusion and refinement module (FRM) to obtain richly detailed fusion and refinement features. ;

[0114] Step 9: Apply the depth features obtained in Step 8 Input to the decoder Obtain depth map Then the depth map The fusion and refinement features obtained in step 8 The input is fed into the geometry injection module (GIM) to inject geometric priors into the fused and refined features. Obtain the final output features ;

[0115] Depth map With fusion and refinement features The input is given to the geometry injection module GIM, and the GIM algorithm is shown in equation (13). Specifically, as follows... Figure 3 As shown. Depth map Geometric prior G is generated through GPG. The GPG algorithm is shown in Equation (14). In the visual Transformer, a two-dimensional input image of size h×w is uniformly divided into HW blocks, which are the number of blocks per row and per column, respectively. Each block P ij Within the spatial domain, there are unique two-dimensional coordinates, namely row and column indices. Given a corresponding depth map, the block at the corresponding location in the depth map reflects its distance from the camera plane. Based on these two types of priors (spatial prior and positional prior), a simple weighted learning is performed to obtain the geometric prior G. Subsequently, the geometric prior G is injected into the fusion and refinement features through a decay method. The final output features are obtained from this process. At this point, Q, K, and V are homologous, all originating from fused and refined features. Since none of the network layers in the Geometric Injection Module (GIM) change the dimensionality of the feature map, the final output features... Fusion and refinement features with input The dimensions are the same.

[0116] ,

[0117] (13);

[0118] in , , The weight matrix is ​​used to convert the features into query vectors. Key vector Value vector The query vector ( ), key vector (K) and value vector ( All of these features are derived from the fusion and refinement features obtained from the first stage through the fusion and refinement module FRM. , The attenuation rate, Let G be the new matrix obtained by raising each element of G to the power of β. G represents the geometric prior obtained by the geometric prior generator GPG from the depth map Z, as shown in equation (14):

[0119] ,

[0120] , (14;

[0121] = GPG(Z) =

[0122] Indicate the position (i, j) and ( The depth distance between blocks j') Indicate the position (i, j) and ( Manhattan distance between blocks, j') and These are learnable weighted coefficients that sum to 1.

[0123] Step 10: Calculate the final output features obtained in Step 9. Input to the decoder This yields a depth map with the same resolution as the input RGB image, richer details, and higher accuracy. ;

[0124] Step 11: Label the ground truth depth map corresponding to the RGB image with the depth map predicted in Step 10. The loss value is calculated, and the network hyperparameters are iteratively optimized to obtain the final trained network model.

[0125] Ground truth depth map and predicted depth map of the input image Total loss function used The definition is the same as in the first stage of network training, the difference being the total loss function at this stage. middle The ratio is 1:1.

[0126] As shown in Table 1, the model achieves performance improvements on most datasets. Compared with the base model DepthAnythingV2, the AbsRel metric is reduced by an average of 8.44%. The improvement is most significant on the DIODE dataset, with an AbsRel reduction of 16.09%, demonstrating the effectiveness of the method.

[0127] Table 1. Zero-sample relative depth assessment

[0128]

[0129] Figure 4 The depth estimation results of this method are shown: Existing models generally fail to achieve accurate depth estimation because they do not make full use of semantic information. Figure 2 The present method optimally preserves fine structures such as the mosquito net in the first row, the bear and distant depth results in the second row, and the tree branches in the third row. While Marigold, based on StableDiffusion, performs well in terms of detail, it makes incorrect predictions about distant areas and the mosquito net area. The results show that the current model can effectively fuse geometric and semantic information, accurately distinguish the edges of objects with similar depths, preserve fine-grained details in complex scenes, and improve depth consistency.

[0130] Figure 5 This demonstrates the impact of the proposed Hybrid Deformable Neighborhood Attention (HDNA) on the fine-grained performance of first-stage feature fusion capture. For example... Figure 5As shown, compared to not using HDNA, using HDNA can capture more detailed structures. For example: the cabinet and mural in the first row, the alarm in the second row, the edge of the bush leaves in the third row, and the dead branches on the wall in the fourth row.

[0131] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A monocular depth estimation method based on two-stage semantic geometry enhancement, characterized in that, Includes the following steps: Step 1: Input the RGB image into the first stage of the network for feature fusion. The first stage of the network includes an encoding part and a decoding part. The encoding part includes a Transformer encoder. Semantic encoder The Fusion and Refinement Module (FRM) is used in the decoding section, which employs a decoder. ; Step 2: Send the RGB images to the Transformer encoder respectively. and the semantic encoder to obtain deep features and semantic features ; Step 3: Process the depth features obtained in Step 2 and semantic features The input is fed into the fusion and refinement module (FRM) to obtain richly detailed fusion and refinement features. The fusion and refinement module (FRM) includes a cross-attention block. A hybrid deformable neighborhood attention block HDNA, and a self-attention block ; Step 4: Refine the fused features obtained in Step 3 Input to the decoder By using skip connections to perform bottom-up feature fusion, image details are gradually reconstructed, ultimately obtaining a depth map with the same resolution as the input RGB image, rich in fine details, and high accuracy. ; Step 5: Label the ground truth depth map corresponding to the RGB image with the depth map predicted in Step 4. The loss value is calculated, and the network hyperparameters are iteratively optimized to obtain the trained first-stage network model; Step 6: Input the RGB image into the second stage of the network for geometric prior injection. The second stage of the network includes an encoding part and a decoding part. The encoding part includes a Transformer encoder. Semantic encoder The module consists of a fusion and refinement module (FRM) and a geometry injection module (GIM). The decoding part uses a decoder. ; Step 7: Import the network model weights obtained in Step 5 and freeze them, then train only for the Geometry Injection Module (GIM). Step 8: Send the RGB images to the Transformer encoder respectively. and the semantic encoder to obtain deep features and semantic features Then, depth features and semantic features The input is fed into the fusion and refinement module (FRM) to obtain richly detailed fusion and refinement features. ; Step 9: Apply the depth features obtained in Step 8 Input to the decoder Obtain depth map Then the depth map The fusion and refinement features obtained in step 8 The input is fed into the geometry injection module (GIM) to inject geometric priors into the fused and refined features. Obtain the final output features ; Step 10: Calculate the final output features obtained in Step 9. Input to the decoder This yields a depth map with the same resolution as the input RGB image, richer details, and higher accuracy. ; Step 11: Label the ground truth depth map corresponding to the RGB image with the depth map predicted in Step 10. The loss value is calculated, and the network hyperparameters are iteratively optimized to obtain the final trained network model.

2. The monocular depth estimation method based on two-stage semantic geometry enhancement according to claim 1, characterized in that: In step 1, the first-stage model uses a fixed-weight encoder-decoder framework and fine-tunes it; during the fine-tuning process, the encoder... , and decoder The parameters are frozen and no training is performed; only the proposed fusion and refinement module FRM is randomly initialized and trained.

3. The monocular depth estimation method based on two-stage semantic geometry enhancement according to claim 1, characterized in that: In step 3, the depth features are... and semantic features The input is given to the fusion and refinement module FRM, and the FRM algorithm is shown in equation (1). First, the depth features and semantic features Through cross attention blocks Obtain fusion features , The algorithm is shown in equation (2), where the query ( ) from deep features semantic features As a key ( ) and value ( ), Then the data is fed into an MLP (Multilayer Perceptron) to obtain fused features. , Subsequently, fusion features The sampled offsets are fed into the hybrid deformable neighborhood attention block HDNA and used for local neighborhood attention by small convolutions (3×3) and large convolutions (7×7), resulting in local inductive bias features that preserve the convolutions. The HDNA algorithm is shown in equation (3). Finally, through the self-attention block Achieve rich and detailed fusion refinement features ,at this time , , Same origin The algorithm is shown in equation (4); since all network layers in the fusion and refinement module FRM do not change the size of the feature map dimension, the final output fused and refined features With input deep features The dimensions are the same. Furthermore, due to the global information processing advantage of Transformer, the final fused features achieve excellent global preservation, while the injection of semantic features enables the network to better handle the prediction of fine details in complex regions. （1）； in Indicates cross attention blocks, Represents a hybrid deformable neighborhood attention block. Represents a self-attention block; （2）； , , The weight matrix maps the features to query vectors respectively. Key vector Value vector ; where the query vector ( ) from deep features Key vector ( ) and value vector ( (From semantic features) ; Indicates matrix transpose. This represents the transposed key vector; This is a scaling factor to avoid the vanishing gradient problem during training; （3）； in , , The weight matrix is ​​used to convert the features into query vectors. Key vector Value vector ; i is the current center position index, MLP is a multilayer perceptron, For offset The induced deformation local neighborhood, where j is the index of the j-th sampling point in the local neighborhood of the center position i; Let represent the key vector of the j-th sampling point in the local neighborhood centered at position i after transposition. For bias terms; express The small offset obtained through a 3×3 convolution kernel express Large offset obtained through 7×7 convolution kernel express Feature map obtained after deformable sampling; ， （4）； , , The weight matrix maps the features to query vectors respectively. Key vector Value vector ; where the query vector ( ), key vector ( ) and value vector ( (From the local inductive bias features of further preserved convolution) ; Indicates matrix transpose. This represents the transposed key vector; This is a scaling factor to avoid the vanishing gradient problem during training.

4. The monocular depth estimation method based on two-stage semantic geometry enhancement according to claim 1, characterized in that: In step 5, the ground truth depth map and predicted depth map of the input image are finally obtained. Total loss function used The definition is shown in equation (5): （5）； in, For scale-shift invariant loss, For multi-scale gradient matching loss; The weights for the scale-invariant loss are set to 1 during model training; The weights for the multi-scale gradient matching loss are set to 2 during model training. , The calculation methods are shown in equations (6) and (7) respectively: （6）； in, Represents the set of all valid pixels. The set of effective pixels representing the gradient in the x-direction. This represents the effective set of pixels representing the gradient in the y-direction. Represents the gradient in the horizontal direction. Represents the gradient in the vertical direction. This represents the predicted depth value at (i, j). This represents the true depth value at (i, j), where k is the scale factor, representing the interval k pixels. （7）； in, Represents the depth of prediction. H represents the true depth, H represents the height of the input image, and W represents the width of the input image. The mean absolute error loss with constant scale offset is calculated as shown in equation (8): （8）； in, and These are the prediction depths. and true depth The scaling and offset results are calculated as shown in equation (9): （9）； in, and Used to align predicted and true values ​​to satisfy zero translation and unit scale; and The calculation methods are shown in equations (10) and (11) respectively: （10）； （11）； （12）； In model training, k has three scales: 1, 2, and 4.

5. The monocular depth estimation method based on two-stage semantic geometry enhancement according to claim 1, characterized in that: In step 9, the depth map is... With fusion and refinement features The input is given to the geometry injection module GIM, and the GIM algorithm is shown in equation (13). Specifically, the depth map Geometric prior G is generated through GPG. The GPG algorithm is shown in Equation (14). In the visual Transformer, the two-dimensional input image of size h×w is uniformly divided into HW blocks, which are the number of blocks in each row and column, respectively; each block P ij Within the spatial domain, there are unique two-dimensional coordinates, namely row and column indices. Given a corresponding depth map, the block at the corresponding position in the depth map reflects its distance from the camera plane. A geometric prior G is obtained through simple weighted learning based on spatial and positional priors. Subsequently, the geometric prior G is injected into the fusion and refinement features using a decay method. The final output features are obtained from this process. At this point, Q, K, and V are homologous, all originating from fused and refined features. Since none of the network layers in the Geometric Injection Module (GIM) change the dimensionality of the feature map, the final output features... Fusion and refinement features with input The dimensions are the same. ， （13）； in , , The weight matrix is ​​used to convert the features into query vectors. Key vector Value vector ; where the query vector ( ), key vector (K) and value vector ( All of these features are derived from the fusion and refinement features obtained from the first stage fusion and refinement module FRM. , The attenuation rate, Let G be the new matrix obtained by raising each element of G to the power of β; G represents the geometric prior obtained by the depth map Z through the geometric prior generator GPG, as shown in equation (14). ， ， （14）； = GPG(Z) = Indicate the position (i, j) and ( The depth distance between blocks j') Indicate the position (i, j) and ( Manhattan distance between blocks, j') and These are learnable weighted coefficients that sum to 1.

6. The monocular depth estimation method based on two-stage semantic geometry enhancement according to claim 1, characterized in that: In step 11, the ground truth depth map and predicted depth map of the input image are finally obtained. Total loss function used The definition is the same as in the first stage of network training, the difference being the total loss function at this stage. middle The ratio is 1:1.

Images