A depth graph completion method based on linear attention and cascaded hourglass networks

Abstract

This invention relates to a depth map completion method based on linear attention and a cascaded hourglass network, comprising the following steps: inputting image samples into a depth map processing model to obtain a dense depth map, wherein the image samples include RGB images and their sparse depth maps; the depth map processing model includes: an RGB encoder for extracting multi-scale semantic features of the input RGB image; a depth path for progressively downsampling the input sparse depth map to obtain multi-scale depth feature maps; and a cascaded hourglass network comprising multiple cascaded hourglass modules, with an alignment and stitching module embedded between two connected hourglass modules. Each hourglass module is used to generate a dense depth map of the corresponding scale based on the current scale depth feature map before and after stitching, as well as the multi-scale semantic features, until the last hourglass module outputs the final dense depth map. This invention can reduce computational overhead and improve computational speed while ensuring accuracy and precision.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision, and in particular to a depth map completion method based on linear attention and cascaded hourglass networks. Background Technology

[0002] Depth completion aims to predict missing depth information for each pixel in a sparse depth map. The sparsity of depth maps stems from limitations in physical hardware: depth sensors often produce inaccurate or missing depth information when encountering smooth surface reflections, semi-transparent or transparent objects, and dark objects. Therefore, depth completion is crucial for various downstream 3D vision tasks such as autonomous driving, robotics, and augmented reality, all of which require dense and accurate depth data of the scene.

[0003] In recent years, most methods have begun to use deep neural networks to directly extract features from RGB-D images and predict dense depth maps through different encoder-decoder structures. Although these methods perform well in terms of accuracy, they are usually accompanied by complex network structures and high computational costs, and are difficult to meet the high frame rate requirements of real-time applications. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a depth map completion method based on linear attention and cascaded hourglass networks, which can reduce computational overhead and improve computational speed while ensuring accuracy and precision.

[0005] The technical solution adopted by this invention to solve its technical problem is: to provide a depth map completion method based on linear attention and cascaded hourglass networks, including the following steps:

[0006] Obtain image samples, which include RGB images and their corresponding sparse depth maps;

[0007] Image samples are input into a depth map processing model to obtain a dense depth map; the depth map processing model includes:

[0008] An RGB encoder is used to extract multi-scale semantic features from an input RGB image.

[0009] The deep path is used to downsample the input sparse depth map step by step to obtain a multi-scale depth feature map;

[0010] The cascaded hourglass network comprises multiple cascaded hourglass modules. An alignment and stitching module is embedded between two connected hourglass modules. The alignment and stitching module is used to align the dense depth map of the previous scale with the depth feature map of the current scale and stitch them together to form a new depth feature map, which is then sent to the hourglass module at the corresponding level. Each hourglass module at each level is used to generate a dense depth map of the corresponding scale based on the current scale depth feature map before and after stitching, as well as multi-scale semantic features, until the last hourglass module outputs the final dense depth map. A linear attention bottleneck block is embedded between the encoder and decoder of the hourglass module to introduce a linear attention mechanism.

[0011] Furthermore, the linear attention bottleneck block includes an input layer, a linear attention layer, and a scaling layer connected in sequence. The linear attention layer includes several linear attention units stacked on top of each other, and each linear attention unit includes a linear attention module and a feedforward network module connected in sequence.

[0012] Furthermore, the linear attention module is constructed based on an improved self-attention mechanism, which includes:

[0013] Replace the similarity function with a ReLU-based similarity function;

[0014] Adjust the matrix multiplication order of Query, Key, and Value to reduce time complexity.

[0015] Furthermore, the improved self-attention mechanism also includes:

[0016] A space reduction module is embedded at the input end, which is used to downsample the Value and Key according to a set ratio.

[0017] Furthermore, the ReLU-based similarity function is expressed as follows:

[0018] Sim(Q,K)=ReLU(Q)ReLU(k) T

[0019] In this context, Q, K, and V represent Query, Key, and Value, respectively.

[0020] Furthermore, adjusting the matrix multiplication order of Query, Key, and Value includes:

[0021] Adjust the matrix multiplication order of Query, Key, and Value to Q(K) T V).

[0022] Furthermore, the feedforward network module is built based on an improved MLP, which includes:

[0023] Squared ReLU is used as the activation function;

[0024] A normalization layer is embedded after the extension layer.

[0025] Furthermore, the improved MLP also includes:

[0026] Embed the MBConv module after the normalization layer.

[0027] Furthermore, during the training of the depth map processing model, supervision is applied to the output of each hourglass module at different resolutions.

[0028] Furthermore, the output of the hourglass module includes a foreground depth map and a background depth map, and applies asymmetric linear error supervision to the foreground depth and reflection asymmetric linear error supervision to the background depth.

[0029] Beneficial effects

[0030] By adopting the above-mentioned technical solution, this invention has the following advantages and positive effects compared with the prior art: This invention embeds a linear attention module into a cascaded hourglass network, and through a channel-learnable scaling and progressive reduction strategy for attention blocks, flexibly balances global and local features, effectively controlling the number of model parameters to only 1.38M. Furthermore, by replacing traditional self-attention with ReLU-based linear attention, it balances global feature extraction and high-speed inference, thus providing a more efficient solution for deep completion. In addition, by combining a spatial downsampling strategy, it achieves global context capture of high-resolution feature maps while reducing computational complexity to O(D). 2 This invention significantly reduces memory and inference time overhead. It enhances the nonlinear representation of the model by integrating a feedforward network at the output of the linear attention module. Furthermore, by introducing squared ReLU activation into the feedforward network and combining it with RMS normalization, the model improves nonlinear representation while incurring minimal additional computational cost. Through a bifacial extrapolation strategy, asymmetric supervision is applied to the foreground and background depths, effectively alleviating the ambiguity of upper and lower bounds in depth prediction. By applying multi-scale supervision at the outputs of the three hourglass modules, depth prediction is gradually refined from coarse to fine, improving the model's ability to recover detailed structures and its overall robustness. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of an RGB encoder according to an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the deep pathway in an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of a cascaded hourglass network according to an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the linear attention hourglass module structure according to an embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram of the linear attention bottleneck block structure according to an embodiment of the present invention;

[0036] Figure 6 This is a schematic diagram of the linear attention module structure according to an embodiment of the present invention;

[0037] Figure 7 This is a schematic diagram of the SR-FFN network structure according to an embodiment of the present invention;

[0038] Figure 8 This is a schematic diagram of the SR-MBConv-FFN network structure according to an embodiment of the present invention. Detailed Implementation

[0039] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0040] The embodiments of the present invention relate to a depth map completion method based on linear attention and cascaded hourglass networks, comprising the following steps:

[0041] Obtain image samples, which include RGB images and their corresponding sparse depth maps;

[0042] Image samples are input into a depth map processing model to obtain a dense depth map; the depth map processing model includes:

[0043] An RGB encoder is used to extract multi-scale semantic features from an input RGB image.

[0044] The deep path is used to downsample the input sparse depth map step by step to obtain a multi-scale depth feature map;

[0045] The cascaded hourglass network comprises multiple cascaded hourglass modules. An alignment and stitching module is embedded between two connected hourglass modules. This alignment and stitching module is used to align the dense depth map of the previous scale with the depth feature map of the current scale and stitch them together to form a new depth feature map, which is then sent to the hourglass module of the corresponding level. Each hourglass module is used to generate a dense depth map of the corresponding scale based on the current scale depth feature map before and after stitching and the multi-scale semantic features, until the last hourglass module outputs the final dense depth map.

[0046] One embodiment of this implementation, Example 1, is based on a cascaded hourglass network, fully leveraging its multi-scale processing advantages. Example 1 consists of a single RGB encoder and three consecutive hourglass modules:

[0047] · RGB encoder, such as Figure 1 As shown, two convolutional layers are applied to the input color image to extract shallow features, and then four residual blocks are used to extract multi-scale semantic information at the original, 1 / 2 and 1 / 4 scales, finally outputting semantic features at the 1 / 1, 1 / 2 and 1 / 4 scales.

[0048] • Deep pathways, such as Figure 2 As shown, the sparse depth map (1 / 1 resolution sparse depth map) generated by two max pooling operations (1 / 2 resolution sparse depth map) and (1 / 4 resolution sparse depth map), which are used as depth inputs for the three-stage hourglass module;

[0049] Cascaded hourglass networks, such as Figure 3 As shown, except for the first hourglass module, subsequent hourglass modules upsample the dense depth map predicted by the previous stage and stitch it with the sparse depth map of the corresponding resolution for the next stage of prediction.

[0050] This design enables the model to incorporate multi-scale information at each stage and continuously improves the quality of deep predictions through iterative refinement.

[0051] In some preferred embodiments, a linear attention bottleneck block can be embedded between the encoder and decoder of the hourglass module to introduce a linear attention mechanism, such as... Figure 4 The Linear Attention Hourglass Module (LAHM) is shown. In each LAHM, depth features are fused with RGB features of the corresponding scale, and finer-grained depth information is gradually recovered through bottom-up convolution and top-down linear attention mechanisms; after three LAHMs are cascaded, a fine and dense depth prediction is finally output at the highest resolution.

[0052] like Figure 5 As shown, the linear attention bottleneck block includes an input layer, a linear attention layer, and a scaling layer connected in sequence. The linear attention layer includes N linear attention units stacked on top of each other. Each linear attention unit includes a linear attention module and a feedforward network module connected in sequence.

[0053] Standard attention has O(n) 2The time complexity of traditional attention mechanisms is insufficient for real-time scenarios. To balance global feature extraction and high-speed inference, a ReLU-based linear attention mechanism can be used instead of the native attention mechanism. Its performance is comparable to standard attention, while providing a more efficient solution for deep completion. Specifically, this includes:

[0054] Replace the similarity function of the standard attention mechanism with a ReLU-based similarity function;

[0055] Adjust the matrix multiplication order of Query, Key, and Value to reduce time complexity.

[0056] Even better, a spatial reduction module can be embedded at the input end to spatially downsample the input features according to a set ratio, and then the Value, Key and Query can be calculated based on the spatially downsampled input features, and then subsequent attention-related calculations can be performed.

[0057] More specifically, standard self-attention can be expressed as:

[0058]

[0059] Where Q, K, and V represent the query vector, key vector, and value vector, respectively. k The dimension representing the key / query vector.

[0060] In the above formula, the original similarity function Replace it with the following ReLU-based similarity function:

[0061] Sim(Q,K) = ReLU(Q) / ReLU(K) T

[0062] And the matrix multiplication order is from (QK) T V is adjusted to Q(K) T V), making the complexity O(DL) 2 ) decreased to O(D 2 L). Where D is the number of channels and L is the sequence length. Furthermore, removing positional encoding from the attention mechanism does not impair model performance, and this strategy is also adopted in this implementation.

[0063] To further reduce the computational and memory overhead of the attention mechanism, a spatial downsampling strategy can be introduced into the linear attention module. Specifically, this strategy applies only to the input feature map. Spatial downsampling is performed by a scale of r (using convolution with stride r and normalization), resulting in... Then, based on F, Key and Value are generated, thereby significantly reducing computation and memory consumption while preserving global information. The formula can be expressed as:

[0064] SRLinear(F) = Norm(conv) r,r (F))

[0065] In the early stages of a network, because the feature map size is still relatively large, applying attention directly at full resolution can lead to over-smoothing of details. Spatial downsampling not only reduces overhead but also helps prevent over-smoothing.

[0066] like Figure 6 This is illustrated in Embodiment 2, which describes the linear attention mechanism proposed in this invention:

[0067] First, linearly map the original feature map of size h×w and channel dimension d to obtain the query Q∈R. hw×d Key K∈R hw×d Sum Q∈R hw×d ;

[0068] The query branch is obtained by applying the ReLU activation function to Q.

[0069] Key branches and value branches first reduce the sequence length from h×w to [a smaller value] through space reduction (downsampling).

[0070] The downsampled keys are linearly mapped and the ReLU activation function is applied to obtain the downsampled values. Then, the values ​​are linearly mapped and multiplied by ~V to obtain a d×d aggregation matrix M.

[0071] Finally, multiplying ~Q by M at the original resolution yields the output attention feature, which is then scaled by a learnable scaling factor.

[0072] This lightweight linear attention module replaces traditional quadratic complexity self-attention with ReLU-based linear attention and incorporates a spatial downsampling strategy. This enables attention computation to achieve good scalability for high-resolution inputs, significantly reducing memory and computational overhead while preserving the global receptive field. The spatially reduced linear attention approach significantly reduces computational complexity by minimizing space while maintaining information representation.

[0073] To enhance the representational power of the attention layer, a squared ReLU feedforward network can be integrated on top of the linear attention module, thereby introducing stronger nonlinearity. A channel-learnable scaling mechanism can be used to dynamically balance the fusion of global semantics and local details.

[0074] Given that linear attention itself lacks sufficient non-linear expression, in some preferred implementations, the original activation function of the standard MLP-FFN can be replaced with squared ReLU activation, and an RMS normalization layer can be added after the extended layer to stabilize the training process. The following two variations can be selected:

[0075] 1.SR-FFN, which uses squared ReLU after the standard MLP-FFN extension layer.

[0076] 2. SR-MBConv-FFN introduces the MBConv module on the basis of SR-FFN, combining squared ReLU with depthwise separable convolution to enhance the ability to capture local patterns.

[0077] like Figure 7 The diagram shows one embodiment, 3, of this implementation. In this feedforward neural network, the input feature map is first expanded by a linear layer to increase the number of channels. After the expansion layer, a squared ReLU activation function is used to enable the model to learn a more complex representation. Then, a linear layer is used to reduce the number of channels back to the original value. The additional squared ReLU can introduce higher-order nonlinear characteristics.

[0078] like Figure 8 The diagram shows one embodiment, 4, of this implementation. To further enhance the model's ability to learn complex patterns, based on embodiment 3, squared ReLU is combined with depthwise separable convolution. The combination of squared ReLU and depthwise separable convolution can better capture local patterns in the data. This feedforward neural network is particularly effective when multiple linear attention blocks are cascaded, because continuous attention operations can easily filter out local details. Although squared ReLU has powerful feature representation capabilities, it may become unstable during training. To mitigate this problem, a root mean square normalization layer (RMSNorm) can be added after the expansion layer—RMSNorm normalizes independently using the root mean square of each sample activation value, thus making the use of squared ReLU more stable.

[0079] One embodiment of this implementation, Example 5, involves training and testing a combination of Examples 1, 2, and 4.

[0080] To mitigate depth ambiguity for the training objective, each hourglass unit outputs a three-channel prediction: a foreground depth map. Background Depth Map and fusion weight The former constrains the predicted value to not exceed the true value, while the latter constrains it to not be lower than the true value, supervised by asymmetric linear error and reflection asymmetric linear error respectively.

[0081]

[0082] γ controls the constraint strength. The final depth is determined by... The loss function is fused and supervised using a smooth L1 loss function:

[0083]

[0084] Supervision is applied to the output of each hourglass module at different resolutions to improve detail and robustness. Let the three-stage output be... The total loss function is:

[0085]

[0086] The model was trained for 30 epochs. In the first 10 epochs, the weights were set to w0 = 1, w1 = 0.5, and w2 = 0.25. In the next 10 epochs, the weights were adjusted to w1 = 0.1 and w2 = 0.01. In the last 10 epochs, only w0 was retained. This strategy allows the model to learn both coarse and detailed structural information simultaneously, thereby achieving more accurate depth predictions.

[0087] In benchmark tests on 1216×352 resolution images, this embodiment achieved an inference speed of up to 126 FPS while maintaining accuracy comparable to or even better than existing lightweight models, which can meet the real-time application requirements of autonomous driving, augmented reality and other applications.

Claims

1. A deep depth completion method based on linear attention and cascaded hourglass network, characterized in that, Includes the following steps: Obtain image samples, which include RGB images and their corresponding sparse depth maps; Image samples are input into a depth map processing model to obtain a dense depth map; The depth map processing model includes: An RGB encoder is used to extract multi-scale semantic features from an input RGB image. The deep path is used to downsample the input sparse depth map step by step to obtain a multi-scale depth feature map; The cascaded hourglass network includes multiple cascaded hourglass modules. An alignment and stitching module is embedded between two connected hourglass modules. The alignment and stitching module is used to align the dense depth map of the previous scale with the depth feature map of the current scale and stitch them together to form a new depth feature map, which is then sent to the hourglass module of the corresponding level. Each hourglass module is used to generate a dense depth map of the corresponding scale based on the current scale depth feature map before and after stitching, as well as multi-scale semantic features, until the last hourglass module outputs the final dense depth map. A linear attention bottleneck block is embedded between the encoder and decoder of the hourglass module to introduce a linear attention mechanism; wherein, The linear attention bottleneck block includes an input layer, a linear attention layer, and a scaling layer connected in sequence. The linear attention layer includes several stacked linear attention units, and each linear attention unit includes a linear attention module and a feedforward network module connected in sequence. The linear attention module is constructed based on an improved self-attention mechanism, which includes: Replace the similarity function with a ReLU-based similarity function; Adjust the matrix multiplication order of Query, Key, and Value to reduce time complexity; A space reduction module is embedded at the input end, which is used to downsample the Value and Key according to a set ratio.

2. The method according to claim 1, characterized in that, The ReLU-based similarity function is expressed as follows: In this context, Q, K, and V represent Query, Key, and Value, respectively.

3. The method according to claim 1, characterized in that, The adjustment of the matrix multiplication order of Query, Key, and Value includes: Adjust the matrix multiplication order of Query, Key, and Value as follows: .

4. The method according to claim 1, characterized in that, The feedforward network module is built based on an improved MLP, which includes: Squared ReLU is used as the activation function; A normalization layer is embedded after the extension layer.

5. The method according to claim 4, characterized in that, The improved MLP also includes: Embed the MBConv module after the normalization layer.

6. The method according to claim 1, characterized in that, During training, the depth map processing model applies supervision to the output of each hourglass module at different resolutions.

7. The method according to claim 6, characterized in that, The output of the hourglass module includes a foreground depth map and a background depth map. An asymmetric linear error supervision is applied to the foreground depth, and a reflection asymmetric linear error supervision is applied to the background depth.

Images