A monocular depth prediction method based on multi-level feature parallel interaction fusion

Abstract

The application provides a monocular depth prediction method based on multi-level feature parallel interaction fusion, attempts to propose a CNN and Transformer hierarchical interaction fusion mode, uses SwinTransformer and CNN different resolution feature maps for fusion, so that a better encoder is obtained, more rich global and local feature information is obtained, and the effect of extracting dense features is achieved. Using hierarchical ResNet and SwinTransformer as the encoder, local adjacent information features and global long-range context information are extracted respectively, rich local and global information is obtained in the encoder stage through the fusion of the hierarchical ResNet network and the Transformer, and too much feature information is avoided from being lost due to repeated downsampling. In addition, the application constructs a new fusion module (FFM), which better promotes the efficient fusion of the convolutional neural network output and the Transformer output in the hierarchical feature extraction stage, further obtains dense depth information, and obtains a high-quality depth map, which is used in the fields of three-dimensional reconstruction, automatic driving, intelligent robots and the like.

Description

Technical fields:

[0001] This invention relates to the field of image processing, and more specifically, to a monocular depth prediction method based on multi-level feature parallel interactive fusion. Background technology:

[0002] The statements in this section only relate to the background technology in connection with this invention and do not necessarily constitute prior art.

[0003] Depth prediction, which involves predicting the depth information of a scene from one or more images, is a classic problem in computer vision. This depth information is crucial for many computer vision tasks, such as simultaneous localization and mapping (SMR), autonomous driving, 3D reconstruction, and object detection. While depth prediction can be accomplished with specialized hardware, such as Microsoft's rangefinder and Kinect camera, these are expensive and have limited application scenarios. Most images are taken with ordinary cameras and contain only color information about the scene. To reconstruct depth information, researchers have attempted to predict depth directly from RGB images, a more universal approach compared to obtaining depth information through specialized hardware. However, predicting depth from RGB images is an ill-posed problem; lack of cues, scale blurring, and semi-transparent or reflective materials can all lead to ambiguity, meaning that appearance cannot infer spatial structure. With the rapid development of deep learning, CNNs have become a key component of mainstream methods for providing reasonable depth maps from a single RGB input. Since the advent of the Transformer, it has received considerable attention across various fields.

[0004] Recently, the Transformer, employing a self-attention mechanism with a multilayer perceptron (MLP), has overcome the limitations of previous RNNs in natural language processing, such as their inability to be parallelized and low training efficiency. Many researchers have attempted to introduce the Transformer into computer vision, achieving some success. However, the Transformer focuses more on global contextual information while neglecting local information, while CNNs emphasize the extraction of local information. Subsequently, researchers combined the Transformer with CNNs, achieving good results and advancing monocular depth prediction. Currently, the combination of Transformer and CNN is based on an encoder-decoder structure. Most methods combine the Transformer and CNN sequentially, using the Transformer as the encoder to extract and transmit global contextual information. However, few methods combine the Transformer and CNN independently and in parallel, fully leveraging the advantages of both and their complementarity. Accurately recovering both large-scale geometry (walls) and local details (boundaries and small objects) simultaneously remains challenging. It is precisely this inaccurate inference across different scale regions that drives the full utilization of hierarchical scene structures for depth prediction. Summary of the Invention:

[0005] To address the aforementioned issues, this invention proposes a monocular depth prediction method based on multi-level feature parallel interactive fusion. It attempts to implement a hierarchical interactive fusion approach between CNN and Transformer, fusing feature maps of different resolutions from both the Swing Transformer and CNN to obtain a better encoder and acquire richer global and local feature information, achieving the effect of extracting dense features. Hierarchical ResNet and Swing Transformer are used as encoders to extract local neighbor information features and global long-range context information, respectively. Through the fusion of the hierarchical ResNet network and Transformer, rich local and global information is obtained in the encoder stage, avoiding the loss of excessive feature information due to repeated downsampling. Furthermore, this invention constructs a new fusion module to better promote the efficient fusion of convolutional neural network output and Transformer output in the hierarchical feature extraction stage, further obtaining dense depth information and high-quality depth maps for use in 3D reconstruction, autonomous driving, intelligent robotics, and other fields.

[0006] 1. A monocular depth prediction method based on multi-level feature parallel interactive fusion, characterized in that the method includes the following steps:

[0007] 1) Collect a public RGB dataset, augment the data, use the network's pre-trained weights, and initialize the encoder.

[0008] 2) ResNet is used to obtain multi-level local features, and Swin Transformer is used to obtain global context features.

[0009] 3) Construct a parallel multi-level interactive fusion module FFM to fully integrate local and global information and generate high-quality dense depth information.

[0010] 4) To calculate the distance between the predicted output depth and the ground truth depth map, the model is trained using scale-invariant loss (SI).

[0011] 2. The monocular depth prediction method based on multi-level feature parallel interactive fusion as described in claim 1, characterized in that: step 1) specifically involves:

[0012] 2.1) The mainstream public datasets collected include the NYU Depth v2 dataset and the KITTI dataset. The NYU Depth v2 dataset is mainly for indoor scenes, ranging from 0 to 10m, while the KITTI dataset is mainly for outdoor scenes and is mostly used in the field of autonomous driving, ranging from 0 to 80m.

[0013] 2.2) Data augmentation is performed using CutDepth to accelerate model training performance without increasing additional computational costs. Part of the input image is replaced with a ground true depth map to provide diversity to the input image, enabling the network to focus on high-frequency regions. In CutDepth, the coordinates (l, u) and size (w, h) of the cut region are randomly selected.

[0014] 2.3) Use MiT-b4 pre-trained weights to initialize the encoder, avoiding the problems of poor performance and unclear feature extraction when training from scratch, and building a model that is faster and more accurate.

[0015] 3. The monocular depth prediction method based on multi-level feature parallel interactive fusion according to claim 1, characterized in that: step 2) specifically involves:

[0016] 3.1) CNN and Transformer have unique and complementary characteristics. In order to make full use of the advantages of the two, a hierarchical interactive fusion network is proposed.

[0017] 3.2) In the encoder stage, in order to obtain rich global context features and local features, a Transformer and CNN parallel structure is used as the encoder. First, the image size is cropped to 320*320. The Swin Transformer branch and ResNet branch of the encoder are used to obtain feature maps with resolutions of [1 / 4, 1 / 8, 1 / 16, 1 / 32], and the number of channels are [64, 128, 256, 512], respectively. The block size of the Swin Transformer branch is 4, the number of channels is 3, the depth is [2, 2, 6, 2], the number of multi-head attention is [4, 8, 16, 32], and the window size is 10. The features of its four stages are [ft1, ft2, ft3, ft4].

[0018] 3.3) Corresponding to the four stages of the ResNet branch [fr1,fr2,fr3,fr4], the proposed fusion module FFM is used to combine the features of different levels and resolutions of Swin Transformer and ResNet to change the number of channels [F1,F2,F3,F4] from 512 to 64. Finally, the original resolution is restored through upsampling and convolution operations for dense prediction.

[0019] 4. The monocular depth prediction method based on multi-level feature parallel interactive fusion according to claim 1, characterized in that: step 3) specifically involves:

[0020] 4.1) Construct a parallel multi-level interactive fusion module FFM to fully integrate local and global information and generate high-quality dense depth information.

[0021] 4.2) Specifically, the positional relationships between vectors in the Transformer are obtained to acquire features. The Transformer model calculates the correlation between all blocks in the space, while ResNet establishes information connections in the local space. The Transformer establishes better contextual semantic information, while ResNet has stronger local information. To better integrate the advantages of both, this invention designs an FFM module to enhance the representation of each other's features. The specific process is as follows:

[0022] F t =multi(x t ,Conv(up(Sigmoid(x t (1)

[0023] Where x tThe expression represents the feature information of the Transformer branch, where up represents the upsampling operation, Sigmoid represents the activation function, Conv represents the 3×3 convolution operation, and multi represents pixel-level multiplication. Further processing of the Transformer branch features yields F. t .

[0024] 4.3) ResNet branch feature information is processed in a similar way, where Softmax represents the activation function and GAP represents global average pooling. This further compresses the spatial features, reducing the network parameters while compressing the global spatial information to the channel descriptors, thus preventing overfitting. The resulting feature F after further processing by ResNet is then obtained. r .

[0025] F r =multi(x res ,GAP(Conv(Softmax(x res (2)

[0026] F mid =BRC(Concat(F t ,up(F r ))) (3)

[0027] BRC stands for Pre-activation Module. A typical convolutional block consists of Conv, BN, and ReLU. In the final step, most of the negative values ​​of the nonlinear properties of the rectified linear unit activation are discarded. This invention uses a pre-activation convolutional block to alleviate this problem, obtaining the intermediate fusion feature F. mid .

[0028] F i =BRC(Concat(up(Conv(F)) t )),F mid ,up(Conv(F r (4)

[0029] Ultimately, through pixel-level addition operations, combined with F t F r With F mid The features are optimized again through the pre-activation module to obtain the output F. i (Where i takes values ​​from 1 to 4), representing the fusion features at each stage. A comparison of images with and without the fusion module shows that without the fusion module, only large targets like cars and railings can be captured, resulting in a blurred depth map boundary. Small targets like utility poles and road signs are captured very little or not at all. With the fusion module, not only can the shape and size features of large objects be captured, but the details of small targets like utility poles are also well captured.

[0030] 5. The monocular depth prediction method based on multi-level feature parallel interactive fusion according to claim 1, characterized in that: step 4) specifically involves:

[0031] To calculate the distance between the predicted output depth and the true ground depth map, this invention uses scale-invariant loss (SI) to train the model.

[0032]

[0033] here Where the actual ground depth is d i The predicted depth is α = 10, λ = 0.85.

[0034] Advantages of this invention: This invention uses CNN and Transformer as encoders and completes the monocular depth prediction task through parallel hierarchical fusion. In order to achieve better results, this invention introduces a new parallel interactive architecture. In addition, this invention designs an efficient hierarchical fusion module FFM to help to effectively fuse global features of Transformer and local features of CNN. Attached Figure Description

[0035] Figure 1 Flowchart of a monocular depth prediction method that integrates multi-level feature interaction

[0036] Figure 2 Comparison chart with other existing methods

[0037] Figure 3 Overall architecture diagram of the model

[0038] Figure 4 Detailed diagram of FFM hierarchical fusion module

[0039] Figure 5 Qualitative comparison chart with current best-performing methods Detailed Implementation

[0040] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Furthermore, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort, as well as the datasets collected, classified, and manually labeled by the present invention, are within the scope of protection of the present invention.

[0041] The process framework of this invention is as follows: Figure 1As shown, a monocular depth prediction method based on multi-level feature parallel interactive fusion is described below:

[0042] 1. Collect a public RGB dataset, augment the data, use the network's pre-trained weights, and initialize the encoder.

[0043] 1.1 The mainstream public datasets collected include the NYU Depth v2 dataset and the KITTI dataset. The NYU Depth v2 dataset is mainly for indoor scenes, ranging from 0 to 10m, while the KITTI dataset is mainly for outdoor scenes and is mostly used in the field of autonomous driving, ranging from 0 to 80m.

[0044] 1.2 CutDepth is used for data augmentation to accelerate model training performance without increasing additional computational cost. It replaces part of the input image with ground true depth map to provide diversity to the input image, enabling the network to focus on high-frequency regions. In CutDepth, the coordinates (l, u) and size (w, h) of the cut region are randomly selected.

[0045] 1.3 The encoder is initialized with MiT-b4 pre-trained weights to avoid the problems of poor performance and unclear feature extraction when training from scratch, thus building a model that is faster and more accurate.

[0046] 2. ResNet is used to obtain multi-level local features, and Swing Transformer is used to obtain global contextual features. The overall model structure is as follows: Figure 2 As shown.

[0047] The specific steps are as follows:

[0048] 2.1 CNN and Transformer have unique and complementary characteristics. To fully utilize the advantages of both, unlike other methods, this invention proposes a hierarchical interactive fusion network, such as... Figure 1 As shown.

[0049] 2.2 In the encoder stage, in order to obtain rich global context features and local features, a Transformer and CNN parallel structure is used as the encoder. First, the image size is cropped to 320*320. The Swin Transformer branch and ResNet branch of the encoder are used to obtain feature maps with resolutions of [1 / 4, 1 / 8, 1 / 16, 1 / 32], with the number of channels being [64, 128, 256, 512], respectively. The block size of the Swin Transformer branch is 4, the number of channels is 3, the depth is [2, 2, 6, 2], the number of multi-head attention is [4, 8, 16, 32], and the window size is 10. The features of its four stages are [ft1, ft2, ft3, ft4].

[0050] 2.3 corresponds to the four stages of the ResNet branch [fr1, fr2, fr3, fr4]. Then, through the proposed fusion module FFM, combined with the features of different layers and resolutions of Swin Transformer and ResNet, the number of channels is changed from 512 to 64. Finally, the original resolution is restored through upsampling and convolution operations for dense prediction. The specific model architecture is as follows: Figure 3 As shown.

[0051] 3. Construct a parallel multi-level interactive fusion module FFM to fully integrate local and global information, generating high-quality, dense, and deep information. The efficient fusion of FFM modules is as follows: Figure 4 As shown.

[0052] The specific steps are as follows:

[0053] 3.1 Obtaining the positional relationships between vectors in the Transformer model yields features. The Transformer model calculates the correlation between all blocks in the space, while ResNet establishes information connections in the local space. The Transformer establishes better contextual semantic information, while ResNet has stronger local information. To better integrate the advantages of both, this invention designs an FFM module to enhance the representation of each other's features. The specific process is as follows:

[0054] F t =multi(x t ,Conv(up(Sigmoid(x t (1)

[0055] Where x t The expression represents the feature information of the Transformer branch, where up represents the upsampling operation, Sigmoid represents the activation function, Conv represents the 3×3 convolution operation, and multi represents pixel-level multiplication. Further processing of the Transformer branch features yields F. t .

[0056] 3.2 A similar approach is used to process the ResNet branch feature information, where Softmax represents the activation function and GAP represents global average pooling. This further compresses the spatial features, reducing the network parameters while compressing the global spatial information to the channel descriptors, thus preventing overfitting. The resulting feature F after further processing by ResNet is then obtained. r .

[0057] F r =multi(x res ,GAP(Conv(Softmax(xres (2)

[0058] F mid =BRC(Concat(F t ,up(F r ))) (3)

[0059] BRC stands for Pre-activation Module. A typical convolutional block consists of Conv, BN, and ReLU. In the final step, most of the negative values ​​of the nonlinear properties of the rectified linear unit activation are discarded. This invention uses a pre-activation convolutional block to alleviate this problem, obtaining the intermediate fusion feature F. mid .

[0060] F i =BRC(Concat(up(Conv(F)) t )),F mid ,up(Conv(F r (4)

[0061] Ultimately, through pixel-level addition operations, combined with F t F r With F mid The features are optimized again through the pre-activation module to obtain the output F. i (Where i takes values ​​from 1 to 4), representing the fusion features at each stage. A comparison of images with and without the fusion module shows that without the fusion module, only large targets like cars and railings can be captured, resulting in a blurred depth map boundary. Small targets like utility poles and road signs are captured very little or not at all. With the fusion module, not only can the shape and size features of large objects be captured, but the details of small targets like utility poles are also well captured.

[0062] 4. Optimize the network using a loss function to obtain a more accurate depth map.

[0063] The specific steps are as follows:

[0064] To calculate the distance between the predicted output depth and the true ground depth map, this invention uses scale-invariant loss (SI) to train the model.

[0065]

[0066] here Where the actual ground depth is d i The predicted depth is α = 10, λ = 0.85.

[0067] 5. To demonstrate the efficiency of our proposed monocular depth prediction method based on multi-level feature parallel interactive fusion, such as... Figure 5 As shown, we present the depth map predicted by the network. The first three columns are the depth maps displayed by the current best methods, the fourth column is the depth map we display, and the fifth column is the actual RGB image.

[0068] The above description is for preferred embodiments of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A monocular depth prediction method based on multi-level feature parallel interactive fusion, characterized in that, The method includes the following steps: 1) Collect public RGB datasets, augment the data, use the pre-trained weights of the network, and initialize the encoder; 2) ResNet is used to obtain multi-level local features, and Swing Transformer is used to obtain global context features; 3) Construct a parallel multi-level interactive fusion module FFM to fully integrate local and global information and generate high-quality, dense, deep information; 3.1) Construct a parallel multi-level interactive fusion module FFM to fully integrate local and global information and generate high-quality, dense, deep information; 3.2) Specifically, the feature x is obtained by obtaining the positional relationship between vectors in the Transformer. t The Transformer model calculates the correlation between all blocks in the space, while ResNet establishes information connections in the local space. The Transformer establishes better contextual semantic information, while ResNet has stronger local information. To better integrate the advantages of both, the method designs an FFM module to enhance the representation of each other's features. The specific process is as follows: F t =multi(x t ,Conv(up(Sigmoid(x t )))) (1) Where x t The expression represents the feature information of the Transformer branch, where up represents the upsampling operation, Sigmoid represents the activation function, Conv represents the 3×3 convolution operation, and multi represents pixel-level multiplication. Further processing of the Transformer branch features yields F. t ; 3.3) ResNet branch feature information is processed in a similar way, where Softmax represents the activation function and GAP represents global average pooling. This further compresses the spatial features, reducing the network parameters while compressing the global spatial information to the channel descriptors, thus preventing overfitting. The resulting feature F after further processing by ResNet is then obtained. r ; F r =multi(x res ,GAP(Conv(Softmax(x res ))) (2) F mid =BRC(Concat(F t ,up(F r ))) (3) BRC stands for Pre-activation Module. The method introduces a pre-activation convolutional block to mitigate the impact of existing convolutional blocks that discard most of the negative values ​​of the nonlinearity of the rectified linear unit activation in the final step, resulting in the intermediate fusion feature F. mid ; F i =BRC(Concat(up(Conv(F t )),F mid ,up(Conv(F r )))) (4) Ultimately, through pixel-level addition operations, combined with F t F r With F mid The features are optimized again through the pre-activation module to obtain the output F. i , where i takes the Arabic numeral 1 to 4, representing the characteristics of the integration at each stage; 4) To calculate the distance between the predicted output depth and the ground truth depth map, the model is trained using scale-invariant loss (SI).

2. The monocular depth prediction method based on multi-level feature parallel interactive fusion as described in claim 1, characterized in that: The specific method for step 1) is as follows: 1.1) The mainstream public datasets collected include the NYU Depth v2 dataset and the KITTI dataset. The NYU Depth v2 dataset is for indoor scenes, ranging from 0 to 10m, while the KITTI dataset is for outdoor scenes and is mostly used in the field of autonomous driving, ranging from 0 to 80m. 1.2) Data augmentation is performed using CutDepth to accelerate model training performance without increasing additional computational costs. Part of the input image is replaced with a ground true depth map to provide diversity to the input image, enabling the network to focus on high-frequency regions. In CutDepth, the coordinates (l, u) and size (w, h) of the cut region are randomly selected. 1.3) Use MiT-b4 pre-trained weights to initialize the encoder, avoiding the problems of poor performance and unclear feature extraction when training from scratch, and building a model that is faster and more accurate.

3. The monocular depth prediction method based on multi-level feature parallel interactive fusion according to claim 1, characterized in that: Step 2) The specific method is: 2.1) CNN and Transformer have unique characteristics and complementarity. In order to fully utilize the advantages of both, a hierarchical interactive fusion network is proposed. 2.2) In the encoder stage, in order to obtain rich global context features and local features, a Transformer and CNN parallel structure is used as the encoder. First, the image size is cropped to 320×320. The SwinTransformer branch and ResNet branch of the encoder are used to obtain feature maps with resolutions of [1 / 4, 1 / 8, 1 / 16, 1 / 32], and the number of channels are [64, 128, 256, 512], respectively. The block size of the SwinTransformer branch is 4, the number of channels is 3, the depth is [2, 2, 6, 2], the number of multi-head attention is [4, 8, 16, 32], and the window size is 10. The features of its four stages are [ft1, ft2, ft3, ft4], respectively. 2.3) Corresponding to the four stages of the ResNet branch [fr1,fr2,fr3,fr4], the proposed fusion module FFM is used to combine the features of different layers and resolutions of Swin Transformer and ResNet to obtain [F1,F2,F3,F4], with the number of channels changing from 512 to 64. Finally, the original resolution is restored through upsampling and convolution operations for dense prediction.

4. The monocular depth prediction method based on multi-level feature parallel interactive fusion according to claim 1, characterized in that: Step 4) The specific method is: To calculate the distance between the predicted output depth and the true ground depth map, the method uses scale-invariant loss (SI) to train the model. here Where the actual ground depth is d i The predicted depth is α = 10, λ = 0.85.

Images