A monocular depth estimation method based on CNN and Transformer feature fusion

Abstract

The present invention belongs to the field of computer vision and discloses a monocular depth estimation method based on CNN and Transformer feature fusion, comprising: first, constructing a mixed depth dataset and preprocessing it, inputting the preprocessed image into the network to extract CNN feature maps, secondly, using a feature conversion module to obtain an image-like feature map and adjusting the scale of the image-like feature map to obtain four Transformer feature maps, and then inputting two feature maps into a guided fusion module to obtain four levels of feature maps and inputting them into a decoding fusion module to obtain a decoded output feature map; finally, inputting the output feature map into a depth output module to generate a depth map, performing loss value calculation on the depth map label and the depth map to obtain a trained model. The dual-branch structure of the present invention improves the global accuracy of the depth map, refines the performance of local textures and edges, and improves the generalization ability of the model.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically relates to a monocular depth estimation method based on CNN and Transformer feature fusion. Background Art

[0002] Monocular depth estimation is a key task in computer vision, aiming to infer the three-dimensional depth information of a scene from a single image. Monocular depth estimation provides the foundation for many applications, such as autonomous driving, 3D reconstruction, and novel viewpoint synthesis. Conventional depth estimation methods rely on multiple cameras or specialized sensors to acquire depth information, while monocular depth estimation requires only a single camera, significantly reducing cost and complexity. It can also be applied to large amounts of existing image and video data, providing richer visual information for understanding complex scenes. The cost-effectiveness and flexibility of this method, driven by deep learning technology, have made it a focus of considerable attention in computer vision research and applications.

[0003] Convolutional neural networks (CNNs) are good at capturing local features and texture information of images, but they are limited in understanding the global context of images. Their ability to represent long-range dependencies is insufficient, resulting in inaccurate global structures of depth maps. In addition, the edges of objects may become blurred due to convolutional neural network processing, resulting in a break in depth continuity.

[0004] Transformers perform well in handling global long-distance dependencies, but may ignore some important local details and lack inductive bias. They may not be as sensitive to position information in the image as convolutional networks, thus affecting the spatial accuracy of the depth map.

[0005] In addition, networks trained on a single dataset usually only cover specific scenarios or environments, resulting in degraded performance in other scenarios. The model may not be able to accurately estimate unseen situations because it has not been exposed to enough diverse data. Summary of the Invention

[0006] To achieve the above objectives, the present invention provides a monocular depth estimation method based on CNN and Transformer feature fusion. This monocular depth estimation method uses a dual-branch structure to combine the advantages of the Transformer network and the CNN network, fully leveraging the advantages of each network and using a hybrid data set for training to produce a higher quality depth map, which not only improves the global accuracy of the depth map, but also refines the performance of local textures and edges.

[0007] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0008] The present invention is a monocular depth estimation method based on CNN and Transformer feature fusion, which includes the following steps:

[0009] Step 1: Construct a mixed depth dataset for training, and perform normalization and data enhancement preprocessing operations on the RGB images and their corresponding depth map labels in the mixed depth dataset;

[0010] Step 2: Input the RGB image preprocessed in step 1 into the network. The network includes an encoding part and a decoding part. The encoding part includes a Transformer branch, a CNN branch, and a guided fusion module. The decoding part includes a decoding fusion module and a depth output module. The Transformer branch processes the RGB image to obtain a one-dimensional Transformer feature map.

[0011] Step 3: The CNN branch processes the RGB image preprocessed in step 1 and extracts four CNN feature maps of different resolutions;

[0012] Step 4: Select any four layers of one-dimensional Transformer feature maps in the Transformer branch, and use the feature conversion module to convert the one-dimensional Transformer feature maps into image-like feature maps, and use transposed convolution to adjust the scale of the image-like feature maps, and finally obtain four Transformer feature maps with a resolution corresponding to the CNN feature map obtained in step 3;

[0013] Step 5: The CNN feature map obtained in step 3 and the four Transformer feature maps obtained in step 4 are input into the guided fusion module to obtain feature maps of four levels, whose resolution is the input RGB image.

[0014] Step 6: Input the feature maps of the four levels obtained in step 5 into the decoding fusion module. The decoding fusion module consists of a residual convolution unit and an upsampling operation. The feature maps of the four levels are fused from bottom to top through skip connections, and the image details are gradually reconstructed to obtain the decoded output feature map with a resolution of 1 / 4 of the input RGB image.

[0015] Step 7: Input the decoded output feature map obtained in step 6 into the depth output module to generate a depth map with the same size as the input image RGB, rich details and high accuracy;

[0016] Step 8: Calculate the loss value of the depth map label of the input RGB image and the depth map predicted in step 7, perform iterative optimization of the network hyperparameters, and obtain a trained model.

[0017] A further improvement of the present invention is that: the mixed depth dataset in step 1 includes the NYU dataset, the ApolloScape dataset, the Town05 dataset, the RedWeb dataset and the HR-WSI dataset, each of the NYU dataset, the ApolloScape dataset, the Town05 dataset, the RedWeb dataset and the HR-WSI dataset consists of an RGB image and a corresponding depth map label, the depth map labels are normalized to the same depth space, and linear normalization is used to make the depth value range between [0,1].

[0018] A further improvement of the present invention is that: Step 5 inputs the CNN feature map and the four Transformer feature maps into the guided fusion module to obtain feature maps of four levels, which specifically includes the following steps:

[0019] Step 5.1: For each guided fusion module, process the Transformer feature map and CNN feature map of the same scale, first concatenate the Transformer feature map and CNN feature map along the channel dimension, and then reduce the channel dimension through 3x3 convolution, BN and RELU activation functions;

[0020] Step 5.2: Use the shuffle attention mechanism to enhance feature representation and fully integrate the Transformer feature map and the CNN feature map.

[0021] Step 5.3: Four levels of feature maps are obtained through four guided fusion modules, where the feature maps of each level have both the global information processing advantages of Transformer and the local feature extraction advantages of CNN.

[0022] A further improvement of the present invention is that the residual convolution unit in step 6 includes a RELU activation function, a 3×3 convolution and a BN, and the specific decoding fusion process includes the following steps:

[0023] Step 6.1. Starting from the feature map output in step 5 with a resolution of 1 / 32 of the input RGB image, the feature map is processed by the first residual convolution unit of the current decoding and fusion module and then added to the feature map output by the previous decoding and fusion module.

[0024] Step 6.2: Pass the second residual convolution unit and upsample it by twice to obtain a feature map with a resolution of 1 / 16 the size of the input RGB image;

[0025] Step 6.3: After three consecutive decoding and fusion modules, the final result is a feature map with a resolution of 1 / 4 of the input RGB image.

[0026] A further improvement of the present invention is that the loss value in step 8 is calculated as follows:

[0027]

[0028] in, represents the predicted depth map, represents the true depth map label, represents the predicted depth value, Represents the true depth value, j represents the index value of the pixel, M represents the effective number of pixels, and R = 0.8M is used to remove 20% of the maximum outliers to reduce the overfitting of the model to abnormal data.

[0029] A further improvement of the present invention is that: in step 2, the Transformer branch includes an image embedding module, 12 consecutive Transformer layers and a feature conversion module, the RGB image is input into the network through the image embedding module to obtain a series of embedding vectors, the output series of embedding vectors are used as input to the 12 Transformer layers, the 12 consecutive Transformer layers process the embedding vectors, and 12 one-dimensional Transformer feature maps are captured through a multi-head self-attention mechanism.

[0030] A further improvement of the present invention is that in step 2, the CNN branch adopts a pre-trained residual network ResNeXt101, and the CNN branch is composed of a plurality of CNN residual blocks stacked together, which is divided into four stages, and the original input image is obtained in each stage. Size feature map.

[0031] The beneficial effects of the present invention are:

[0032] The present invention uses a dual-branch structure to combine the advantages of the Transformer network and the CNN network. First, by processing the input RGB image through the Transformer branch, the accuracy of depth estimation is improved, enabling the model to better understand the overall layout of the scene and the relative position of distant objects; secondly, the CNN branch is good at capturing spatial features when processing RGB images, enhancing the representation of edge details and texture information, which makes the generated depth map more refined and clear in close objects and surface details. This dual-branch feature map fusion method fully utilizes the advantages of each network, not only improving the global accuracy of the depth map, but also refining the representation of local textures and edges, producing a higher quality depth map.

[0033] This paper uses a hybrid training method for a monocular depth estimation network using multiple datasets, leveraging information from different datasets to improve depth estimation performance. The network combines inputs from multiple data sources and learns depth features from different environments, scenes, and objects to infer depth from monocular images in a more comprehensive and robust manner. This approach helps the network generalize better in various practical applications because it can adapt to diverse training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the monocular depth estimation method of the present invention.

[0035] Figure 2 Schematic diagram of the framework of the depth estimation method of the present invention.

[0036] Figure 3 Schematic diagram of the feature conversion module of the present invention.

[0037] Figure 4 Schematic diagram of the guided fusion module of the present invention.

[0038] Figure 5 It is a schematic diagram of the decoding fusion module of the present invention. DETAILED DESCRIPTION

[0039] The following diagrams illustrate embodiments of the present invention. For clarity, many practical details are included in the following description. However, it should be understood that these practical details are not intended to limit the present invention. In other words, in some embodiments of the present invention, these practical details are not essential.

[0040] like Figure 1-2 As shown, the present invention is a monocular depth estimation method based on CNN and Transformer feature fusion, which specifically includes the following steps:

[0041] Step 1: Construct a hybrid depth dataset for training, and perform normalization and data enhancement preprocessing operations on the RGB images and their corresponding depth map labels in the hybrid depth dataset. The hybrid depth dataset of the present invention includes the NYU dataset, the ApolloScape dataset, the Town05 dataset, the RedWeb dataset, and the HR-WSI dataset. Each of the NYU dataset, the ApolloScape dataset, the Town05 dataset, the RedWeb dataset, and the HR-WSI dataset consists of RGB images and corresponding depth map labels. The depth map labels are normalized to the same depth space and linearly normalized so that the depth values ​​range between [0, 1].

[0042] Step 2: Input the RGB image preprocessed in step 1 into the network. The network includes an encoding part and a decoding part. The encoding part includes a Transformer branch, a CNN branch, and a guided fusion module. The decoding part includes a decoding fusion module and a depth output module. The Transformer branch processes the RGB image to obtain a one-dimensional Transformer feature map.

[0043] The Transformer branch includes an image embedding module, 12 consecutive Transformer layers, and a feature conversion module. The structure of the feature conversion module is as follows: Figure 3 As shown in the figure, an RGB image is input into the network through the image embedding module to obtain a series of embedding vectors. The output series of embedding vectors is used as the input of 12 Transformer layers. 12 consecutive Transformer layers process the embedding vectors and capture 12 one-dimensional Transformer feature maps through the multi-head self-attention mechanism.

[0044] The CNN branch uses the pre-trained residual network ResNeXt101. The CNN branch is composed of multiple CNN residual blocks stacked together and divided into four stages. The original input image is obtained in each stage. Size feature map.

[0045] Step 3: The CNN branch processes the RGB image preprocessed in step 1 and extracts four CNN feature maps of different resolutions;

[0046] Step 4: Select any four layers of one-dimensional Transformer feature maps in the Transformer branch. In this embodiment, the third, sixth, ninth, and twelfth layers are selected, and the one-dimensional Transformer feature maps are converted into image-like feature maps using the feature conversion module. The scale of the image-like feature maps is adjusted using transposed convolution, and finally four Transformer feature maps with a resolution corresponding to the CNN feature map obtained in step 3 are obtained.

[0047] Step 5: The CNN feature maps obtained in step 3 and the four Transformer feature maps obtained in step 4 are input into the guided fusion module, as shown in Figure 4 As shown, four levels of feature maps are obtained, whose resolution is the input RGB image The specific steps include:

[0048] Step 5.1: For each guided fusion module, process the Transformer feature map and CNN feature map of the same scale, first concatenate the Transformer feature map and CNN feature map along the channel dimension, and then reduce the channel dimension through 3x3 convolution, BN and RELU activation functions;

[0049] Step 5.2: Use the shuffle attention mechanism to enhance feature representation and fully integrate the Transformer feature map and the CNN feature map.

[0050] Step 5.3: Four levels of feature maps are obtained through four guided fusion modules, where the feature maps of each level have both the global information processing advantages of Transformer and the local feature extraction advantages of CNN.

[0051] Step 6: Input the feature maps of the four levels obtained in step 5 into the decoding fusion module, as shown in Figure 5 As shown in the figure, the decoding fusion module consists of a residual convolution unit and an upsampling operation. The residual convolution unit includes a RELU activation function, a 3×3 convolution, and a BN. It fuses the feature maps of four levels from bottom to top through skip connections, gradually reconstructs the image details, and obtains the decoded output feature map with a resolution of 1 / 4 of the input RGB image.

[0052] The decoding fusion process includes the following steps:

[0053] Step 6.1. Starting from the feature map output in step 5 with a resolution of 1 / 32 of the input RGB image, the feature map is processed by the first residual convolution unit of the current decoding and fusion module and then added to the feature map output by the previous decoding and fusion module.

[0054] Step 6.2: Pass the second residual convolution unit and upsample it by twice to obtain a feature map with a resolution of 1 / 16 the size of the input RGB image;

[0055] Step 6.3: After three consecutive decoding and fusion modules, the final result is a feature map with a resolution of 1 / 4 of the input RGB image.

[0056] Step 7: Input the decoded output feature map obtained in step 6 into the depth output module to generate a depth map with the same size as the input image RGB, rich details and high accuracy;

[0057] Step 8: Calculate the loss value of the depth map label of the input RGB image and the depth map predicted in step 7, perform iterative optimization of the network hyperparameters, and obtain a trained model. The loss value is calculated as follows:

[0058]

[0059] in, represents the predicted depth map, represents the true depth map label, represents the predicted depth value, Represents the true depth value, j represents the index value of the pixel, M represents the effective number of pixels, and R = 0.8M is used to remove 20% of the maximum outliers to reduce the overfitting of the model to abnormal data.

[0060] Figure 2 The convolution branch in can select different types of pre-trained convolutional neural networks, such as EfficientNet, Densenet, etc. Figure 5 The attention mechanism of the fusion module can also be adjusted, such as adopting a more complex or simplified attention model, or trying different attention mechanisms such as spatial attention and channel attention, to better adapt to specific application scenarios and performance requirements.

[0061] This paper adopts a dual-branch hybrid network architecture, combining the advantages of Transformer networks and CNN networks. This structural design not only improves the global accuracy of depth maps, but also refines the representation of local textures and edges. The paper uses a hybrid training of multiple datasets to improve model performance and generalization by integrating information from different data sources.

[0062] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A monocular depth estimation method based on CNN and Transformer feature fusion, characterized by: The monocular depth estimation method specifically comprises the following steps: Step 1: Construct a mixed depth dataset for training, and perform normalization and data enhancement preprocessing operations on the RGB images and their corresponding depth map labels in the mixed depth dataset; Step 2: Input the RGB image preprocessed in step 1 into the network. The network includes an encoding part and a decoding part. The encoding part includes a Transformer branch, a CNN branch, and a guided fusion module. The decoding part includes a decoding fusion module and a depth output module. The Transformer branch processes the RGB image to obtain a one-dimensional Transformer feature map. Step 3: The CNN branch processes the RGB image preprocessed in step 1 and extracts four CNN feature maps of different resolutions; Step 4: Select the four-layer one-dimensional Transformer feature map in the Transformer branch, and use the feature conversion module to convert the one-dimensional Transformer feature map into an image-like feature map, and use transposed convolution to adjust the scale of the image-like feature map, and finally obtain four Transformer feature maps with a resolution corresponding to the CNN feature map obtained in step 3; Step 5: The CNN feature map obtained in step 3 and the four Transformer feature maps obtained in step 4 are input into the guided fusion module to obtain feature maps of four levels, whose resolution is the input RGB image. , wherein the CNN feature map and the four Transformer feature maps are input into the guided fusion module to obtain feature maps of four levels, specifically including the following steps: Step 5.1: For each guided fusion module, process the Transformer feature map and CNN feature map of the same scale, first concatenate the Transformer feature map and CNN feature map along the channel dimension, and then reduce the channel dimension through 3x3 convolution, BN and RELU activation functions; Step 5.2: Use the shuffle attention mechanism to enhance feature representation and fully integrate the Transformer feature map and the CNN feature map. Step 5.3: Obtain four levels of feature maps through four guided fusion modules, where each level of feature maps combines the global information processing advantages of Transformer and the local feature extraction advantages of CNN; Step 6: Input the feature maps of the four levels obtained in step 5 into the decoding fusion module. The decoding fusion module consists of a residual convolution unit and an upsampling operation. The feature maps of the four levels are fused from bottom to top through skip connections, and the image details are gradually reconstructed to obtain the decoded output feature map with a resolution of 1 / 4 of the input RGB image. Step 7: Input the decoded output feature map obtained in step 6 into the depth output module to generate a depth map with the same size as the input image RGB, rich details and high accuracy; Step 8: Calculate the loss value of the depth map label of the input RGB image and the depth map predicted in step 7, perform iterative optimization of the network hyperparameters, and obtain a trained model.

2. The monocular depth estimation method based on CNN and Transformer feature fusion according to claim 1, characterized in that: The mixed depth dataset in step 1 includes the NYU dataset, ApolloScape dataset, Town05 dataset, RedWeb dataset and HR-WSI dataset. Each of the NYU dataset, ApolloScape dataset, Town05 dataset, RedWeb dataset and HR-WSI dataset consists of RGB images and corresponding depth map labels. The depth map labels are normalized to the same depth space and linearly normalized so that the depth value range is between [0,1].

3. The monocular depth estimation method based on CNN and Transformer feature fusion according to claim 1, characterized in that: The residual convolution unit in step 6 includes a RELU activation function, a 3×3 convolution, and a BN. The specific decoding fusion process includes the following steps: Step 6.

1. Starting from the feature map output in step 5 with a resolution of 1 / 32 of the input RGB image, the feature map is processed by the first residual convolution unit of the current decoding and fusion module and then added to the feature map output by the previous decoding and fusion module. Step 6.2: Pass the second residual convolution unit and upsample it by twice to obtain a feature map with a resolution of 1 / 16 the size of the input RGB image; Step 6.3: After three consecutive decoding and fusion modules, the final result is a feature map with a resolution of 1 / 4 of the input RGB image.

4. The monocular depth estimation method based on CNN and Transformer feature fusion according to claim 1, characterized in that: The loss value in step 8 is calculated as follows: ， in, represents the predicted depth map, represents the true depth map label, represents the predicted depth value, Represents the true depth value, j represents the index value of the pixel, M represents the number of valid pixels, and R=0.8M is used to remove 20% of the maximum outliers to reduce the overfitting of the model to abnormal data.

5. The monocular depth estimation method based on CNN and Transformer feature fusion according to any one of claims 1 to 4, characterized in that: In step 2, the Transformer branch includes an image embedding module, 12 consecutive Transformer layers, and a feature conversion module. The RGB image is input into the network through the image embedding module to obtain a series of embedding vectors. The output series of embedding vectors are used as input to the 12 Transformer layers. The 12 consecutive Transformer layers process the embedding vectors and capture 12 one-dimensional Transformer feature maps through a multi-head self-attention mechanism.

6. The monocular depth estimation method based on CNN and Transformer feature fusion according to any one of claims 1 to 4, characterized in that: In step 2, the CNN branch uses the pre-trained residual network ResNeXt101. The CNN branch is composed of a plurality of CNN residual blocks stacked together and is divided into four stages. The original input image is obtained in each stage. Size feature map.

Images