Monocular image depth estimation method based on multi-scale information fusion

Abstract

The invention discloses a monocular image depth estimation method based on multi-scale information fusion, and the method comprises the steps: outputting a first feature map representing local information through a CNN branch, and outputting each second feature map representing global feature correlation at a Transform branch; enhancing the input first feature map and each second feature map from three dimensions of channel, space and cross-scale by using an adaptive fine-grained channel space collaborative gating module to obtain each third feature map; using attention mechanism decoders constructed based on an up-sampling module to sequentially process the input third feature maps to obtain a monocular image depth estimation feature map; according to the invention, the problem that the processing capability of a model on a remote area in a monocular depth estimation scene cannot be improved because local features and global features cannot be effectively combined in the prior art can be solved.

Description

Technical Field

[0001] The present application relates to the technical field of monocular depth estimation methods, and more specifically to a monocular image depth estimation method based on multi-scale information fusion. Background Art

[0002] In practical applications in many fields such as robotics, autonomous driving, and augmented reality, depth maps play a vital role. They can accurately present the 3D geometry of the scene. Therefore, achieving accurate monocular depth estimation is extremely valuable in these practical application scenarios. However, depth estimation is essentially an inverse problem of recovering three-dimensional information from a two-dimensional image, which is highly ambiguous. Especially in complex scenes, the boundary contours of distant objects are often very subtle and difficult to distinguish clearly. Existing depth estimation methods have many drawbacks, such as inaccurate depth prediction of the farthest objects in the scene, and detailed information at the boundary contours of objects is easily lost during network transmission, which ultimately leads to distortion of the scene structure. Therefore, how to effectively combine local and global features to achieve the interaction of contextual information, thereby improving the model's processing ability for depth estimation of distant areas in the scene, has become a key issue that needs to be solved urgently. Summary of the Invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the first aspect of the present application proposes a monocular image depth estimation method based on multi-scale information fusion, including: a dual-branch encoder constructed based on a Transformer branch and a CNN branch extracts features of the image to be estimated, outputs a first feature map representing local information in the CNN branch, and outputs each second feature map representing global feature correlation in the Transformer branch; constructs an adaptive fine-grained channel space collaborative gating module based on an adaptive fine-grained channel interaction module, a spatial attention module and a cross-scale feature fusion module, and uses the adaptive fine-grained channel space collaborative gating module to enhance the input first feature map and each second feature map from the three dimensions of channel, space and cross-scale to obtain each third feature map; uses each attention mechanism decoder constructed based on the upsampling module to process each input third feature map in turn to obtain a monocular image depth estimation feature map.

[0004] Optionally, the dual-branch encoder is constructed based on the Transformer branch and the CNN branch to extract features of the image to be estimated, the CNN branch outputs a first feature map representing local information, and the Transformer branch outputs each second feature map representing global feature correlation, including: constructing a CNN branch based on the first Bottleneck residual block of ResNet50; constructing a Transformer branch based on the patch partitioning module, the application of a linear embedding layer and each Transformer layer connected in sequence; wherein, the first Bottleneck residual block of ResNet50 is used to extract local features of the image to be estimated to obtain a first feature map; the patch partitioning module is used to divide the image to be estimated into multiple non-overlapping patches; the linear embedding layer is used to project multiple non-overlapping patches to a preset dimension to obtain a feature map with unified dimension; the feature map with unified dimension is input into the first layer of each Transformer layer connected in sequence, and the corresponding second feature map is output in each Transformer layer.

[0005] Optionally, the CNN branch includes: a convolutional layer, a batch layer, an activation layer and a maximum pooling layer connected in sequence: wherein the convolution kernel size of the convolutional layer is 7x7, and the activation layer adopts the ReLU activation function; each of the Transformer layers includes: a multi-head self-attention module and a multi-layer perceptron module connected in sequence; the multi-head self-attention module and the multi-layer perceptron module both use residual connection and then perform layer normalization processing.

[0006] Optionally, the adaptive fine-grained channel space collaborative gating module is used to enhance the first feature map and each second feature map input from the three dimensions of channel, space and cross-scale to obtain each third feature map, including: inputting the first feature map and each second feature map into the adaptive fine-grained channel interaction module one by one, and correspondingly obtaining each channel information interaction feature map representing the interaction results of local channel information and global channel information; processing each channel information interaction feature map based on the spatial attention module, and correspondingly obtaining each spatial information interaction feature map representing the spatial multi-semantic information interaction result; processing each channel information interaction feature map based on the cross-scale feature fusion module, and correspondingly obtaining each cross-scale interaction feature map representing the multi-level information interaction result, wherein each spatial information interaction feature map and each spatial information interaction feature map are processed. Figure 1 One-to-one correspondence; the corresponding spatial information interaction feature maps and cross-scale interaction feature maps in each spatial information interaction feature map and each channel information interaction feature map are weighted and summed based on the hyperparameter to obtain a third feature map.

[0007] Optionally, the adaptive fine-grained channel interaction module includes an average pooling layer, a band matrix calculation module, a diagonal matrix calculation module, a correlation matrix calculation module and a weight vector calculation module; the first feature map and each second feature map are input into the adaptive fine-grained channel interaction module one by one, and each channel information interaction feature map representing the interaction result of local channel information and global channel information is obtained, including: inputting the first feature map and each second feature map into the average pooling layer one by one to obtain each pooling channel information matrix; using the band matrix calculation module to process each pooling channel information matrix to obtain each local channel information matrix; using the diagonal matrix calculation module to process each pooling channel information matrix to obtain each global channel information matrix, wherein each local channel information matrix corresponds to each global channel information matrix one by one; based on a one-to-one correspondence Corresponding local channel information matrices and global channel information matrices, calculate each correlation matrix and the transpose of each correlation matrix; extract the row elements and column elements of each correlation matrix, and sum all row elements and column elements of each correlation matrix to obtain each global channel weight; and extract the row elements and column elements of the transpose of each correlation matrix, and sum all row elements and column elements of the transpose of the correlation matrix to obtain each local channel weight, wherein each global channel weight corresponds to each local channel weight one by one; add the corresponding global channel weights and local channel weights in each global channel weight and reconstruct and normalize them to obtain each channel weight; multiply the first feature map and each second feature map by the corresponding channel weights respectively to obtain each channel information interaction feature map.

[0008] Optionally, the spatial attention module is used to process each channel information interaction feature map to obtain each spatial information interaction feature map representing the spatial multi-semantic information interaction result, including: decomposing each channel information interaction feature map along the height dimension and the width dimension respectively to obtain a height sequence structure and a width sequence structure; average pooling is performed on the height sequence structure and the width sequence structure of each channel information interaction feature map to obtain a first 1D sequence structure and a second 1D sequence structure of each channel information interaction feature map; the first 1D sequence structure and the second 1D sequence structure of each channel information interaction feature map are divided into first independent sub-feature groups and second independent sub-feature groups respectively in the channel dimension; each first independent sub-feature group and each second independent sub-feature group are sequentially subjected to convolution operations with different kernel sizes and a depth of 1, splicing operations, normalization operations and activation operations, to obtain a first spatial attention feature map and a second spatial attention feature map respectively; the first spatial attention feature map, the second spatial attention feature map, and the product of each channel information interaction feature map are multiplied to obtain each spatial information interaction feature map corresponding to each channel information interaction feature map.

[0009] Optionally, the cross-scale feature fusion module includes a first branch and a second branch, and the cross-scale feature fusion module processes each channel information interaction feature map to obtain each cross-scale interaction feature map representing the multi-level information interaction results, including: in the first branch, each channel information interaction feature map is subjected to channel adjustment through a convolution layer with a convolution kernel size of 1x1, and each first-scale feature map is obtained; in the second branch, each channel information interaction feature map is sequentially processed by a convolution layer with a convolution kernel size of 1x1 and N RepBlocks to obtain each corresponding second-scale feature map; based on the corresponding first-scale feature maps and the corresponding second-scale feature maps after element-wise addition, each cross-scale interaction feature map corresponding to each channel information interaction feature map is obtained.

[0010] Optionally, each attention mechanism decoder is hierarchically connected; the attention mechanism decoders constructed based on the upsampling module process the third feature maps of the input in turn to obtain a monocular image depth estimation feature map, including: using the attention mechanism decoder of the previous layer to process a third feature map of the input to obtain an initial output feature map, and adding the initial output feature map to the third feature map of the next input to obtain the input of the attention mechanism decoder of the next layer, and so on, using the hierarchically connected attention mechanism decoders to process the third feature map of each input to obtain a monocular image depth estimation feature map.

[0011] Optionally, the upsampling module includes a nearest neighbor sampling layer, a depth-separable convolution layer, a normalization layer, a ReLU layer, and a convolution layer with a convolution kernel size of 1x1.

[0012] In order to achieve the above-mentioned purpose, the present application also provides a monocular image depth estimation device based on multi-scale information fusion, including: an encoder, which is used to extract features of the image to be estimated based on a dual-branch encoder constructed based on a Transformer branch and a CNN branch, output a first feature map representing local information in the CNN branch, and output each second feature map representing global feature correlation in the Transformer branch; a feature collaboration module, which is used to construct an adaptive fine-grained channel spatial collaborative gating module based on an adaptive fine-grained channel interaction module, a spatial attention module and a cross-scale feature fusion module, and use the adaptive fine-grained channel spatial collaborative gating module to enhance the input first feature map and each second feature map from three dimensions of channel, space and cross-scale to obtain each third feature map; a decoder, which is used to use each attention mechanism decoder constructed based on the upsampling module to process the input third feature maps in sequence to obtain a monocular image depth estimation feature map.

[0013] The embodiment of the present application provides a monocular image depth estimation method based on multi-scale information fusion. Compared with the existing technology, its beneficial effects are as follows: a dual-branch encoder constructed based on the Transformer branch and the CNN branch extracts the features of the image to be estimated, the CNN branch outputs the first feature map representing the local information, and the Transformer branch outputs the second feature maps representing the correlation of global features. Combining the advantages of both Transformer and CNN can improve the performance of local feature and global feature extraction of the model, which can achieve more accurate depth estimation, especially for depth estimation of distant objects; an adaptive fine-grained channel spatial collaborative gating is constructed based on the adaptive fine-grained channel interaction module, the spatial attention module and the cross-scale feature fusion module. Module, and uses the adaptive fine-grained channel space collaborative gating module to enhance the first feature map and each second feature map of the input from the three dimensions of channel, space and cross-scale to obtain each third feature map. This application designs an adaptive fine-grained channel space collaborative gating module to fuse the features extracted by the Transformer branch and the CNN branch, which can solve the problem of difference in feature representation between the two at the spatial level and the scale level, so that the decoder can make full use of global and local information; using each attention mechanism decoder constructed based on the upsampling module, the input third feature maps are processed in turn to obtain the monocular image depth estimation feature map. The upsampling module of the decoder can reduce the computational complexity and realize efficient multi-stage feature aggregation and refinement. The use of the upsampling module can also retain channel information and reduce computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To more clearly illustrate the technical solution of this application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0015] Figure 1 Flowchart of a monocular image depth estimation method based on multi-scale information fusion provided in an embodiment of the present application;

[0016] Figure 2 This is a diagram showing the overall network architecture of the monocular image depth estimation method based on multi-scale information fusion provided in an embodiment of the present application;

[0017] Figure 3 This is a structural diagram of the adaptive fine-grained channel space collaborative gating module of the monocular image depth estimation method based on multi-scale information fusion provided in an embodiment of the present application;

[0018] Figure 4Structural diagram of the adaptive fine-grained channel interaction module of the monocular image depth estimation method based on multi-scale information fusion provided in an embodiment of the present application:

[0019] Figure 5 A structural diagram of the spatial attention module and cross-scale feature fusion module of the monocular image depth estimation method based on multi-scale information fusion provided in an embodiment of the present application;

[0020] Figure 6 Schematic diagram of depth estimation results of different technical solutions of the monocular image depth estimation method based on multi-scale information fusion provided in the embodiments of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] This specification provides method operation steps such as embodiments or flowcharts, but may include more or fewer operation steps based on routine or non-inventive work. When implemented in an actual system or server product, the methods shown in the embodiments or figures may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0023] In traditional computer vision technology, monocular depth estimation mainly relies on traditional methods such as stereo vision and structured light (such as lidar).

[0024] Existing monocular depth estimation methods based on convolutional neural networks (CNNs) primarily predict the depth value of each pixel in a single image by training a regression CNN using annotated depths. Some work on monocular depth estimation methods focuses on designing more efficient CNN architectures, such as designing multi-scale networks that combine global coarse depth maps with local fine depth maps. In subsequent research, processing techniques such as conditional random fields (CRFs) have been introduced, as well as new upsampling modules and inverse Huber losses for monocular depth estimation. Other work on monocular depth estimation methods focuses on using multi-scale networks to treat depth estimation as an ordinal regression task. Other existing techniques employ multi-scale feature fusion for monocular depth estimation. The concept of multi-scale feature fusion originated from pyramid representations in image processing. With the development of deep learning, architectures such as Feature Pyramid Networks (FPNs) and Pyramid Scene Parsing Networks (PSPNets) have emerged. For example, some existing techniques employ multi-scale network architectures to process global context and detail information, while others employ structure-aware residual pyramid networks (SRPNs) that combine residual learning with pyramid structures to capture multi-scale depth features. In the prior art, the Transformer architecture in the Transformer-based monocular depth estimation method has a powerful global dependency modeling capability, and the visual converter (ViT) in the Transformer architecture can model the global context thanks to the attention mechanism. Therefore, some work has applied it to monocular depth estimation. In the prior art, there are also hybrid models that use convolutional neural networks and Transformer architectures for monocular depth estimation. For example, in some prior art, Transformer is used as an encoder to provide global features. In some prior art, a hybrid model of convolutional neural networks and Transformer architectures is used in supervised depth estimation algorithms to improve depth prediction accuracy.

[0025] However, existing technologies all suffer from several drawbacks. For example, traditional computer vision techniques are difficult to infer depth directly from a single image. Furthermore, traditional computer vision techniques rely heavily on image geometry and texture information, resulting in poor performance in complex scenes. Depth sensors also increase production costs. Existing monocular depth estimation methods based on convolutional neural networks (CNNs) suffer from the local receptive field of convolution operations. Most networks employ 3×3 convolutions, resulting in a small receptive field and poor global feature extraction. To achieve better results, deeper backbone networks or more complex network architectures are often used, which often results in increased model size and computational complexity. In contrast, Transformer-based monocular depth estimation methods lack a spatial inductive bias when modeling local information, resulting in unsatisfactory performance. Furthermore, the complexity of the Transformer increases the computational cost of training. While multi-scale feature fusion methods can improve accuracy for monocular depth estimation, they often suffer from insufficient interaction between local and global features, leading to inaccurate depth predictions for distant objects.

[0026] In order to solve the above technical problems, the present application proposes a monocular image depth estimation method based on multi-scale information fusion. The method can be executed by a processor in a terminal or server. The method can combine the advantages of both Transformer and CNN to improve the performance of model feature extraction and achieve more accurate depth estimation, especially depth estimation of distant objects. The method also fuses the features extracted by the Transformer branch and the CNN branch through an adaptive fine-grained channel space collaborative gating module CSF to solve the problem of difference in feature representation between the two, so that the decoder can make full use of global and local information. Finally, the present application reduces the computational complexity through an upsampling module, realizes efficient multi-stage feature aggregation and refinement and retains channel information to reduce computational costs.

[0027] The technical solution of this application is described in detail below with reference to the accompanying drawings. Figure 1 This is a flowchart of a method for monocular image depth estimation based on multi-scale information fusion provided in an embodiment of the present application, refer to Figure 1 , the monocular image depth estimation method based on multi-scale information fusion can include:

[0028] S10. A dual-branch encoder constructed based on the Transformer branch and the CNN branch extracts features of the image to be estimated, outputs a first feature map representing local information in the CNN branch, and outputs second feature maps representing global feature correlations in the Transformer branch.

[0029] In an embodiment of the present application, step S10 may include the following execution process:

[0030] S101. Build a CNN branch based on the first Bottleneck residual block of ResNet50.

[0031] Among them, the first Bottleneck residual block of ResNet50 is used to extract the local features of the image to be estimated to obtain the first feature map.

[0032] refer to Figure 2 , specifically, the CNN branches can include:

[0033] Convolutional layer, batch layer, activation layer and max pooling layer connected in sequence:

[0034] Among them, the convolution kernel size of the convolution layer is 7x7, and the activation layer uses the ReLU activation function.

[0035] S102: Based on the sequentially connected patch partition modules, a linear embedding layer and each Transformer layer are applied to construct a Transformer branch.

[0036] Continue to refer Figure 2 , each Transformer layer can include:

[0037] The multi-head self-attention module and the multi-layer perceptron module are connected in sequence.

[0038] Both the multi-head self-attention module and the multi-layer perceptron module use residual connections and then perform layer normalization.

[0039] S103 : Using a patch partitioning module to divide the image to be estimated into multiple non-overlapping patches.

[0040] S104: Project multiple non-overlapping patches to a preset dimension using a linear embedding layer to obtain a feature map with a unified dimension.

[0041] S105. Input a feature map of uniform dimension into the first layer of each sequentially connected Transformer layer, and output a corresponding second feature map in each Transformer layer.

[0042] It should be noted that the encoder composed of the Transformer branch and the CNN branch is the first key component of the network structure in the monocular image depth estimation method based on multi-scale information fusion provided by this application.

[0043] By executing steps S101-S105, the processor can extract image features using an encoder composed of a Transformer branch and a CNN branch, thereby fully extracting long-range correlations, global features, and local features of the target to be detected in the image to be estimated.

[0044] Specifically, for the CNN branch, the processor can call the first Bottleneck residual block of the ResNet encoder to extract the local features in the depth estimation method, that is, the first feature map. This can effectively prevent the low-level features from being washed away by continuous multiplication and reduce the calculation time. Among them, the feature map output by the CNN convolution branch can be expressed as

[0045] refer to Figure 2 For the Transformer branch, the processor first divides the input image I into non-overlapping patches through the patch partition module. The initial features of each patch can be represented as the concatenation of the pixel RGB values. Then, the processor uses the linear embedding layer of the Transformer branch to project the initial features to an arbitrary preset dimension and uses the preset dimension as the input of the first Transformer layer, which can be represented as z o Finally, the processor can extract features using L Transformer layers. Generally speaking, each Transformer layer can be composed of a Multi-Head Self-Attention (MSA) module and a Multi-Layer Perceptron (MLP) module. LayerNorm (LN) can be added after the MSA and MLP modules. Both the MSA and MLP modules use residual connections.

[0046] The above process can be expressed by the following expression:

[0047]

[0048] in, and z l Represent the output features of the MSA and MLP modules of the lth layer, Figure 2 For a network architecture diagram provided in an embodiment of the present invention, please refer to Figure 1 , this application uses 4 Transformer layers for feature extraction, and reassembles the 4 feature maps extracted from the feature as the output of the 4 Transformer branches. The output of the 4 Transformer branches is recorded as in, t n Represents the second feature map output by the nth layer.

[0049] After obtaining the second feature map T output by the Transformer branch and the first feature map G output by the CNN branch, due to the lack of effective information interaction and collaborative working mechanism between the first feature maps and the second feature maps output by the two independent branches of the Transformer branch and the CNN branch, sending the first feature map and the second feature maps directly to the subsequent decoder will lead to a lack of connection between the functions and insufficient aggregation capabilities of local features and global features. In order to make up for this defect, the present application sends the first feature map and the second feature maps to the adaptive fine-grained channel space collaborative gating module CSF for further processing. The adaptive fine-grained channel space collaborative gating module can enhance the first feature map and the second feature map output by the Transformer branch and the CNN branch respectively from three dimensions: channel, space, and cross-scale.

[0050] S20. Construct an adaptive fine-grained channel spatial collaborative gating module based on the adaptive fine-grained channel interaction module, the spatial attention module and the cross-scale feature fusion module, and use the adaptive fine-grained channel spatial collaborative gating module to enhance the first feature map and each second feature map of the input from three dimensions of channel, space and cross-scale to obtain each third feature map.

[0051] The adaptive fine-grained channel space collaborative gating module is the second key component of the network structure in the monocular image depth estimation method based on multi-scale information fusion provided by this application.

[0052] refer to Figure 3 In the monocular depth estimation task, in order to more effectively handle the fusion of local and global features, this application constructs an adaptive fine-grained channel space collaborative gating module. The input of the adaptive fine-grained channel space collaborative gating module is the reassembled five-layer feature. The reassembled five-layer feature includes: the first feature map G output by the CNN branch is the first layer feature, and the four second feature maps T output by the Transformer branch after reassembly are the second, third, fourth, and fifth layer features respectively. The feature output by the encoder is recorded as F.

[0053] Step S20 may include the following execution process:

[0054] S201: Input the first feature map and each second feature map into the adaptive fine-grained channel interaction module one by one, and obtain each channel information interaction feature map corresponding to the interaction result between the local channel information and the global channel information.

[0055] In one embodiment of the present application, the adaptive fine-grained channel interaction module includes an average pooling layer, a band matrix calculation module, a diagonal matrix calculation module, a correlation matrix calculation module, and a weight vector calculation module. Step S201 may include the following execution process:

[0056] S2011. Input the first feature map and each second feature map into the average pooling layer one by one to obtain each pooling channel information matrix.

[0057] S2012. Use a diagonal matrix calculation module to process each pooled channel information matrix to obtain each global channel information matrix, wherein each local channel information matrix corresponds one-to-one to each global channel information matrix.

[0058] S2013. Based on the one-to-one correspondence between each local channel information matrix and each global channel information matrix, calculate each correlation matrix and the transpose of each correlation matrix.

[0059] S2014. Extract the row elements and column elements of each correlation matrix, and sum all the row elements and column elements of each correlation matrix to obtain the corresponding global channel weights.

[0060] In addition, the transposed row elements and column elements of each correlation matrix are extracted, and all the transposed row elements and column elements of the correlation matrix are summed to obtain the corresponding local channel weights, wherein each global channel weight corresponds to each local channel weight one by one.

[0061] S2015. Add the one-to-one corresponding global channel weights and local channel weights, and then reconstruct and normalize them to obtain the channel weights.

[0062] S2016: Multiply the first feature map and each second feature map by the corresponding channel weights to obtain each channel information interaction feature map.

[0063] It's important to note that in monocular depth estimation, local channel interactions help the model focus on feature variations and correlations within a local area. Local channels can also highlight the representation of local details such as object edges or textures, improving the understanding and utilization of local features. Global channel interactions, on the other hand, capture long-range dependencies between channels and integrate global semantic information, which is crucial for localizing distant objects.

[0064] refer to Figure 4The processor of this application first uses the adaptive fine-grained channel interaction module (GLC module) in the adaptive fine-grained channel spatial collaborative gating module (CSF module) to enhance the interaction between the first feature map and each second feature map in the channel dimension. Specifically, the GLC module can effectively enhance features by fusing local channel information with global channel information and reasonably allocating their respective weights, laying the foundation for subsequent feature processing.

[0065] In the specific execution process, in order to obtain channel information from the feature map, the processor converts the feature map F (the first feature map and each second feature map) into each channel information interaction feature map F through the GLC module. C .

[0066] For example, assume that the input feature map F∈R C×H×W , where C is the number of channels, H is the height, and W is the width, the processor can convert the shape of the first feature map and each second feature map from C×H×W to C×1×1 through the average pooling (GAP) operation, thereby obtaining the pooled channel information matrix V∈R C , where the nth channel element V n The calculation method is:

[0067]

[0068] Wherein, the value of n ranges from 1 to C.

[0069] The processor combines V with the matrix B = [b1, b2, ..., b k ] multiplied to obtain the local channel information matrix V lc , the processor combines V with the diagonal matrix D = [d1,d 2, ...,d C ] multiplied to obtain the global channel information matrix V gc The processor converts the global channel information matrix V gc And the local feature information matrix V lc Combined, and through the cross-correlation operation to capture the correlation between the two at different fine-grained levels, the correlation matrix M is obtained. The mathematical expression used in the above process is as follows:

[0070]

[0071] Where k represents the number of adjacent channels.

[0072] In order to accurately assign feature weights, the processor can extract row and column information from the correlation matrix M and the correlation matrix transpose as the global channel weight vector and the local channel weight vector. Specifically, the processor can extract the global channel weight vector after element addition according to the element-wise summation. and the local channel weight vector Get the channel weight W. Finally, the processor multiplies the obtained channel weight W with the first feature map and each second feature map to obtain each channel information interaction feature map F C .

[0073]

[0074] in

[0075] S202. Process each channel information interaction feature map based on the spatial attention module to obtain each spatial information interaction feature map representing the spatial multi-semantic information interaction result.

[0076] In an embodiment of the present application, step S202 may include the following execution process:

[0077] S2021. Decompose the information interaction feature map of each channel along the height dimension and the width dimension respectively to obtain a height sequence structure and a width sequence structure.

[0078] S2022 . Average pooling is performed on the height sequence structure and the width sequence structure of each channel information interaction feature graph to obtain a first 1D sequence structure and a second 1D sequence structure of each channel information interaction feature graph.

[0079] S2023 : Divide the first 1D sequence structure and the second 1D sequence structure of each channel information interaction feature graph into first independent sub-feature groups and second independent sub-feature groups in the channel dimension respectively.

[0080] S2024. Perform convolution operations with different kernel sizes and a depth of 1, splicing operations, normalization operations, and activation operations on each first independent sub-feature group and each second independent sub-feature group, respectively, to obtain a first spatial attention feature map and a second spatial attention feature map respectively.

[0081] S2025. Multiply the first spatial attention feature map, the second spatial attention feature map, and each channel information interaction feature map of each channel information interaction feature map to obtain each spatial information interaction feature map corresponding to each channel information interaction feature map.

[0082] refer to Figure 5 In the first half, in the monocular image depth estimation method based on multi-scale information fusion proposed in this application, the processor first transforms each of the aforementioned channel information interaction feature maps F C Perform global average pooling in the height direction and width direction respectively to obtain two unidirectional 1D sequence structures, namely the first 1D sequence structure F h and the second 1D sequence structure F ω Then, the processor converts the first 1D sequence structure F hDivide into n groups in the channel dimension to obtain the first independent sub-feature groups, and transform the second 1D sequence structure F ω The second independent sub-feature groups are obtained by dividing them into n groups in the channel dimension. In this way, the processor can effectively utilize the multi-semantic information in the spatial dimension and make up for the shortcomings of traditional CNN in spatial feature processing. The mathematical expression of the above process is as follows:

[0083] F h =Pοοl W (F C )∈R Β×C×Η

[0084] F ω =Pοοl Η (F C )∈R Β×C×W

[0085]

[0086] The processor performs a convolution operation on each sub-feature in each first independent sub-feature group and each second independent sub-feature group with a depth of 1 and different convolution kernel sizes to capture different semantic spatial structure information, connects and normalizes the sub-features after the convolution operation in each group, and then uses the Sigmoid function to process the normalized sub-features of each group to generate the first spatial attention feature map Attn H and the second spatial attention feature map Attn W Finally, the processor outputs each channel information interaction feature map and the corresponding first spatial attention feature map Attn H and the second spatial attention feature map Attn W Multiplying them together yields the spatial information interaction feature map. The mathematical expression of the above specific process is as follows:

[0087]

[0088] F S =Attn H ×Attn W ×F C

[0089] S203, based on the cross-scale feature fusion module, the information interaction feature map of each channel is processed to obtain the cross-scale interaction feature map representing the multi-level information interaction results, wherein the spatial information interaction feature map and the cross-scale interaction feature map are Figure 1 One to one correspondence.

[0090] In one embodiment of the present application, based on the cross-scale feature fusion module including a first branch and a second branch, step S203 may include the following execution process:

[0091] S2031. In the first branch, perform channel adjustment on each channel information interaction feature map through a convolution layer with a convolution kernel size of 1x1 to obtain corresponding first-scale feature maps.

[0092] S2032. In the second branch, each channel information interaction feature map is sequentially processed by a convolution layer with a convolution kernel size of 1x1 and N RepBlocks to obtain corresponding second-scale feature maps.

[0093] S2033 : Obtaining cross-scale interaction feature maps corresponding to each channel information interaction feature map based on the corresponding first-scale feature maps and the second-scale feature maps after element-wise addition.

[0094] refer to Figure 5 In the second half, in the monocular image depth estimation method based on multi-scale information fusion proposed in this application, the feature F is fused through two branches. C The processing is performed to achieve the effective fusion of features of different scales, thereby making up for the limitations of the single scale of the features. The first branch of the CCF module in this application processes the information interaction feature map of each channel. The first branch is a 1x1Conv module. In addition to a 1x1Conv module, the second branch also adds N RepBlocks composed of RepConv after the 1x1conv branch for feature fusion. After completing the processing of the two branches, the processor fuses the first scale feature map output by the first branch and the second scale feature map output by the second branch by element-wise addition, and finally outputs each cross-scale interaction feature map F R Assume that the i-th RepBlock module is RepBlock i , the above calculation process can be expressed by the following expression:

[0095] F C21 =RepBlock1(Conv2(F C ))

[0096] F C22 =RepBlock2(F C21 )

[0097] F C2N =RepBlock N (F C2(N-1) )

[0098] F R =Conv1(F C )+F C2N

[0099] S204 , performing weighted summation based on hyperparameters on the corresponding spatial information interaction feature maps and cross-scale interaction feature maps in each spatial information interaction feature map and each channel information interaction feature map to obtain a third feature map.

[0100] It should be noted that each third feature map corresponds to each input first feature map or one of the second feature maps, that is, for the first feature map, it outputs a unique corresponding third feature map.

[0101] The expression used in step S204 can be:

[0102] F * =σ(θ)×F S +(1-σ(θ))×F R

[0103] The one-channel information interaction feature map F output by the SMA module and the CCF module S , a cross-scale interaction feature map F R , use hyperparameters for weighted fusion and output a third feature map F *

[0104] S30. Use each attention mechanism decoder built based on the upsampling module to process each input third feature map in turn to obtain a monocular image depth estimation feature map.

[0105] The attention mechanism decoding is the third key component of this application.

[0106] In an embodiment of the present application, step S30 may include the following execution process:

[0107] The initial output feature map obtained by processing the third feature map of the input by the attention mechanism decoder of the previous layer is element-wise added to the third feature map of the next input to obtain the input of the attention mechanism decoder of the next layer. Similarly, the third feature map of each input is processed by the attention mechanism decoder of the hierarchical connection to obtain the monocular image depth estimation feature map.

[0108] Among them, the upsampling module includes a nearest neighbor sampling layer, a depth-separable convolutional layer, a normalization layer, a ReLU layer, and a convolutional layer with a convolution kernel size of 1x1.

[0109] During the execution of this application, the attention mechanism decoder of this application can be an efficient upsampling module. The main function of the efficient upsampling module EUSB is to efficiently upsample and perform feature enhancement so as to fuse with another third feature map input in the next stage, which is crucial to improving the performance and efficiency of the decoder. The processor of this application first expands the feature map resolution to twice the original through the nearest neighbor upsampling layer, and then applies deep convolution to the upsampled feature map for processing. Deep convolution replaces traditional 3x3 convolution, which can reduce computational complexity and maintain the contextual relationship of the feature map, so that the decoder can achieve efficient multi-stage feature aggregation and refinement at a low computational cost.

[0110] refer to Figure 6 , Figure 6 A schematic diagram of the results of monocular depth prediction obtained by the technical solution of the present application and the comparative technical solution provided in an embodiment of the present application. As can be seen from the figure, the monocular depth estimation method of the present application can more accurately predict depth.

[0111] Based on the above method embodiment, the present application also provides a monocular image depth estimation device based on multi-scale information fusion, which is used to solve the same technical problem as the method claim. The monocular image depth estimation device based on multi-scale information fusion may include an encoder, a feature collaboration module and a decoder, wherein the encoder is used to extract the features of the image to be estimated based on a dual-branch encoder constructed based on a Transformer branch and a CNN branch, output a first feature map representing local information in the CNN branch, and output each second feature map representing the global feature correlation in the Transformer branch. The feature collaboration module is used to construct an adaptive fine-grained channel spatial collaborative gating module based on an adaptive fine-grained channel interaction module, a spatial attention module and a cross-scale feature fusion module, and use the adaptive fine-grained channel spatial collaborative gating module to enhance the input first feature map and each second feature map from the three dimensions of channel, space and cross-scale to obtain each third feature map. The decoder is used to use each attention mechanism decoder constructed based on the upsampling module to process the input third feature maps in sequence to obtain a monocular image depth estimation feature map.

[0112] It should be pointed out that the aforementioned method embodiment has already described in detail the effects of the present application. The device embodiment corresponds to the aforementioned method embodiment and therefore has the same technical effects as the method embodiment, which will not be repeated here.

[0113] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0114] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0115] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A monocular image depth estimation method based on multi-scale information fusion, characterized in that: include: A dual-branch encoder constructed based on the Transformer branch and the CNN branch extracts features of the image to be estimated. The CNN branch outputs a first feature map representing local information, and the Transformer branch outputs second feature maps representing global feature correlations. An adaptive fine-grained channel-spatial collaborative gating module is constructed based on the adaptive fine-grained channel interaction module, the spatial attention module, and the cross-scale feature fusion module. The adaptive fine-grained channel-spatial collaborative gating module is used to enhance the input first feature map and each second feature map from the channel, spatial, and cross-scale dimensions to obtain each third feature map. Using the attention mechanism decoders built based on the upsampling module, the third feature maps of the input are processed in sequence to obtain the monocular image depth estimation feature map.

2. The method for monocular image depth estimation based on multi-scale information fusion according to claim 1, wherein: The dual-branch encoder constructed based on the Transformer branch and the CNN branch extracts features of the image to be estimated, outputs a first feature map representing local information in the CNN branch, and outputs second feature maps representing global feature correlations in the Transformer branch, including: Build a CNN branch based on the first Bottleneck residual block of ResNet50; Build a Transformer branch based on the sequentially connected patch partitioning modules, applying linear embedding layers and each Transformer layer; Among them, the first Bottleneck residual block of ResNet50 is used to extract the local features of the image to be estimated to obtain the first feature map; Use the patch partitioning module to divide the image to be estimated into multiple non-overlapping patches; A linear embedding layer is applied to project multiple non-overlapping patches into a preset dimension to obtain a feature map with uniform dimension. A feature map with uniform dimension is input into the first layer of each sequentially connected Transformer layer, and the corresponding second feature map is output in each Transformer layer.

3. The method for monocular image depth estimation based on multi-scale information fusion according to claim 2, wherein: The CNN branches include: Convolutional layer, batch layer, activation layer and max pooling layer connected in sequence: Among them, the convolution kernel size of the convolution layer is 7x7, and the activation layer uses the ReLU activation function; Each of the Transformer layers includes: A multi-head self-attention module and a multi-layer perceptron module connected sequentially; Both the multi-head self-attention module and the multi-layer perceptron module use residual connections and then perform layer normalization.

4. The method for monocular image depth estimation based on multi-scale information fusion according to claim 1, wherein: The adaptive fine-grained channel-space collaborative gating module is used to enhance the input first feature map and each second feature map from the three dimensions of channel, space and cross-scale to obtain each third feature map, including: Inputting the first feature map and each second feature map into the adaptive fine-grained channel interaction module one by one, and obtaining each channel information interaction feature map representing the interaction result of local channel information and global channel information; Based on the spatial attention module, the information interaction feature maps of each channel are processed to obtain the spatial information interaction feature maps representing the results of spatial multi-semantic information interaction; Based on the cross-scale feature fusion module, the information interaction feature maps of each channel are processed to obtain the cross-scale interaction feature maps representing the multi-level information interaction results. Among them, each spatial information interaction feature map corresponds to each spatial information interaction feature map one by one. The corresponding spatial information interaction feature maps and cross-scale interaction feature maps in each spatial information interaction feature map and each channel information interaction feature map are weighted and summed based on the hyperparameter to obtain a third feature map.

5. The method for monocular image depth estimation based on multi-scale information fusion according to claim 4, wherein: The adaptive fine-grained channel interaction module includes an average pooling layer, a band matrix calculation module, a diagonal matrix calculation module, a correlation matrix calculation module and a weight vector calculation module; The first feature map and each second feature map are input into the adaptive fine-grained channel interaction module one by one, and each channel information interaction feature map representing the interaction result of local channel information and global channel information is correspondingly obtained, including: Input the first feature map and each second feature map into the average pooling layer one by one to obtain the information matrix of each pooling channel; Use the matrix calculation module to process the information matrix of each pooled channel to obtain the corresponding local channel information matrix; Use the diagonal matrix calculation module to process each pooled channel information matrix to obtain each global channel information matrix, where each local channel information matrix corresponds one-to-one to each global channel information matrix; Based on the one-to-one correspondence between each local channel information matrix and each global channel information matrix, each correlation matrix and the transpose of each correlation matrix are calculated; Extract the row elements and column elements of each correlation matrix, and sum all the row elements and column elements of each correlation matrix to obtain the corresponding global channel weights; and extracting the transposed row elements and column elements of each correlation matrix, and summing all the transposed row elements and column elements of the correlation matrix to obtain the corresponding local channel weights, wherein each global channel weight corresponds to each local channel weight one by one; The global channel weights and local channel weights corresponding to each other are added together, and then reconstructed and normalized to obtain the channel weights. Multiply the first feature map and each second feature map by the corresponding channel weights to obtain the channel information interaction feature map.

6. The method for monocular image depth estimation based on multi-scale information fusion according to claim 4, wherein: The spatial attention module is used to process each channel information interaction feature map, and the spatial information interaction feature maps corresponding to the spatial multi-semantic information interaction results are obtained, including: Decompose the information interaction feature map of each channel along the height dimension and width dimension respectively to obtain the height sequence structure and width sequence structure; Average pooling processes the height sequence structure and width sequence structure of each channel information interaction feature map to obtain the first 1D sequence structure and the second 1D sequence structure of each channel information interaction feature map; Divide the first 1D sequence structure and the second 1D sequence structure of each channel information interaction feature graph into first independent sub-feature groups and second independent sub-feature groups in the channel dimension respectively; Perform convolution operations with different kernel sizes and a depth of 1, concatenation operations, normalization operations, and activation operations on each first independent sub-feature group and each second independent sub-feature group, respectively, to obtain a first spatial attention feature map and a second spatial attention feature map respectively; The first spatial attention feature map, the second spatial attention feature map, and the product of each channel information interaction feature map are multiplied to obtain each spatial information interaction feature map corresponding to each channel information interaction feature map.

7. The method for monocular image depth estimation based on multi-scale information fusion according to claim 4, wherein: The cross-scale feature fusion module includes a first branch and a second branch. The cross-scale feature fusion module processes each channel information interaction feature map to obtain each cross-scale interaction feature map representing the multi-level information interaction result, including: In the first branch, each channel information interaction feature map is channel-adjusted through a convolution layer with a convolution kernel size of 1x1 to obtain each first-scale feature map; In the second branch, each channel information interaction feature map is processed in sequence by a convolution layer with a convolution kernel size of 1x1 and N RepBlocks to obtain the corresponding second-scale feature maps; Based on the corresponding first-scale feature maps and the second-scale feature maps after element-wise addition, each cross-scale interaction feature map corresponding to each channel information interaction feature map is obtained.

8. The method for monocular image depth estimation based on multi-scale information fusion according to claim 1, wherein: Connections between the attention mechanism decoder layers; The attention mechanism decoders constructed based on the upsampling module are used to sequentially process the input third feature maps to obtain a monocular image depth estimation feature map, including: The initial output feature map obtained by processing the third feature map of the input by the attention mechanism decoder of the previous layer is element-wise added to the third feature map of the next input to obtain the input of the attention mechanism decoder of the next layer. Similarly, the third feature map of each input is processed by the attention mechanism decoder of the hierarchical connection to obtain the monocular image depth estimation feature map.

9. The method for monocular image depth estimation based on multi-scale information fusion according to claim 1, wherein: The upsampling module includes a nearest neighbor sampling layer, a depth-separable convolution layer, a normalization layer, a ReLU layer, and a convolution layer with a convolution kernel size of 1x1.

10. A monocular image depth estimation device based on multi-scale information fusion, characterized in that: include: An encoder is configured to extract features of the image to be estimated using a dual-branch encoder constructed based on a Transformer branch and a CNN branch, outputting a first feature map representing local information in the CNN branch and second feature maps representing global feature correlations in the Transformer branch; A feature collaboration module is used to construct an adaptive fine-grained channel-spatial collaborative gating module based on the adaptive fine-grained channel interaction module, the spatial attention module, and the cross-scale feature fusion module, and use the adaptive fine-grained channel-spatial collaborative gating module to enhance the input first feature map and each second feature map from the three dimensions of channel, space, and cross-scale to obtain each third feature map; The decoder is used to use the attention mechanism decoders built based on the upsampling module to process the input third feature maps in sequence to obtain the monocular image depth estimation feature map.

Images