A multi-scale attention-based dual-flow monocular depth estimation method

Abstract

The application discloses a kind of based on multi-scale attention dual-flow monocular depth estimation method, comprising: obtaining original monocular graph and the real depth image corresponding to the original monocular graph, based on original monocular graph and real depth image establishes training set and test set, constructs dual-flow monocular depth estimation network based on multi-scale attention, the dual-flow monocular depth estimation network includes dual-flow encoder, multi-scale attention feature fusion unit, guide decoder and vertical attention unit;Loss function of dual-flow monocular depth estimation network is built, trains dual-flow monocular depth estimation network on the training set divided, the gradient value of the network is transferred using back propagation algorithm, repeatedly iterates the network parameter, control predicted depth approximates real depth to obtain best weight, complete the training process of the dual-flow monocular depth estimation network;Image in test set is input to the dual-flow monocular depth network loaded with best weight, and the depth graph corresponding to RGB monocular graph is obtained.

Description

Technical Field

[0001] This invention relates to the field of image depth estimation technology, and in particular to a two-stream monocular depth estimation method based on multi-scale attention. Background Technology

[0002] Monocular depth estimation is a key task in computer vision, with the core idea being the extraction of depth information from a single image. This technique can be extended to downstream tasks such as 3D reconstruction, navigation, and object detection, and has been widely applied in various fields including scene understanding, autonomous driving, depth perception, and robotics. Traditional depth estimation systems typically rely on the extraction of manually generated features or the mining of structural information. For example, some studies utilize optical geometric constraints to recover structures from motion or match them to stereo images. However, these methods are overly reliant on expert prior knowledge, have high computational complexity, and perform poorly in low-texture scenes. In recent years, convolutional neural networks (CNNs) have been increasingly applied to computer vision, leading researchers to attempt to map the nonlinear relationship between real-world scenes and depth images using CNNs, training the neural network by minimizing errors, ultimately achieving the goal of obtaining the corresponding depth map simply by inputting an image into the system. However, these techniques largely ignore small-scale objects and vertical information, making the extracted features susceptible to edge blurring. Summary of the Invention To address the problems existing in the prior art, this invention discloses a dual-stream monocular depth estimation method based on multi-scale attention, specifically including: Obtain the original monocular image and the corresponding real depth image. Establish training and testing sets based on the original monocular image and the real depth image, and perform data augmentation processing on the original monocular image in the training set. A dual-stream monocular depth estimation network based on multi-scale attention is constructed. The dual-stream monocular depth estimation network includes a dual-stream encoder, a multi-scale attention feature fusion unit, a guided decoder, and a vertical attention unit. Construct the loss function of the two-stream monocular depth estimation network, train the two-stream monocular depth estimation network on the divided training set and update the network parameters, use the backpropagation algorithm to pass the gradient value of the network, iterate the network parameters repeatedly, control the predicted depth to approach the true depth to obtain the optimal weight, and complete the training process of the two-stream monocular depth estimation network. The images in the test set are input into a two-stream monocular depth estimation network with optimal weights to obtain the depth map corresponding to the RGB monocular image.

[0003] Furthermore, the loss function of the dual-stream monocular depth estimation network is expressed as follows:

[0004] in , Represents the true value of depth. This represents the network's predicted depth value. Indicates the weight value. It is a constant. The output of the loss function, This represents the number of pixels with valid truth values.

[0005] Furthermore, the dual-stream encoder includes two weight-shared aggregated residual transform deep network branches, which are respectively input into original monocular images of different sizes to extract features at two different scales.

[0006] Furthermore, the multi-scale attention feature fusion unit includes a self-attention dilated convolution module and an attention feature fusion module, wherein the self-attention dilated convolution module is composed of a self-attention mechanism and a dilation rate. The dilated convolutional layers are constructed to extract multi-scale features; the attention feature fusion module uses two pooling operations to fuse multi-scale information.

[0007] Furthermore, the guided decoder includes multiple guided recovery units, each of which includes an upsampling layer, a channel attention mechanism, and a feature refinement unit. The feature refinement unit includes a convolutional layer, an attention mechanism, a batch normalization layer, and a ReLU activation function. The guided decoder restores depth features by incorporating the features output by the residual block in the dual-stream encoder into multiple guided recovery units.

[0008] Furthermore, the guided recovery unit performs a global average pooling operation on the input image in the channel attention mechanism to obtain a feature map. A one-dimensional convolutional layer is used to perform convolution operations on it, and then the convolution result is input into a sigmoid activation function to obtain the feature map after weight redistribution:

[0009] Where C1D represents a one-dimensional convolutional layer, and the size of the adaptive convolutional kernel is... The adaptive selection of convolutional kernels is achieved through... The mapping relationship between the channel number and the channel number is achieved, and the mapping formula is:

[0010] in Represents a mapping function. The dimension of the input channels. Indicates closest In this invention, the number, and They are constants.

[0011] The vertical attention unit obtains the vertical features of the input image: it uses an average pooling layer to reduce the dimensionality of the input image and uses two convolutional layers to perform different mappings to obtain the features. and The features to be reshaped After transpose and features Multiply the products and apply a normalized exponential function operation to calculate the vertical attention map, which is then used to enhance the mapped features.

[0012] Let the output of the guided decoder be a feature. The vertical attention unit first focuses on the features Horizontal pooling is performed to obtain vertical features. ,

[0013] This represents the average pooling layer along the horizontal direction, with vertical features. The input is fed into two convolutional layers of the same kernel size and mapped differently to obtain features. and ,in This indicates the factor by which the feature channels are reduced in dimensionality. After performing the transpose operation and the feature Perform matrix multiplication, and feed the result into a normalized exponential function to obtain the vertical attention map:

[0014] Among them, symbols Represents matrix multiplication. ,in This represents the normalized exponential function; Use convolutional layers to process the initial input Mapped to Using vertical attention maps Enhance the mapped features:

[0015] in, This represents the attention coefficient, which increases throughout the training process.

[0016] By employing the above technical solution, the dual-stream monocular depth estimation method based on multi-scale attention provided by this invention has the following advantages: The network encoding unit in the two-stream monocular depth estimation system includes two ResNeXt-101 branches with shared weights and a multi-scale attention feature fusion unit. The two-stream residual architecture can extract image features at two different scales, while the multi-scale attention feature fusion unit ensures comprehensive extraction of local features and global features with long-range dependencies, capturing multi-scale information and promoting information exchange between features. The encoding stage extracts multi-scale local details and global contours, laying a solid foundation for depth information reconstruction in the decoding stage.

[0017] The decoding unit employs a guided decoder, where the guided recovery unit integrates residual layer output information from the encoder stage layer by layer, accurately and effectively assisting in the reconstruction of depth information. Furthermore, the guided recovery unit introduces a channel attention mechanism, capturing local response features of cross-channel interactions without dimensionality reduction, promoting information exchange and fusion between multi-branch structures carrying encoded information. The secondary combination of the encoding and decoding units fully utilizes the guiding information and, through a clever combination of the attention mechanism and upsampling layers, extracts detailed depth features.

[0018] Vertical depth often changes drastically, therefore, vertical prior knowledge can significantly improve the efficiency of depth estimation. The bi-stream monocular depth estimation system constructed in this invention inserts a vertical attention mechanism at the end of the depth feature extraction network, enabling the acquisition of depth information from the vertical angle and enhancing feature extraction. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the overall process of the system in the embodiment of the present invention; Figure 2 This is a schematic diagram of the architecture of the monocular depth estimation network in this invention; Figure 3 This is a schematic diagram of the multi-scale attention feature fusion unit in this invention; Figure 4 This is a schematic diagram of the boot recovery unit in this invention; Figure 5 This is a schematic diagram of the ECA-Net unit for the channel attention mechanism in this invention; Figure 6 This is a schematic diagram of the vertical attention unit in this invention; Figure 7This is a comparison chart of experimental results on the NYU_Depth_V2 dataset in this invention; Figure 8 This is a comparison chart of experimental results on the KITTI dataset in this invention. Detailed Implementation

[0021] To make the technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention: To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] like Figure 1 The method shown here is a two-stream monocular depth estimation method based on multi-scale attention, which specifically includes the following steps: S1. Obtain and divide the dataset, which includes a training set and a test set consisting of original monocular images and their corresponding real depth images, and perform data augmentation on the original images in the training set.

[0024] In practice, training and test sets are obtained from the NYU_Depth_v2 and KITTI datasets. The NYU_Depth_v2 dataset contains 120,000 RGB-depth image pairs with an image resolution of 480. 640. In this embodiment, 24,000 images were sampled to form the training set, and the method performance was evaluated on 654 test set images. The KITTI dataset consists of 61 scenes, totaling 93,000 RGB-depth image pairs, with an image resolution of approximately 1240. 374. In this embodiment, 23,000 images were sampled for training, and 697 images were used as a test set to evaluate the proposed model. During implementation, data augmentation operations were performed on the training set, including horizontal flipping, random rotation, and color enhancement (such as random gamma correction, brightness adjustment, and color shift) with a probability of 50% on the input images.

[0025] S2. Construct a multi-scale attention dual-stream monocular depth estimation network, wherein the monocular depth estimation network includes a dual-stream encoder, a multi-scale attention feature fusion unit, a guided decoder, and a vertical attention unit. Figure 2 The overall structure of a two-stream monocular depth estimation network based on multi-scale attention is shown.

[0026] The encoding part consists of a two-stream encoder and a multi-scale attention feature fusion unit. In this embodiment, the two-stream encoder includes two weight-shared ResNeXt-101 networks, and the structural details of the multi-scale attention feature fusion unit are as follows. Figure 3 As shown, it uses void ratios of... The system employs self-attention dilated convolutional layers and attention feature fusion units with two types of pooling layers. Specifically, the original monocular image is first fed into a two-stream encoder to extract features. Then, the outputs of the two branches are fed into four self-attention dilated convolutional layers with different dilation rates for further feature enhancement. An attention feature fusion module then fuses the enhanced four branches. This enhancement and fusion process is repeated three times, ultimately outputting the multi-scale features with long-range dependencies extracted by the encoder.

[0027] Among them, the self-attention dilated convolutional layer combines the advantages of multi-scale and long-range attention, enabling it to focus on information from different regions and receptive domains, thereby improving the effectiveness of feature extraction. Specifically, it first extracts information at location... In pixels The local area centered on this area is taken as... The system uses a storage block within a spatial range. Then, the pixel output is obtained through a single-head attention mechanism. .

[0028]

[0029] in, Represents pixels coordinates and These are the dimensions of the input and output, respectively. , and It represents The weighted transformations of the surrounding pixels are referred to as the query, key, and value, respectively. Local self-attention integrates spatial information at each pixel by using a convex combination of content-interaction-parameterized value vectors and hybrid weights.

[0030] During implementation, features The output is then fed into dilated convolutions to extract multi-scale information. The dilation rate of the dilated convolutions in the model is set to 3, 6, 12, or 18. It is worth noting that the dilation rate... The dilated convolution inserts between consecutive filtered values. The zeros will reduce the size of the convolution kernel from... Extended to However, it does not introduce additional computation or parameters. Dilated convolution can be defined as:

[0031] in, For having A convolution kernel with an expansion factor of 1, This represents the activation function. During convolution operations, the dilation rate is introduced... This increases the distance between pixels in the feature map, expands the receptive field of the convolution kernel, and thus enables the extraction of more extensive regional information.

[0032] The decoding section includes a guided decoder and a vertical attention unit. For example... Figure 2 As shown, the bootloader decoder contains multiple boot recovery units. The internal structure of the boot recovery unit is as follows: Figure 4 As shown, it consists of an upsampling layer, a feature refinement unit, and a channel attention mechanism (ECA-Net). The feature refinement unit includes convolutional layers, an attention mechanism, a batch normalization layer, and a ReLU activation function. In implementation, ECA-Net is an improvement on SE-Net. SE-Net redistributes feature weights through squeezing and activation operations, capturing local response features. However, dimensionality reduction inevitably disrupts the direct correspondence between channels and their weights. Therefore, ECA-Net avoids dimensionality reduction through fast one-dimensional convolutional operations, aggregating features and promoting information interaction between channels, ultimately achieving weight recalibration. To more accurately capture depth information, the guided recovery unit fuses the output of each residual block in the ResNeXt model during the encoding stage, guiding the restoration of depth information during the decoding stage through features extracted by the encoder. In implementation, two feature refinement units are concatenated and fed into the channel attention mechanism (ECA-Net) for weight redistribution. The structure of ECA-Net is as follows. Figure 5 As shown. Specifically, the input image is first subjected to global average pooling to obtain the feature map. Then, a one-dimensional convolutional layer is used to perform convolution operations on it (the kernel size is selected adaptively), and finally the convolution result is fed into the sigmoid activation function to obtain the feature map after weight redistribution:

[0033] Where C1D represents a one-dimensional convolutional layer, and the size of the adaptive convolutional kernel is... The adaptive selection of the convolutional kernel is achieved through... The mapping relationship between the channel number and the channel number is achieved, and the mapping formula is:

[0034] In the formula Represents a mapping function. The dimension of the input channels. Indicates closest The number. In this invention, and Set them to 2 and 1 respectively.

[0035] Vertical attention units can capture dramatic changes in depth in the vertical direction. They are applied at the end of the network structure to improve the efficiency of depth estimation by utilizing vertical prior knowledge. Figure 6 The structural details of the vertical attention unit are shown, specifically for the output features of the guided decoder. First, perform horizontal pooling to obtain vertical features. , This represents the average pooling layer along the horizontal direction.

[0036]

[0037] Then, F is input into two convolutional layers with a kernel size of 1×1 for different mappings to obtain the features. and ,in This indicates the factor by which the feature channels are reduced in dimensionality. After performing the transpose operation and the feature Perform matrix multiplication, and feed the result into a normalized exponential function to obtain the vertical attention map:

[0038] Among them, symbols Represents matrix multiplication. (in ) represents the normalized exponential function.

[0039] Subsequently, a convolution kernel size of [size missing] was used. The convolutional layer will initialize the input Mapped to Finally, vertical attention maps are used. To enhance the mapped features:

[0040] in, This represents the attention coefficient, which gradually increases throughout the training process.

[0041] S3. Construct the loss function for the network, train the bi-stream monocular depth estimation network on the pre-defined training set, and update the parameters of the proposed network using the loss function. Specifically, the parameter update involves using the backpropagation algorithm to propagate gradients, iterating repeatedly to reduce errors, making the predicted depth approximate the true depth, and ultimately obtaining the optimal weight model.

[0042] In this embodiment, the loss function is:

[0043] in, , and These represent the ground truth depth value and the model-predicted depth value, respectively. Weight values. and constant Set them to 0.85 and 10 respectively. The output of the loss function, This represents the number of pixels with valid truth values.

[0044] S4. Load the optimal weight model obtained in step S3, and use the dual-stream monocular depth estimation network based on multi-scale attention constructed in S2 to extract depth information of the test set.

[0045] In the specific implementation process, experiments were conducted on the monocular depth estimation model designed within the system on the NYU_Depth_V2 and KITTI datasets. For example... Figure 7 and Figure 8 As shown, the proposed design method can extract more details and edge information on any dataset, resulting in a richer depth information map.

[0046] To facilitate implementation by those skilled in the art, the hardware platform used in this invention is an NVIDIA GeForce RTX 4090 GPU, and the software platform is the PyTorch deep learning framework. During training, the initial learning rate is... Adam is used to optimize network parameters, where , , The batch size is 6 and the number of training generations is 80.

[0047] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A dual-stream monocular depth estimation method based on multi-scale attention, characterized in that... include: Obtain the original monocular image and the corresponding real depth image. Establish training and testing sets based on the original monocular image and the real depth image, and perform data augmentation processing on the original monocular image in the training set. A dual-stream monocular depth estimation network based on multi-scale attention is constructed. The dual-stream monocular depth estimation network includes a dual-stream encoder, a multi-scale attention feature fusion unit, a guided decoder, and a vertical attention unit. Construct the loss function of the two-stream monocular depth estimation network, train the two-stream monocular depth estimation network on the divided training set and update the network parameters, use the backpropagation algorithm to pass the gradient value of the network, iterate the network parameters repeatedly, control the predicted depth to approach the true depth to obtain the optimal weight, and complete the training process of the two-stream monocular depth estimation network. The images in the test set are input into a two-stream monocular depth estimation network with optimal weights to obtain the depth map corresponding to the RGB monocular image; The dual-stream encoder includes two weight-shared deep network branches with aggregated residual transform. The two deep network branches are respectively input into original monocular images of different sizes to extract features at two different scales. The multi-scale attention feature fusion unit includes a self-attention dilated convolution module and an attention feature fusion module, wherein the self-attention dilated convolution module is composed of a self-attention mechanism and a dilation rate. The dilated convolutional layers are constructed to extract multi-scale features; The attention feature fusion module utilizes two pooling operations to fuse multi-scale information; The guided decoder includes multiple guided recovery units, each of which includes an upsampling layer, a channel attention mechanism, and a feature refinement unit. The feature refinement unit includes a convolutional layer, an attention mechanism, a batch normalization layer, and a ReLU activation function. The guided decoder restores depth features by incorporating the features output by the residual block in the dual-stream encoder into multiple guided recovery units. The vertical attention unit obtains the vertical features of the input image: it uses an average pooling layer to reduce the dimensionality of the input image and uses two convolutional layers to perform different mappings to obtain the features. and The features to be reshaped After transpose and features Multiply the products and apply a normalized exponential function operation to calculate the vertical attention map, which is then used to enhance the mapped features.

2. The dual-stream monocular depth estimation method based on multi-scale attention as described in claim 1, characterized in that: The loss function of the dual-stream monocular depth estimation network is expressed as follows: in , Represents the true value of depth. This represents the network's predicted depth value. Indicates the weight value. It is a constant. The output of the loss function, This represents the number of pixels with valid truth values.

3. The dual-stream monocular depth estimation method based on multi-scale attention as described in claim 1, characterized in that: The guided recovery unit performs a global average pooling operation on the input image in the channel attention mechanism to obtain a feature map. A one-dimensional convolutional layer is used to perform convolution operations on it, and then the convolution result is input into the sigmoid activation function to obtain the feature map after weight redistribution: Where C1D represents a one-dimensional convolutional layer, and the size of the adaptive convolutional kernel is... The adaptive selection of convolutional kernels is achieved through... The mapping relationship between the channel number and the channel number is achieved, and the mapping formula is: in Represents a mapping function. The dimension of the input channels. Indicates closest In this invention, the number, and They are constants.

4. The dual-stream monocular depth estimation method based on multi-scale attention as described in claim 1, characterized in that: [The following is a partial translation of the original text, which is not directly related to the previous sentence.] The output of the guided decoder is a feature The vertical attention unit first focuses on the features Horizontal pooling is performed to obtain vertical features. This represents the average pooling layer along the horizontal direction, with vertical features. The input is fed into two convolutional layers of the same kernel size and mapped differently to obtain features. and ,in This indicates the factor by which the feature channels are reduced in dimensionality. After performing the transpose operation and the feature Perform matrix multiplication, and feed the result into a normalized exponential function to obtain the vertical attention map: Among them, symbols Represents matrix multiplication. ,in This represents the normalized exponential function; Use convolutional layers to process the initial input Mapped to Using vertical attention maps Enhance the mapped features: in, This represents the attention coefficient, which increases throughout the training process.

Images