Monocular depth estimation method based on CNN-Transform hybrid architecture

Abstract

The invention discloses a monocular depth estimation method based on hybrid CNN-Transformer, belongs to the aviation field and the computer vision field, and is applied to the field of environment depth perception in low-altitude flight of an unmanned aerial vehicle. Aiming at the problem that local-global information cannot be effectively extracted by a current depth estimation method, the design content mainly comprises the following steps: improving a visual Transform architecture based on an additive attention mechanism, optimizing feature interaction operation between image blocks, and improving the capability of capturing global semantic information; the method comprises the following steps: designing a deep network encoder mixed with a CNN-Transform module, and effectively extracting local-global feature information; a channel attention mechanism is introduced, and the feature fusion capability of the decoder is enhanced. The method is simple in process, an excellent test result is obtained through training on the flight data set Mid-Air of the low-altitude unmanned aerial vehicle, the extraction capability of local-global feature information is effectively enhanced, and the depth perception accuracy of the monocular depth estimation method in the flight scene of the low-altitude unmanned aerial vehicle is improved.

Description

Technical Field

[0001] The present invention relates to the fields of aviation industry and computer vision, and in particular to a monocular vision depth estimation method in a low-altitude flight scene of an unmanned aerial vehicle. Background Art

[0002] With the rapid development of UAV technology, it plays an important role in many fields such as agricultural monitoring, logistics distribution, geographic mapping, emergency rescue, etc. However, when performing low-altitude flight missions, UAVs face complex environments and changeable flight conditions, which puts higher requirements on the depth perception ability of UAVs. Depth estimation, as one of the key technologies for autonomous navigation and obstacle avoidance of UAVs, plays a vital role in the fields of UAVs, autonomous driving, saliency monitoring and three-dimensional reconstruction. Although traditional depth perception methods, such as lidar, can provide accurate depth information, their disadvantages such as high cost, large size and susceptibility to external environmental factors limit their application in miniaturized and low-cost devices such as UAVs. Monocular depth estimation technology uses a single image to simulate the ability of humans to perceive depth through visual cues. It has the advantages of simple hardware and low cost, and has become the preferred solution for practical applications.

[0003] In recent years, monocular visual depth estimation methods based on convolutional neural networks (CNNs) have made significant progress. CNNs effectively process image data through their local feature extraction capabilities, which has promoted the development of depth estimation technology. Subsequently, after the success of the Transformer architecture in the field of natural language processing, it was also introduced into the field of computer vision to solve the problem of depth estimation. However, the CNN-based method has limited ability to perceive global feature information, resulting in deficiencies in the model's spatial relationship features and the integrity of semantic information; the visual Transformer-based method has repeated redundancy problems in the feature matching calculation between image blocks, and the accuracy of capturing local object details is reduced.

[0004] In order to solve these problems, CNN and Transformer can be combined to provide a method for monocular depth estimation that can extract local-global feature information. By improving the visual Transformer module based on the additive attention mechanism to optimize the feature interaction operation mode, the hybrid CNN-Transformer module is designed from scratch to improve the ability to obtain local-global features, and the channel attention mechanism is introduced to deeply fuse the features, so as to improve the depth perception performance of self-supervised monocular depth estimation in complex flight scenes of low-altitude UAVs. Summary of the invention

[0005] The problem to be solved by the present invention is to provide a monocular depth estimation method based on a CNN-Transformer hybrid architecture to solve the problem that the pure CNN method cannot perceive global feature information and the method based on visual Transformer is limited in its ability to capture local information.

[0006] To solve the above problems, the present invention provides a monocular depth estimation method based on a CNN-Transformer hybrid architecture, and the specific steps are as follows:

[0007] Step 1: Build a self-supervised monocular depth estimation network;

[0008] Step 2: Build a Transformer architecture based on the improved additive attention mechanism;

[0009] Step 3: Construct a deep network encoder for monocular depth estimation based on a CNN-Transformer hybrid architecture.

[0010] Step 4: Construct a self-supervised monocular depth estimation deep network decoder that introduces a channel attention mechanism;

[0011] Step 5: Build a self-supervised monocular depth estimation posture network with a lightweight architecture as the backbone network;

[0012] Step 6: Construct a total loss function with the view reconstruction loss function and the smoothness loss function as the core;

[0013] Step 7: Select a suitable data set to train the network, and save the optimal network model after the loss is stable;

[0014] Step 8: Input the test data set into the optimal network model to obtain the final depth estimation prediction map.

[0015] Preferably, in step 1, the constructed self-supervised monocular depth estimation network includes a deep network, a posture network and a loss function. The deep network includes an encoder based on a CNN-Transformer hybrid architecture, a Transformer structure based on an improved additive attention mechanism, an EC Block combined convolution module, a CT Block hybrid module, and a channel attention mechanism. The loss function includes a minimum photometric reprojection error loss function and a smoothness loss function.

[0016] Preferably, in step 2, the Transformer module based on the improved additive attention mechanism uses multiplication operations, only performs operations on the query and the key, sets a global attention query vector to simplify the interactive operation of feature information between image blocks, and learns the hidden representation between image blocks. Taking the feature map x as input, only two matrices W are used qand W k , W q , Transform the input to generate query matrix Q and key matrix K, Q, Then, the query matrix Q is combined with a learnable parameter vector Multiply them together to learn the attention weight of the query. This process finally generates the global attention query vector Q att , d represents the dimension, and the specific formula is shown below:

[0017]

[0018] The learned attention query vector is further processed by performing a pooling operation on the query matrix to aggregate information and generate a single global query vector γ, as shown below:

[0019]

[0020] Therefore, the output x′ of the improved additive attention mechanism is as follows:

[0021]

[0022] in represents the standardized query matrix, F Linear Represents a linear transformation.

[0023] Preferably, in step 3, the self-supervised monocular depth estimation deep network encoder based on the CNN-Transformer hybrid architecture performs local-global feature extraction on the input image in four stages, and each stage is composed of the same hybrid CNN-Transformer module. Among them, CT Block is used to learn local-global information, and EC Block is used to optimize the feature mixing method. The encoder based on the CNN-Transformer hybrid module first inputs the feature map into the combined convolution module in the hybrid CNN-Transformer module of the first stage to optimize the feature mixing method, and then inputs it into the hybrid module to extract local-global feature information. By analogy, the second, third and fourth stages all extract multi-scale feature information through the same module.

[0024] Preferably, in step 3, the combined convolution module uses deep convolution, point-by-point convolution, normalization layer and GeLU activation function to improve the semantic representation ability of the image block. First, the feature map is input into the deep convolution and point-by-point convolution. This design enables the model to improve performance without increasing parameters and delays. Normalization is performed between the two layers of convolution. Subsequently, the feature map is nonlinearly transformed by the GELU activation function and then sent to the point-by-point convolution again to extract features. Finally, a jump connection is implemented between the obtained feature map and the original input feature map to output the feature map.

[0025] Preferably, in step 3, the hybrid module divides the network structure into a CNN local feature representation module and a Transformer global feature representation module. The CNN local feature representation module uses continuous depth convolution and point-by-point convolution operations to achieve accurate extraction of local features to obtain a feature map. The Transformer global feature representation module learns the global information of the feature map through a Transformer structure based on an improved additive attention mechanism. Finally, the output of the local feature representation module and the output of the global feature representation module are jointly input into the linear processing layer to obtain the final output.

[0026] Preferably, in step 4, a channel attention mechanism is used in the decoder of the self-supervised monocular depth estimation deep network based on the CNN-Transformer hybrid architecture, and the implementation of the mechanism mainly includes two steps, information compression and excitation. First, the size of the feature map needs to be converted by a convolution operator. Subsequently, the feature map of each channel is compressed in the spatial dimension through a global average pooling operation to form a descriptor for capturing global spatial information, and then a fully connected layer containing a ReLU activation function and a fully connected layer containing a Sigmoid activation function are used to generate the weight of each channel, and finally the size of the feature map is resized by a scale factor and a weight vector to obtain the final output.

[0027] Preferably, in step 5, the posture network for self-supervised monocular depth estimation based on the CNN-Transformer hybrid architecture takes the image sequence as input, uses ReaNet18 as a lightweight encoder with a backbone network to extract image features, predicts the camera's motion posture through upsampling and fusion, and reprojects the source view to the target view based on the predicted posture information, and outputs a relative posture with six degrees of freedom.

[0028] Preferably, in step 6, the view reconstruction loss function is composed of a minimum photometric error loss function and a pixel mask. The minimum value of the reprojection errors of all source images is selected instead of the average value to calculate the photometric error, so as to solve the problem that the average reprojection of multiple frames of view leads to blurred edges of the predicted image. The pixel mask μ is introduced to filter out pixels with the same appearance in the two frames before and after the sequence, thereby avoiding the adverse effects of static pixels on image prediction and reducing the structural error in image prediction. The view reconstruction loss function is as follows:

[0029]

[0030] In the formula, F pre is the photometric reprojection function; α = 0.85 is taken empirically; SSIM is the structural similarity calculation between two images; L p is the minimum photometric reprojection error loss; μ is the pixel mask; L ss For the view reconstruction loss function. In order to optimize the smoothness of the inverse depth map, a gradient-based edge smoothness loss is introduced, and the formula is as follows:

[0031]

[0032] In the formula, represents the average normalized inverse depth. The multi-scale view reconstruction loss L ss and smoothness loss L sm The average value of is used as the overall loss function L to guide model training t , the specific formula is as follows:

[0033]

[0034] Where: n is the number of output layers, here it is 4; s is the scale of the depth decoder output, respectively λ is 10 -3 .

[0035] Preferably, in step 7, the Mid-Air low-altitude UAV flight data set is selected for training verification and testing, the hardware platform used is a single NVIDIA GeForce RTX 3090 24GB GPU, the operating environment is Ubuntu 20.04 operating system, CUDA version 11.7, and the Adam optimizer is used to optimize the network parameters, and the learning rate is set to 1×10 -4 , the batch size (bach size) is 16, the total training cycle (epoch) is set to 30 rounds, and the performance verification is performed once every iteration to save the model. During the detection process, the test image is input to verify the performance of the method and save the experimental results.

[0036] The beneficial effects of the present invention are: (1) Designing a Transformer architecture based on an improved additive attention mechanism effectively optimizes the way the model obtains global information, reduces model parameters and computational complexity, and achieves performance improvement; (2) Designing a deep network encoder of a hybrid CNN-Transformer module to effectively extract local-global feature information and improve model performance; (3) Introducing the SE channel attention mechanism effectively enhances the decoder's ability to fuse key information, further improving the model's ability to predict deep information. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the method of the present invention

[0038] Figure 2 The overall structure diagram of the present invention is

[0039] Figure 3 Schematic diagram of the improved additive attention mechanism of the present invention

[0040] Figure 4 EC Block combined convolution module diagram of the present invention

[0041] Figure 5 The CT Block hybrid module diagram of the present invention is

[0042] Figure 6 This is the predicted depth map of the present invention on the test data set DETAILED DESCRIPTION

[0043] In order to make the contents of the present invention clearer and easier to understand, the contents of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention and the accompanying drawings.

[0044] To solve the problem of insufficient local-global feature information extraction in depth estimation tasks, the present invention provides a hybrid CNN-Transformer self-supervised monocular depth estimation method, such as Figure 1 The specific process is as follows:

[0045] Step S10 constructs a self-supervised monocular depth estimation network;

[0046] Specifically, based on the characteristics of the depth estimation task, the self-supervised monocular depth estimation network consists of three parts: the deep network, the posture network, and the loss function. The deep network consists of a multi-layer multi-scale encoder with a hybrid CNN-Transformer architecture as the backbone and a decoder adapted to it. The deep network encoder includes an EC Block combined convolution module (such as Figure 4 As shown) and CT Block hybrid modules (as Figure 5As shown in Figure 1, the CT Block module contains a Transformer architecture based on an improved additive attention mechanism; each layer of the deep network decoder introduces a channel attention mechanism to achieve full feature fusion. The posture network uses a lightweight ResNet18 as the backbone and outputs a relative posture with six degrees of freedom through a posture transformation module. Figure 2 shown.

[0047] S20 builds a visual Transformer architecture based on an improved additive attention mechanism

[0048] Specifically, the Transformer module based on the improved additive attention mechanism uses a multiplication operation, takes the feature map x as input, and only uses two matrices W q and W k , W q , Transform the input to generate query matrix Q and key matrix K, Q, Then, the query matrix Q is combined with a learnable parameter vector Multiply them together to learn the attention weight of the query. This process finally generates the global attention query vector Q att , d represents the dimension, and the specific formula is shown below:

[0049]

[0050] The learned attention query vector is further processed by performing a pooling operation on the query matrix to aggregate information and generate a single global query vector γ, as shown below:

[0051]

[0052] Therefore, the output x′ of the improved additive attention mechanism is as follows. The specific principle is as follows Figure 3 As shown:

[0053]

[0054] Where: represents the standardized query matrix, F Linear Represents a linear transformation.

[0055] Step S30 constructs a deep network encoder for a monocular depth estimation method based on a CNN-Transformer hybrid architecture

[0056] Specifically, the encoder performs local-global feature extraction on the input image in four stages, and each stage is composed of the same hybrid CNN-Transformer module. Among them, CT Block is used to learn local-global information, and EC Block is used to optimize the feature mixing method. The encoder based on the CNN-Transformer hybrid module first inputs the feature map into the combined convolution module in the hybrid CNN-Transformer module of the first stage to optimize the feature mixing method, and then inputs it into the hybrid module to extract local-global feature information. Similarly, the second, third and fourth stages all use the same module to extract multi-scale feature information. After completing the first stage process of the deep network encoder, repeat this stage three times, that is, a total of four times of hybrid CNN-Transformer modules at different scales are completed to extract multi-scale feature information, as shown below. Figure 2 shown.

[0057] S31 builds EC Block combined convolution module

[0058] Specifically, the image sequence is input into the self-supervised monocular depth estimation deep network, and first enters the multi-layer multi-scale encoder with the hybrid CNN-Transformer architecture as the backbone. After entering the first stage, the feature map first completes the deep convolution and point-by-point convolution operations in the ECBlock combined convolution module, and is normalized between the two layers of convolution to obtain the intermediate output χ DC The formula is:

[0059] χ DC =F Conv (F BN (F DWConv (x i )))

[0060] Where: i represents the input feature map; F DWConv Indicates the depth convolution operation: F BN represents normalization processing; F Conv represents the point-by-point convolution operation. Subsequently, the feature map is nonlinearly transformed by the GeLU activation function and then sent to the point-by-point convolution again to extract features. Finally, a jump connection is implemented between the obtained feature map and the original input feature map, and the output feature of this part is represented by χ ec The corresponding formula is as follows:

[0061] χ ec =F Conv (F GeLU (x DC ))+χ i

[0062] Where: FGeLU Represents the GeLU activation function.

[0063] S32 builds CT Block hybrid module

[0064] Specifically, the module is divided into a CNN local feature representation part and a Transformer global feature representation part. The CNN local feature representation module uses continuous depth convolution and point-by-point convolution operations to achieve accurate extraction of local features, and obtains the intermediate output χ Local , the formula is as follows:

[0065] χ Local =F Conv (F DWConv (x ec ))

[0066] Where: ec Represents the feature map output from the previous module EC Block; F DWConv represents the depthwise convolution operation, F Conv represents the point-wise convolution operation. Then, χ Local The feature map is sent to the Transformer global feature representation module, which learns the global information of the feature map at each scale by improving the additive attention mechanism, and obtains the intermediate output χ of the global feature representation part. Global , the formula is as follows:

[0067] χ Global =Attention(χ Local )+χ Local In the formula: Attention is the improved additive attention operation in the Transformer module. Finally, the output of the local feature representation part and the output of the global feature representation part are input into the linear calculation layer to generate a local-global feature with nonlinear characteristics. The formula is:

[0068]

[0069] Step S40 constructs a self-supervised monocular depth estimation deep network decoder that introduces a channel attention mechanism

[0070] Specifically, the channel attention mechanism is used in the deep network decoder. The implementation of this mechanism mainly includes two steps, information compression and excitation. First, the feature map X needs to be transformed into the feature map U through the convolution operator. Subsequently, the compression operation compresses the feature map containing global information into a single feature vector through average pooling. The excitation operation uses a fully connected layer and an activation function to output a weight vector. Finally, the final output is obtained by resizing U through the weight vector. Figure 2As shown, the mechanism is reused in the four-layer network of the decoder, and is connected with the encoder through jump connections to form a deep network for self-supervised monocular depth estimation of this method.

[0071] Step S50 constructs a self-supervised monocular depth estimation posture network with a lightweight architecture as the backbone network

[0072] Specifically, the posture network takes an image sequence as input, uses ReaNet18 as a lightweight encoder with a backbone network to extract image features, predicts the camera's motion posture through upsampling and fusion, and reprojects the source view to the target view based on the predicted posture information, outputting a relative posture with six degrees of freedom.

[0073] Step S60 constructs a total loss function with the view reconstruction loss function and the smoothness loss function as the core

[0074] S61 build view reconstruction function

[0075] Specifically, the minimum view reconstruction loss function composed of the minimum photometric error loss function and the pixel mask is used, in which the minimum value of the reprojection error of all source images is selected instead of the average value to calculate the photometric error, solving the problem of blurred edges of the predicted image caused by calculating the average reprojection of multiple frames. The formula is expressed as:

[0076]

[0077] In the formula, F pre is the photometric reprojection function; α = 0.85 is taken empirically; SSIM is the structural similarity calculation between two images; L p is the minimum photometric reprojection error loss. The pixel mask μ is introduced to filter out pixels with the same appearance in the previous and next frames of the sequence, thereby avoiding the adverse effects of static pixels on image prediction and reducing the structural error in image prediction. The specific formula is as follows:

[0078]

[0079] Therefore, the view reconstruction loss function L ss The formula is as follows:

[0080]

[0081] S62 constructs smoothness loss function

[0082] Specifically, in order to optimize the smoothness of the inverse depth map and solve the problem that the edges of objects in the depth estimation task are often not smooth due to large differences in depth information, a gradient-based edge smoothness loss is introduced. The function is expressed as:

[0083]

[0084] In the formula, represents the mean normalized inverse depth.

[0085] S63 constructs the total loss function

[0086] Specifically, the multi-scale view reconstruction loss L ss and smoothness loss L sm The average value of is used as the overall loss function L to guide model training t , the specific formula is as follows:

[0087]

[0088] Where: n is the number of output layers, here it is 4; s is the scale of the depth decoder output, respectively λ is 10 -3 .

[0089] Step S70: Iterate the network training, and save the optimal network model after the loss is stable.

[0090] Specifically, the Mid-Air dataset was selected for training. The hardware platform was a single NVIDIA GeForce RTX3090 24GB GPU, the operating environment was the Ubuntu 20.04 operating system, CUDA version 11.7, and the Adam optimizer was used to optimize the network parameters. The learning rate was set to 1×10 -4 , the batch size is 16, the total training cycle (epoch) is set to 30 rounds, and the performance verification is performed once every iteration to save the model. During the detection process, the test image is input to verify the performance of the method and save the experimental results.

[0091] Step 880 inputs the test data set into the optimal network model to obtain the final depth estimation prediction map

[0092] Specifically, through testing, the present invention has achieved good depth estimation performance in low-altitude UAV flight scenarios and has made significant progress in extracting local-global feature information from images. The visualization results are shown in the figure below. Figure 6 shown.

Claims

1. A monocular depth estimation method based on a hybrid architecture, characterized by: It includes a deep network, a posture network, and a loss function. The deep network includes a Transformer structure based on an improved additive attention mechanism, a combined convolution module, a hybrid module, and a channel attention mechanism. The loss function includes the minimum photometric reprojection error loss function and the smoothness loss function.

2. The monocular depth estimation method based on hybrid architecture as claimed in claim 1, characterized in that: The deep network encoder based on the hybrid architecture extracts local-global features from the input image in four stages. Each stage is composed of the same combined convolution module and module. The combined convolution module is used to optimize the feature mixing method, and the mixing module is used to learn local-global information. The deep network encoder based on the hybrid architecture first inputs the feature map into the combined convolution module of the first stage to optimize the mixing method of the image block, and then inputs it into the mixing module to extract local-global feature information. Similarly, the second, third and fourth stages all use the same module to extract multi-scale feature information.

3. The monocular depth estimation method based on hybrid architecture as claimed in claim 1, characterized in that: The Transformer module based on the improved additive attention mechanism uses dot product operations to simplify the feature information interaction operations between image blocks by learning the query matrix and the key matrix and setting the global attention query vector to learn the hidden representations between image blocks.

4. The monocular depth estimation method based on hybrid architecture as claimed in claim 1, characterized in that: The combined convolution module uses depthwise convolution, pointwise convolution, normalization layer and GeLU activation function to improve the semantic representation ability of image blocks. First, the feature map is input into the depthwise convolution and pointwise convolution. This design enables the model to improve performance without increasing parameters and delays. Normalization is performed between the two layers of convolution. Then the feature map is nonlinearly transformed through the GeLU activation function and then sent to the pointwise convolution again to extract features. The final feature map is output after a jump connection is implemented between the obtained feature map and the original input feature map.

5. The monocular depth estimation method based on hybrid architecture as claimed in claim 1, characterized in that: The hybrid module divides the network structure into a CNN local feature representation module and a Transformer global feature representation module. The CNN local feature representation module uses continuous depth convolution and point-by-point convolution operations to achieve accurate extraction of local features. The Transformer global feature representation module learns the global information of the feature map through the Transformer structure based on the improved additive attention mechanism. Finally, the output of the local feature representation module and the output of the global feature representation module are jointly input into the linear layer to obtain the final output.

6. The monocular depth estimation method based on hybrid architecture as claimed in claim 1, characterized in that: The channel attention mechanism is used in the deep network decoder based on the hybrid architecture. First, the feature map needs to be transformed through the convolution operator, and then the feature map containing global information is compressed into a single feature vector through the average pooling operation. Then, the fully connected layer and activation function are used to output the weight vector. Finally, the feature map is resized by the scale factor and the weight vector to obtain the final output.

7. The monocular depth estimation method based on hybrid architecture as claimed in claim 1, characterized in that: The posture network based on the hybrid architecture takes image sequences as input, adopts ResNet18 as a lightweight backbone network to extract image features, predicts the camera's motion posture through upsampling and fusion, and reprojects the source view to the target view based on the predicted posture information, outputting a relative posture with six degrees of freedom.

8. The monocular depth estimation method based on hybrid architecture as claimed in claim 1, characterized in that: The average of the view reconstruction loss and smoothness loss under comprehensive multi-scale conditions is used as the overall loss function to guide model training, where the view reconstruction loss consists of the minimum photometric error loss function and the pixel mask, and the gradient-based edge smoothness loss is used to optimize the smoothness of the inverse depth map.

Images