Monocular depth estimation method for large-depth-of-field scene under view angle of unmanned aerial vehicle

Abstract

The invention discloses a monocular depth estimation method for a large-depth-of-field scene under a view angle of an unmanned aerial vehicle. The method comprises the steps that a target convolutional neural network is constructed, and the target convolutional neural network comprises a Transform branch, a Convolution branch, an attention mechanism enhancement module, a multi-scale fusion module, a plurality of up-sampling modules and a plurality of feature addition modules; a target image is obtained, and the target image is a large-depth-of-field scene image of the unmanned aerial vehicle at any view angle; the method comprises the steps that a target image is input to a Transform branch and a Convolution branch, four features of different sizes output by four Transform layers of the Transform branch and features output by the Convolution branch are obtained, the Transform branch comprises a plurality of Transform layers, and the Convolution branch is a ResNet50 encoder. The technical problem of inaccurate monocular depth prediction of a large depth-of-field scene under the view angle of an unmanned aerial vehicle in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a monocular depth estimation method for a scene with a large depth of field from the perspective of an unmanned aerial vehicle. Background Art

[0002] In scenes with large depth of field from the perspective of drones, early depth estimation techniques mainly relied on physical methods to obtain depth information. However, these methods were greatly restricted in their scope of application due to the high cost of equipment and the limitation of applicable scenarios. Subsequently, methods based on traditional machine learning began to emerge, which inferred depth information by analyzing the characteristics of the image itself. Although these technologies have achieved results in some tasks, they still face the challenges of high complexity, large computational workload and difficulty in real-time processing when dealing with scenes with large depth of field from the perspective of drones. Among image-based depth estimation methods, they can be divided into two categories according to the source of the image: multi-view and monocular depth estimation. Multi-view depth estimation techniques calculate disparity by comparing images from different perspectives captured by different cameras, and then estimate depth information, such as triangulation, multi-view reconstruction and image sequence depth estimation. These methods usually require paired image sequences as input, and require parameters such as the focal length of the camera to be known, so they are costly and not suitable for resource-constrained scenarios. In contrast, monocular image depth estimation has become an ideal choice for obtaining depth information in scenes with large depth of field from the perspective of drones with its sufficient depth information, flexibility and low cost. At present, the mainstream monocular depth estimation methods include unsupervised learning and supervised learning. However, in large depth of field scenes from the perspective of drones, these methods still need to solve problems such as insufficient feature extraction, loss of spatial structure information, and depth map scale blur and distortion.

[0003] In view of this, it is necessary to propose a monocular depth estimation method for large depth of field scenes from the perspective of a drone, to improve the ability of monocular depth estimation to process global depth and to improve the accuracy of depth estimation. Summary of the invention

[0004] The embodiment of the present invention provides a monocular depth estimation method for a scene with a large depth of field from the perspective of a drone, so as to at least solve the technical problem of inaccurate monocular depth prediction for a scene with a large depth of field from the perspective of a drone in the prior art.

[0005] According to one aspect of an embodiment of the present invention, a monocular depth estimation method for a scene with a large depth of field from the perspective of a drone is provided. The method may include: constructing a target convolutional neural network, wherein the target convolutional neural network includes a Transformer branch, a Convolution branch, an attention mechanism enhancement module, a multi-scale fusion module, several upsampling modules, and several feature addition modules; obtaining a target image, wherein the target image is an image of a scene with a large depth of field from any perspective of the drone; inputting the target image into the Transformer branch and the Convolution branch respectively, and obtaining four features of different sizes output by the four Transformer layers of the Transformer branch and the features output by the Convolution branch, wherein the Transformer branch includes several Transformer layers, and the Convolution branch is a ResNet50 encoder; inputting the four features of different sizes into the attention mechanism enhancement module, and obtaining the output features of the other four layers of the attention mechanism enhancement module except the first layer, and at the same time, the Convolution The features of the branch output are input into the attention mechanism enhancement module to obtain the output features of the first layer of the attention mechanism enhancement module, and the output features of the first layer and the output features of the other three layers except the last layer are used as the four emission features of the multi-scale fusion module, and the output features of the last layer are used as the receiving features of the multi-scale fusion module, wherein the output of the attention mechanism enhancement module includes 5 layers; the receiving features and the four emission features are input into the multi-scale fusion module to obtain the target features, wherein the four emission features are the first emission feature, the second emission feature, the third emission feature and the fourth emission feature; based on the four emission features and the target feature, a depth map corresponding to the target image is obtained.

[0006] Optionally, the method of inputting the target image into the Transformer branch to obtain four features of different sizes output by four Transformer layers of the Transformer branch includes: inputting the target image into a patch partitioning model to obtain a plurality of sub-target images, wherein the initial feature of each sub-target image is set as a concatenation of pixel RGB values; inputting the plurality of sub-target images into a linear embedding layer to obtain total features corresponding to the plurality of sub-target images; and inputting the total features into a plurality of Transformer layers to obtain four features of different sizes output by four Transformer layers, wherein each Transformer layer includes two feature fusion layers, two normalization layers, a multi-head self-attention layer and a multi-layer perceptron.

[0007] Optionally, inputting the target images into the Convolution branches respectively to obtain features output by the Convolution branches includes: extracting features of the target images through the first layer of a ResNet50 encoder to obtain features output by the Convolution branches.

[0008] Optionally, the step of inputting four features of different sizes into the attention mechanism enhancement module to obtain output features of the other four layers of the attention mechanism enhancement module except the first layer includes: inputting four features of different sizes into a 1 x 1 convolution layer to obtain four target features with the same number of channels and different sizes; expanding the four target features with the same number of channels and different sizes to obtain a first initial two-dimensional matrix; inputting the first initial two-dimensional matrix into the deformable self-attention to obtain a first target two-dimensional matrix; folding the first target two-dimensional matrix to obtain enhanced features of four different scales; fusing the enhanced features of each different scale with the features of the corresponding size at the channel level to obtain the fused features of each scale; performing 1x1 convolution on the fused features of each scale to obtain the output features of the other four layers of the attention mechanism enhancement module except the first layer.

[0009] Optionally, the features output by the Convolution branch are input into the attention mechanism enhancement module to obtain the output features of the first layer of the attention mechanism enhancement module, including: performing a 1x1 convolution on the features output by the Convolution branch to obtain the initial features of the features output by the Convolution branch; expanding the initial features of the features output by the Convolution branch to obtain a second initial two-dimensional matrix; inputting the second initial two-dimensional matrix into the deformable self-attention to obtain a second target two-dimensional matrix; folding the second target two-dimensional matrix to obtain enhanced features corresponding to the second target two-dimensional matrix; fusing the enhanced features corresponding to the second target two-dimensional matrix and the features output by the Convolution branch at the channel level to obtain the target features of the features output by the Convolution branch; performing a 1x1 convolution on the target features of the features output by the Convolution branch to obtain the output features of the first layer of the attention mechanism enhancement module.

[0010] Optionally, the method of obtaining a depth map corresponding to the target image based on four emission features and the target feature includes: upsampling the target feature to obtain a first upsampling feature; adding the first upsampling feature and the fourth emission feature to obtain a first added feature; upsampling the first added feature to obtain a second upsampling feature; adding the second upsampling feature and the third emission feature to obtain a second added feature; upsampling the second added feature to obtain a third upsampling feature; adding the third upsampling feature and the second emission feature to obtain a third added feature; upsampling the third added feature to obtain a fourth upsampling feature; and adding the fourth upsampling feature and the first emission feature to obtain a depth map corresponding to the target image.

[0011] Beneficial effects of the present invention: (1) Using Transformer and Resnet dual-branch encoders for feature extraction: Transformer branch: has a self-attention mechanism that can well capture long-distance dependencies and effectively extract the correlation information between different areas in large scenes; Resnet: as a classic convolutional neural network, it can well extract local features; through the dual-branch encoder, it can fully combine global and local features to provide richer and more accurate feature information for subsequent depth estimation.

[0012] (2) Feature enhancement is performed through the feature enhancement module. After feature extraction, the feature enhancement module is used to further optimize and enhance the features, which helps to highlight the features that are valuable for depth estimation and suppress noise and irrelevant information, thereby improving the accuracy of depth estimation.

[0013] (3) The enhanced features are sent to the multi-scale fusion module of the decoder for feature fusion: At different scales, the features of the scene are expressed differently; the multi-scale fusion module can effectively fuse features at different scales and integrate multi-scale information, so that the model can fully understand the scene structure and further improve the accuracy of depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 is a flow chart of a monocular depth estimation method for a scene with a large depth of field from the perspective of a drone according to an embodiment of the present invention; Figure 2 is a framework diagram of a monocular depth estimation method for a scene with a large depth of field from the perspective of a drone according to an embodiment of the present invention; Figure 3is a framework diagram of a multi-scale fusion module according to an embodiment of the present invention; FIG4 is a framework diagram of an attention mechanism enhancement module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] Example 1 According to an embodiment of the present invention, a monocular depth estimation method for a scene with a large depth of field from the perspective of a drone is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0018] Figure 1 is a flow chart of a monocular depth estimation method for a scene with a large depth of field from the perspective of a drone according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps: Step S101, constructing a target convolutional neural network, wherein the target convolutional neural network includes a Transformer branch, a Convolution branch, an attention mechanism enhancement module, a multi-scale fusion module, several upsampling modules and several feature addition modules.

[0019] In the technical solution provided in the above step S101 of the present invention, Figure 2It is a framework diagram of a monocular depth estimation method for a scene with a large depth of field from the perspective of a drone according to an embodiment of the present invention. The target convolutional neural network includes a Transformer branch, a Convolution branch, an attention mechanism enhancement module, a multi-scale fusion module, several upsampling modules and several feature addition modules.

[0020] Step S102, obtaining a target image, wherein the target image is a scene image with a large depth of field at any viewing angle of the drone.

[0021] In the technical solution provided in the above step S102 of the present invention, the target image is a scene image with a large depth of field taken by the drone at any viewing angle, such as Figure 2 The RGB image in , the RGB image is a photo taken by the drone at any one of the multiple perspectives.

[0022] Step S103, input the target image into the Transformer branch and the Convolution branch respectively, and obtain features of four different sizes output by the four Transformer layers of the Transformer branch and features output by the Convolution branch, wherein the Transformer branch includes several Transformer layers, and the Convolution branch is a ResNet50 encoder.

[0023] In the technical solution provided in step S103 of the present invention, if Figure 2As shown in the figure, the target image is input into the Transformer branch and the Convolution branch respectively. Each Transformer layer in the Transformer branch processes the target image, but only obtains the four different-sized features output by the four specific Transformer layers of the Transformer branch. The four different-sized features include: first-size features, second-size features, third-size features, and fourth-size features. The Convolution branch processes the target image to obtain the features output by the Convolution branch. The Transformer layer is Swim-Transformer, because considering the particularity of the large depth of field scene of the drone's perspective, although ViT is good at modeling long-distance correlation It has advantages in terms of scalability, but is relatively weak in processing local information and multi-scale features, and its global attention mechanism has high computational complexity; Swim-Transformer can provide hierarchical features and reduce computational complexity, which is very beneficial for tasks that require multi-scale features (such as monocular depth estimation). The size of the features output by the Convolution branch is (2, 64, 176, 352). The first column is the batch-size, the second column is the number of channels, the third column is the height of the feature, and the fourth column is the width of the feature. The sizes of the four different sizes of features output by the four Transformer layers are: (2, 192, 88, 176), (2, 384, 44, 88), (2, 768, 22, 44) and (2, 1536, 11, 22). The first column is the batch-size, the second column is the number of channels, the third column is the height of the feature, and the fourth column is the width of the feature.

[0024] Step S104, input the features of four different sizes into the attention mechanism enhancement module, and obtain the output features of the other four layers of the attention mechanism enhancement module except the first layer. At the same time, input the features output by the Convolution branch into the attention mechanism enhancement module, and obtain the output features of the first layer of the attention mechanism enhancement module. The output features of the first layer and the output features of the remaining three layers except the last layer are used as the four emission features of the multi-scale fusion module, and the output features of the last layer are used as the receiving features of the multi-scale fusion module. The output of the attention mechanism enhancement module includes 5 layers.

[0025] In the technical solution provided in step S104 of the present invention, Figure 2 As shown in Figure 2, the four features F of different sizes and the feature G output by the Convolution branch are input into the attention mechanism enhancement module to obtain the output of the attention mechanism enhancement module. ,in, for , for ,Will The output features of the last layer in As the receiving feature of the multi-scale fusion module, it is recorded as , the four output features of the other four layers As the four emission features of the multi-scale fusion module, they are denoted as .

[0026] Step S105, inputting the receiving feature and the four transmitting features into a multi-scale fusion module to obtain the target feature, wherein the four transmitting features are respectively a first transmitting feature, a second transmitting feature, a third transmitting feature and a fourth transmitting feature.

[0027] In the technical solution provided in step S105 of the present invention, if Figure 2 As shown, and Input into the multi-scale fusion module to obtain the target features.

[0028] Output the attention mechanism enhancement module ,in, Recorded as = ,in represents the Nth reassembled feature map in the feature enhancement module; It represents the output feature after the input feature G passes through the feature enhancement module. It is input to the AMD module of the decoder for multi-scale information fusion; the decoder includes an AMD module and an upsampling module.

[0029] Figure 3 is a framework diagram of a multi-scale fusion module according to an embodiment of the present invention, such as Figure 3 The multi-scale fusion module (AMD) shown in the figure can better model multi-scale dependencies by fusing multi-scale information. AMD consists of two parts: attention part and information part. The attention part calculates spatial attention and channel attention and uses conditional kernel for weighting. The information part multiplies the attention map with the received features to obtain the final output features. Specifically, AMD includes the following five processes: Step 1: Input feature selection Multi-scale features obtained by processing images through the Attention Mechanism Enhanced Module (AMEM) , , among these features, specify the output features of the last layer As receiving feature , while other features and The emission characteristics are recorded as , that is, a specific layer of the Transformer branch or CNN branch output by the feature enhancement module is selected as the receiving feature map, and the corresponding receiving and transmitting features are determined.

[0030] The output features of the last layer of the feature enhancement module are selected As receiving feature , other layers As emission characteristics ( are the features output by the 1st, 2nd, 3rd, and 4th layers respectively).

[0031] (1) Input feature preparation The five-layer features that will be output by the AMDM feature enhancement module Input to the multi-scale fusion module and select the output features of the last layer of the feature enhancement module As receiving feature , other layers As the emission characteristics ( are the features output by the 1st, 2nd, 3rd, and 4th layers respectively).

[0032] (2) Spatial Attention Calculation First, define a convolution kernel (Its size and number of channels match the input feature map), assuming that the input feature map has C channels, H height, and W width, for each emission feature ( Represents the features in each emission feature) and its corresponding convolution kernel Perform convolution operation to obtain a new feature map corresponding to each emission feature , They are The convolution operation here is performed on the spatial dimension of the feature map, that is, the convolution kernel is applied to the feature map of each channel to obtain a new feature map , whose size is the same as the input feature map (c, h, w), and then calculate each spatial attention map corresponding to each emission feature , They are Through the corresponding feature map after convolution Sum the values ​​in the channel dimension and divide by the number of channels C to get each spatial attention weight matrix, which is calculated as:

[0033]

[0034] Among them, [c,h,w] represents the index in the channel, height and width dimensions. This operation summarizes all channel information at each spatial position and obtains a weight matrix reflecting the spatial importance, whose size is HxW.

[0035] (3) Channel attention calculation (a) Emission characteristics ( ) Perform an average pooling operation in the spatial dimension to obtain a feature vector of size Cx1x1, which represents the average feature value of each channel in the entire space.

[0036] (b) Also use the convolution kernel Perform convolution operation on each average pooled feature vector to obtain each new feature vector , They are (The size is still Cx1x1), the calculation formula is:

[0037] Among them, AvgPool represents the average pooling operation.

[0038] (c) Calculate the channel attention map corresponding to each emission feature , They are By summing each feature vector after convolution in the spatial dimension (the spatial dimension size is 1 at this time) and dividing it by the spatial dimension size (i.e. 1), we get the channel attention map corresponding to each emission feature. , and its calculation formula is:

[0039] Among them, HW represents the total size of the spatial dimension (i.e. HxW). This operation obtains a weight vector of size C that reflects the importance of each channel.

[0040] (4) Attention Weighting (a) By obtaining each spatial attention And each channel attention Then, perform a softmax operation on each spatial attention to obtain each normalized spatial attention weight matrix corresponding to each spatial attention. , They are , and its calculation formula is:

[0041] This operation converts each element in the spatial attention weight matrix into a probability value between 0 and 1, indicating the relative importance of each spatial position, and the sum of the probabilities of all positions is 1.

[0042] (b) Perform a sigmoid operation on each channel attention to obtain each normalized channel attention weight vector corresponding to each channel attention , They are , and its calculation formula is:

[0043] This operation converts each element in the channel attention weight vector into a probability value between 0 and 1, indicating the relative importance of each channel.

[0044] (c) Finally, each spatial attention weight matrix , each channel attention weight vector With each original emission feature Multiply them together to get the weighted features of each attention, and the calculation formula is:

[0045] The multiplication operation here is performed in the spatial and channel dimensions, that is, for each spatial position and each channel, the corresponding spatial attention weight, channel attention weight and original eigenvalue are multiplied to obtain the attention-weighted eigenvalue. They are .

[0046] Through the above complete attention calculation process, AMD can effectively capture the spatial and channel importance information in the feature map, so as to better integrate multi-scale features and improve the performance of the model in continuous pixel-level prediction tasks.

[0047] (2) Predicting the attention kernel Predict kernels from input features using linear transformation ; These prediction kernels are like some "connecting bridges" that can connect the receiving features and the transmitting features in different ways. Specifically, by (Right now ) and receiving characteristics (Right now ) and then linearly transformed to obtain , They are ; By linearly transforming each transmission feature and receiving feature, we can obtain and ,in, They are , They are Then, by convolving these kernels with the feature map, we get the integrated attention of each target ,Here in order to reduce the computational complexity of convolution, grouped convolution ,is used.

[0048] Calculation: First, each emission feature (Right now ) and receiving characteristics (Right now ) are concatenated (concat operation) on the channel dimension to obtain a new feature vector. Then, this concatenated feature vector is combined with the weight matrix Convolution, and add a bias vector ,get Kernel. The role of this kernel is to capture the complex relationship between the emission features and the reception features so that the features can be fused and adjusted in subsequent calculations. In order to reduce the computational complexity, group convolution is used to first combine the concatenated features. The channels are grouped into groups, and the number of channels in each group is (C e +C r ) / Y; For each emission feature after the channel number grouping (i.e. ), The received features after grouping by the number of channels, for each group, use the convolution kernel Perform convolution operation to get the output , and finally output all groups Put it together and get , the specific calculation process is as follows:

[0049]

[0050] Calculation: Directly transform the emission characteristics (Right now ) and the weight matrix Convolution, and add a bias vector ,get Kernel. This kernel is mainly used to adjust the information transfer from the receiving features to the transmitting features according to the transmitting features. Grouped convolution is also used here: First, (Right now ) are grouped according to the set grouping number Y. Assume The number of channels is C e , then the number of channels in each group is Ce / Y, for each group, use an independent convolution kernel (y represents the group index) convolution and add the bias vector Get the output O of this group y (y=1,2,…,Y), the shape of each group is (N, , H, W), N is the batch size, It is The number of channels in the group, H and W are the height and width of the feature map, and finally the output of all groups is Splicing along the channel dimension The specific calculation process is as follows:

[0051]

[0052] Calculation: and prediction kernel, will receive features (Right now ) and the weight matrix Multiply and add the bias vector ,get Kernel. This kernel is used to adjust the information transfer from the transmission features to the reception features according to the reception features. Grouped convolution is also used here: Will The channels are grouped into groups, and the number of channels in each group is C r / Y(Assume The number of channels is C r ), for each grouping feature ( ), using the corresponding convolution kernel Perform convolution operation and add bias vector Get group output (y=1,2,…,Y), output of all groups Put it together and get , the specific operation is as follows Figure 3 The grouped convolution on the lower left; the specific calculation process is as follows:

[0053]

[0054] Finally, by convolving these kernels with the feature map, we get the integrated attention of each target , They are , each target attention can represent the complex relationship between features of different scales, telling the model which features are more closely related at different scales; the specific process is as follows:

[0055] Item 1 Indicates based on Nuclear emission characteristics Extract and receive features Related information.

[0056] Item 2 Indicates based on Core Slave Receiver Features Extraction and emission features Related information.

[0057] Item 3 Indicates the emission characteristics through L core and receiving characteristics (Right now ) for joint processing to further integrate the information between the two.

[0058] The generation of these attention kernels is dynamic and is calculated in real time based on the input features. They can adaptively adjust the information flow and fusion between features of different scales according to the relationship between features and task requirements. The attention kernel can help the model focus on areas in the image that are related to depth changes and better capture the spatial layout and depth hierarchy of objects.

[0059] Step 3: Messaging The information part of the multi-scale fusion module is used To calculate, is the kernel predicted by the above linear transformation; specifically, the information module multiplies each transmitted feature map with the corresponding attention weight and fuses the result with the received feature map.

[0060] Step 4: Multi-scale fusion module output Final Output It is obtained by concatenating the results of attention processing of emission features of different scales with the received features, as shown in the following formula:

[0061] Step S106, obtaining a depth map corresponding to the target image based on the four emission features and the target features.

[0062] In the technical solution provided in step S106 of the present invention, if Figure 2 As shown, based on the four emission features and the target features, a depth map corresponding to the target image is obtained.

[0063] The above method of this embodiment is further introduced below.

[0064] As an optional implementation method, step S103, the target image is input into the Transformer branch to obtain four features of different sizes output by four Transformer layers of the Transformer branch, including: inputting the target image into a patch partitioning model to obtain a number of sub-target images, wherein the initial feature of each sub-target image is set to a series of pixel RGB values; inputting the several sub-target images into a linear embedding layer to obtain total features corresponding to the several sub-target images; inputting the total features into several Transformer layers to obtain four features of different sizes output by four Transformer layers, wherein each Transformer layer includes two feature fusion layers, two normalization layers, a multi-head self-attention layer and a multi-layer perceptron.

[0065] In this embodiment, if Figure 2 As shown in the figure, the target image is input into the patch partitioning model (Ppm) to obtain non-overlapping patches, where the initial feature of each patch is set as the concatenation of the pixel RGB values. The linear embedding layer processes the features of each patch to obtain the total features corresponding to all patches. The total features are processed in turn by several Transformer layers, and finally the features output by specific 4 Transformer layers are extracted, as shown in Figure 2 As shown in the figure, the order of modules in a Transformer layer is normalization layer, multi-head self-attention, addition layer, normalization layer, multi-layer perceptron and addition layer.

[0066] The total features are input into several Transformer layers, and the features of four different sizes output by four Transformer layers are obtained as follows: Apply 4 Transformer layers to extract features, sample and reassemble N feature maps from N selected Transformer layers as the output of the Transformer branch, denoted as ,in, represents the Nth reassembled feature map.

[0067] Each Transformer layer consists of an MSA and an MLP, and LN is used before MSA and MLP. Considering the particularity of large depth of field scenes under the perspective of drones, the MLP layer is replaced with KAN to enable the model to learn more complex features and improve the nonlinear expression ability and generalization of the model. The layer process is expressed as:

[0068]

[0069] in, , Respectively represent The output features of the MSA module and the KAN module, Represents the input of the Transformer layer (N=4 is selected, that is, the Transformer branch outputs features of 4 layers of different scales).

[0070] As an optional implementation method, step S102, inputting the target image into the Convolution branch respectively to obtain the features output by the Convolution branch, includes: extracting features of the target image through the first layer of the ResNet50 encoder to obtain the features output by the Convolution branch.

[0071] In this embodiment, the target image is input into the Convolution branch respectively, the Convolution branch is a ResNet50 encoder, the output features of the first layer of the ResNet50 encoder are determined as the features output by the Convolution branch, and the CNN branch uses the first block of the standard ResNet50 encoder (the first part of the structure in the network, which is responsible for preliminary feature extraction and processing of the input image and provides basic features for subsequent operations) to mine local information such as edges and textures to avoid low-level features being washed out by continuous multiplication and reduce calculation time; the feature mapping of the output channel (first layer) .

[0072] As an optional implementation method, step S104, the inputting of four features of different sizes into the attention mechanism enhancement module to obtain the output features of the other four layers of the attention mechanism enhancement module except the first layer, includes: inputting the four features of different sizes into the 1x1 convolution layer to obtain four target features with the same number of channels and different sizes; expanding the four target features with the same number of channels and different sizes to obtain a first initial two-dimensional matrix; inputting the first initial two-dimensional matrix into the deformable self-attention to obtain a first target two-dimensional matrix; folding the first target two-dimensional matrix to obtain enhanced features of four different scales; fusing the enhanced features of each different scale with the features of the corresponding size at the channel level to obtain the fused features of each scale; performing 1x1 convolution on the fused features of each scale to obtain the output features of the other four layers of the attention mechanism enhancement module except the first layer.

[0073] In this embodiment, FIG4 is a framework diagram of an attention mechanism enhancement module according to an embodiment of the present invention, FIG4 (a) is a schematic diagram of a feature enhancement module, and FIG4 (b) is a folding and unfolding process of FIG4 (a) and an enlarged diagram of a feature map. The multi-scale feature F output by the Transformer branch and the single-scale feature G output by the CNN branch are passed through an attention mechanism enhancement module (AMEM), such as the attention mechanism enhancement module shown in FIG4 (a), and the specific process is as follows: (1) Input feature processing First, the features output by the Transformer branch Project their features to the same number of channels C through 1x1 convolution h , denoted as F h = , where N is 4. Consider a set of hierarchical features extracted by Transformer. These feature maps have different sizes and numbers of channels. Use convolution to project all hierarchical features to the same channel. This step makes the features of different levels consistent in the channel dimension, which is convenient for subsequent calculations.

[0074] (2) Then F h The features of are expanded (i.e. flattened and concatenated) into a two-dimensional matrix X, where each row is a C from the hierarchical features h dimensional feature vector, representing the features of a pixel; then calculate Q (query feature), K (key feature), and V (value feature) through the linear projection of X: Q=XP Q , K = XP K , V=XP V ; where P Q , P K , P V represents the linear projection matrix; (3) For the unfolded two-dimensional matrix X, use the variability self-attention module to enhance the features. For the query features Sum value characteristics , the deformable self-attention is calculated as follows: Attention Weight and sampling offset By querying the features Linear projection is performed to obtain, and the attention weights are normalized so that ; Calculate the attention result:

[0075] in, represents the position of the query vector, since (a fraction), calculated using bilinear interpolation ,At the same time, a hierarchical embedding is added to identify the feature level where each query pixel is located.

[0076] (4) The two-dimensional matrix output by the deformable self-attention module Fold (i.e. split and reshape) back to the original resolution (feature map) to obtain the hierarchical enhanced features F enh Original resolution, then F enh After concatenation with the original feature F through channels, it is fused through 1x1 convolution to obtain the output , which realizes the enhancement of Transformer features.

[0077] As an optional implementation method, step S104, the feature output by the Convolution branch is input into the attention mechanism enhancement module to obtain the output feature of the first layer of the attention mechanism enhancement module, including: performing a 1x1 convolution on the feature output by the Convolution branch to obtain the initial feature of the feature output by the Convolution branch; expanding the initial feature of the feature output by the Convolution branch to obtain a second initial two-dimensional matrix; inputting the second initial two-dimensional matrix into the deformable self-attention to obtain a second target two-dimensional matrix; folding the second target two-dimensional matrix to obtain enhanced features corresponding to the second target two-dimensional matrix; fusing the enhanced features corresponding to the second target two-dimensional matrix and the features output by the Convolution branch at the channel level to obtain the target features of the features output by the Convolution branch; performing a 1x1 convolution on the target features of the features output by the Convolution branch to obtain the output features of the first layer of the attention mechanism enhancement module.

[0078] In this embodiment, as shown in FIG4 , it is divided into the following three parts: (1) For the convolutional branch CNN, the feature map G is used as the second input of AMEM to perform CNN feature enhancement. First, Project G to the channel dimension through 1x1 convolution of , then Flattening obtains the two-dimensional query matrix Q.

[0079] (2) Using the previously calculated As key features (K) and value features (V), calculate cross attention. Also due to memory cost issues, use the variability attention module.

[0080] (a) First, a learnable linear transformation is performed on the affinity query embedding (which can be understood as a representation related to the convolutional feature and used to guide the calculation of attention) to obtain an intermediate result. Then, this intermediate result is mapped through the sigmoid function to limit its value between 0 and 1, thereby obtaining the reference point position This process is dynamic, which means that the model can flexibly determine the reference point location each time the cross attention is calculated based on the input features and learned parameters, so that the attention mechanism can better adapt to different feature relationships and task requirements.

[0081] (b) Once the reference point is determined Position, deformable attention calculation can be performed. Similar to self-attention calculation, it involves calculating attention weights and sampling offset , but these calculations are based on dynamically predicted reference point positions carried out.

[0082] (c) Attention weight and sampling offset It is still obtained by linearly projecting the query feature (here, the query matrix Q after the convolution feature G is processed), but the reference point position is taken into account in the calculation process For example, sampling offset Based on the reference point position Adjustments are made so that attention can focus more on areas related to the reference point, thereby better modeling the interaction between Transformer and CNN features.

[0083] (d) The final cross-attention result is obtained in a similar way to the self-attention calculation, that is, the value feature V (here is ) is weighted summed, where the weight is determined by the attention weight and according to the reference point position and sampling offset The sampling locations are determined together.

[0084] (3) Reshape the cross-attention calculation results to the original resolution to form an attention representation , then G is fused with channel concatenation and 1×1 convolution to obtain another output of the feature enhancement module, thereby realizing CNN branch feature enhancement and interaction.

[0085] As an optional implementation method, step S106, the depth map corresponding to the target image is obtained based on the four emission features and the target feature, including: upsampling the target feature to obtain a first upsampling feature; adding the first upsampling feature and the fourth emission feature to obtain a first added feature; upsampling the first added feature to obtain a second upsampling feature; adding the second upsampling feature and the third emission feature to obtain a second added feature; upsampling the second added feature to obtain a third upsampling feature; adding the third upsampling feature and the second emission feature to obtain a third added feature; upsampling the third added feature to obtain a fourth upsampling feature; adding the fourth upsampling feature and the first emission feature to obtain a depth map corresponding to the target image.

[0086] In this embodiment, if Figure 2 As shown, the four emission features and the target features are upsampled and added repeatedly to obtain a depth map corresponding to the target image.

[0087] In an embodiment of the present invention, a target convolutional neural network is constructed, wherein the target convolutional neural network includes a Transformer branch, a Convolution branch, an attention mechanism enhancement module, a multi-scale fusion module, a plurality of upsampling modules and a plurality of feature addition modules; a target image is obtained, wherein the target image is a large depth of field scene image under any viewing angle of the drone; the target image is respectively input into the Transformer branch and the Convolution branch to obtain four features of different sizes output by four Transformer layers of the Transformer branch and features output by the Convolution branch, wherein the Transformer branch includes a plurality of Transformer layers, and the Convolution branch is a ResNet50 encoder; the four features of different sizes are input into the attention mechanism enhancement module to obtain the attention mechanism enhancement module except The output features of the other four layers passing through the first layer are inputted into the attention mechanism enhancement module at the same time, and the output features of the Convolution branch are inputted into the attention mechanism enhancement module to obtain the output features of the first layer of the attention mechanism enhancement module, and the output features of the first layer and the output features of the other three layers except the last layer are used as the four emission features of the multi-scale fusion module, and the output features of the last layer are used as the receiving features of the multi-scale fusion module, wherein the output of the attention mechanism enhancement module includes 5 layers; the receiving features and the four emission features are inputted into the multi-scale fusion module to obtain the target features, wherein the four emission features are the first emission feature, the second emission feature, the third emission feature and the fourth emission feature respectively; based on the four emission features and the target features, a depth map corresponding to the target image is obtained, which solves the technical problem of inaccurate monocular depth prediction of large depth of field scenes from the perspective of drones in the prior art, and achieves the technical effect of accurate monocular depth prediction of large depth of field scenes from the perspective of drones through a new network framework.

[0088] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0089] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0091] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0092] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0093] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A monocular depth estimation method for a scene with a large depth of field from the perspective of a drone, characterized in that: include: Construct a target convolutional neural network, where the target convolutional neural network includes a Transformer branch, a Convolution branch, an attention mechanism enhancement module, a multi-scale fusion module, several upsampling modules and several feature addition modules; Acquire a target image, wherein the target image is a scene image with a large depth of field at any viewing angle of the drone; The target image is input into the Transformer branch and the Convolution branch respectively, and the features of four different sizes output by the four Transformer layers of the Transformer branch and the features output by the Convolution branch are obtained. The Transformer branch contains several Transformer layers, and the Convolution branch is a ResNet50 encoder. Input the features of four different sizes into the attention mechanism enhancement module to obtain the output features of the other four layers of the attention mechanism enhancement module except the first layer. At the same time, input the features output by the Convolution branch into the attention mechanism enhancement module to obtain the output features of the first layer of the attention mechanism enhancement module. The output features of the first layer and the output features of the other three layers except the last layer are used as the four emission features of the multi-scale fusion module, and the output features of the last layer are used as the receiving features of the multi-scale fusion module. The output of the attention mechanism enhancement module includes 5 layers. Inputting the receiving feature and the four transmitting features into a multi-scale fusion module to obtain the target feature, wherein the four transmitting features are respectively a first transmitting feature, a second transmitting feature, a third transmitting feature and a fourth transmitting feature; Based on the four emission features and the target features, a depth map corresponding to the target image is obtained.

2. The method according to claim 1, characterized in that The target image is input into the Transformer branch to obtain four features of different sizes output by the four Transformer layers of the Transformer branch, including: Input the target image into the patch partitioning model to obtain several sub-target images, where the initial feature of each sub-target image is set as the concatenation of pixel RGB values; Input several sub-target images into the linear embedding layer to obtain the total features corresponding to the several sub-target images; The total features are input into several Transformer layers to obtain features of four different sizes output by four Transformer layers, where each Transformer layer includes two feature fusion layers, two normalization layers, a multi-head self-attention layer and a multi-layer perceptron.

3. The method according to claim 1, characterized in that The target images are input into the Convolution branches respectively to obtain the features output by the Convolution branches, including: The first layer of the ResNet50 encoder is used to extract features of the target image and obtain the features output by the Convolution branch.

4. The method according to claim 1, characterized in that: The four features of different sizes are input into the attention mechanism enhancement module to obtain the output features of the other four layers of the attention mechanism enhancement module except the first layer, including: Input four features of different sizes into the 1 x 1 convolutional layer to obtain four target features with the same number of channels and different sizes; Expand four target features with the same number of channels and different sizes to obtain a first initial two-dimensional matrix; Input the first initial two-dimensional matrix into the deformable self-attention to obtain a first target two-dimensional matrix; The first target two-dimensional matrix is ​​folded to obtain four enhanced features of different scales; The enhanced features of each different scale are fused with the features of the corresponding size at the channel level to obtain the fused features of each scale; After performing 1x1 convolution on the fused features of each scale, the output features of the other four layers except the first layer of the attention mechanism enhancement module are obtained.

5. The method according to claim 1, characterized in that The features output by the Convolution branch are input into the attention mechanism enhancement module to obtain the output features of the first layer of the attention mechanism enhancement module, including: After performing 1x1 convolution on the features output by the Convolution branch, the initial features of the features output by the Convolution branch are obtained; Expand the initial features of the features output by the Convolution branch to obtain a second initial two-dimensional matrix; Input the second initial two-dimensional matrix into the deformable self-attention to obtain a second target two-dimensional matrix; Folding the second target two-dimensional matrix to obtain enhanced features corresponding to the second target two-dimensional matrix; The enhanced features corresponding to the second target two-dimensional matrix and the features output by the Convolution branch are fused at the channel level to obtain the target features of the features output by the Convolution branch; After performing 1x1 convolution on the target features of the features output by the Convolution branch, the output features of the first layer of the attention mechanism enhancement module are obtained.

6. The method according to claim 1, characterized in that The method of obtaining a depth map corresponding to the target image based on the four emission features and the target features includes: Upsampling the target feature to obtain a first upsampling feature; Adding the first up-sampled feature and the fourth emission feature to obtain a first added feature; Upsampling the first added feature to obtain a second upsampled feature; Adding the second up-sampled feature and the third emission feature to obtain a second added feature; Upsampling the second added feature to obtain a third upsampled feature; Adding the third up-sampled feature and the second emission feature to obtain a third added feature; Upsampling the third added feature to obtain a fourth upsampled feature; The fourth up-sampling feature and the first emission feature are added to obtain a depth map corresponding to the target image.

7. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

8. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

9. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images