Unmanned aerial vehicle ground target positioning method based on monocular depth estimation

Abstract

The present application relates to the technical field of computer vision, and particularly relates to a UAV ground target positioning method based on monocular depth estimation. Based on a preset sampling interval, a sampling image frame in a remote sensing video collected by a monocular camera of a UAV is acquired; the sampling image frame is input into a preset network architecture for feature extraction, and a depth feature map is obtained, wherein the preset network architecture comprises a plurality of visual transformers and a single convolutional neural network, the preset network architecture comprises a pose network for outputting a translation matrix and a rotation matrix, a scene reconstruction guiding network for scene reconstruction of the sampling image frame, a heterogeneous feature interaction module for mixing feature mapping of the visual transformers and the convolutional neural network, and a local texture smoothing module for reducing texture performance level of a target region in the sampling image frame; and a target object positioning result in the remote sensing video is determined based on the depth feature map, aiming at solving the problem of inaccurate positioning of a ground target by a UAV.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for UAV ground target localization based on monocular depth estimation. Background Technology

[0002] Depth estimation is a computer vision application task designed to infer depth information of a scene from visible light images. Accurately estimating the distance and height of targets in remote sensing images provides crucial information for subsequent applications such as terrain modeling, 3D reconstruction, target detection, and recognition. Currently, the application of depth estimation in remote sensing images faces several challenges. Traditional depth estimation methods, such as SLAM (Simultaneous Localization and Mapping) algorithms based on camera images, typically require physical equipment to obtain depth information within the field of view.

[0003] However, when traditional depth estimation methods are applied to UAV scenarios, the accuracy of depth estimation is easily reduced due to factors such as changes in lighting and occlusion in the remote sensing images collected by the UAV, which are limited by the hardware performance of the UAV and the complex terrain such as mountains, rivers and forests encountered by the UAV during flight.

[0004] In relevant technical solutions, convolutional neural networks (CNNs) with a self-supervised learning mechanism are typically used to estimate the depth of remote sensing images acquired by UAVs. However, the synthetic images generated by the convolutional layers of traditional neural networks rely on four neighboring pixels for each pixel, resulting in a limited receptive field size that is difficult to meet the larger receptive field requirements of UAVs during flight. Furthermore, the depth changes of the surrounding scene during UAV flight are uneven, leading to uneven depth changes in the acquired remote sensing images. These factors can easily cause the loss of the depth estimation advantages of locally perceptual convolutional neural networks applied to UAVs, resulting in inaccurate estimation of object depths in remote sensing images and consequently, inaccurate localization of ground targets by the UAV.

[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this invention is to provide a ground target localization method for UAVs based on monocular depth estimation, which aims to solve the problem of inaccurate ground target localization by UAVs.

[0007] To achieve the above objectives, the present invention provides a UAV ground target localization method based on monocular depth estimation, the method comprising:

[0008] Based on a preset sampling interval, sampled image frames are acquired from remote sensing videos captured by the monocular camera of the UAV.

[0009] The sampled image frame is input into a preset network architecture for feature extraction to obtain a depth feature map. The preset network architecture includes multiple visual transformers and a single convolutional neural network. The preset network architecture includes a pose network for outputting translation and rotation matrices, a scene reconstruction guidance network for reconstructing scenes from the sampled image frame, a heterogeneous feature interaction module for mixing the feature maps of the visual transformers and the convolutional neural network, and a local texture smoothing module for reducing the texture representation level of the target region in the sampled image frame.

[0010] The location of the target object in the remote sensing video is determined based on the depth feature map.

[0011] Optionally, the step of inputting the sampled image frame into a preset network architecture for feature extraction to obtain a depth feature map includes:

[0012] Based on a preset convolutional layer, the sampled image frame is cropped to a preset size;

[0013] The sampled image cropped to the preset scale is input into a multi-scale path to extract local features of multiple different pixels, wherein patch labels of different scales are embedded in parallel in the multi-scale path.

[0014] Each of the local features is input in parallel to the multipath transformer layer to obtain the processing results based on the outputs of multiple visual transformers and a single convolutional neural network in the multipath transformer layer. The processing results are then connected by skip connections to obtain a feature map. In the visual transformer, the input local features are globally modeled using a multi-head self-attention strategy to adjust the focus of different regions in the local features.

[0015] The feature map is input into the heterogeneous feature interaction module to enhance the global structural information of the feature map and obtain an interactive feature map.

[0016] The interactive feature map is input into the decoder layer, and the features output by the decoder layer are obtained as the deep feature map. The decoder layer includes a convolutional layer and an activation function layer.

[0017] Optionally, the step of determining the target object localization result in the remote sensing video based on the depth feature map includes:

[0018] Calculate the normal map corresponding to the depth feature map;

[0019] Determine the ground reference point based on the orientation of the pixels in the normal map;

[0020] Calculate the relative flight altitude of the UAV based on the ground reference point;

[0021] Obtain the current flight altitude of the drone, and determine the scaling factor based on the ratio of the current flight altitude to the relative flight altitude;

[0022] The absolute depth value of a pixel is determined by multiplying the scaling factor by the relative depth value of each pixel in the depth feature map.

[0023] The current latitude and longitude of the UAV are obtained, and the positioning result of the target object is determined based on the absolute depth value and the current latitude and longitude.

[0024] Optionally, before the step of determining the target object localization result in the remote sensing video based on the depth feature map, the method further includes:

[0025] Obtain a preset combination of loss functions;

[0026] The training loss result of the preset network architecture is constrained by the preset loss function combination, wherein the preset loss function combination includes image reconstruction loss function, image gradient discrimination loss function, matching guidance loss function, minimum luminosity loss function and total loss function;

[0027] The step of constraining the training loss result of the preset network architecture based on the preset loss function combination includes:

[0028] Based on the image reconstruction loss function, the variation error loss of the depth feature map relative to the sampled image frame is constrained; and,

[0029] Based on the image gradient discrimination loss function, the gradient discrimination loss is constrained after performing second-order gradient smoothing on the image region transition of the depth feature map; and...

[0030] Based on the matching-guided loss function, the error loss result is calculated by constraining the maximum scale in the depth feature map; and...

[0031] Based on the minimum photometric loss function, constrain the photometric calculation error loss in the depth feature map; and...

[0032] Based on the total loss function, the change error loss, the gradient discrimination loss, the maximum scale calculation error loss, and the photometric calculation error loss are averaged to determine the overall loss result of the preset network architecture.

[0033] Optionally, the pose network adopts a ResNet-18 network architecture, and the pose network calls preset network weights to perform self-motion transformation on the sampled image frames, so as to transform multiple sampled image frames into the translation matrix and the rotation matrix.

[0034] Optionally, the scene reconstruction guidance network includes a feature point extraction and descriptor construction module, and an optimal feature point matching module;

[0035] The feature point extraction and descriptor construction module includes an interest point decoder for obtaining feature points at specific locations and their confidence levels, and a descriptor decoder for adjusting the channel of the feature map and normalizing the feature map to obtain feature point descriptors.

[0036] The optimal feature point matching module is used to obtain the mixed features of the outputs of the interest point decoder and the descriptor decoder, and to determine the attention map corresponding to the mixed features using a self-attention mechanism and a cross-attention mechanism, to determine the optimal feature points between the attention maps of adjacent frames, and to perform scene reconstruction based on the optimal feature points.

[0037] Optionally, the heterogeneous feature interaction module uses a variety of preset heterogeneous information aggregation strategies, employs at least two parallel one-dimensional convolutions along the X-axis and at least two parallel one-dimensional convolutions along the Y-axis to extract the feature maps, and then concatenates the extracted results before inputting them into the subsequent convolution module.

[0038] Optionally, the processing steps of the local texture smoothing module include:

[0039] Based on the parallel computation model, the sum of the absolute values ​​of the gradients in the two vertical directions of the sampled image frame is calculated, and the norm of the sum of the absolute values ​​of the gradients is calculated.

[0040] Determine whether the texture threshold corresponding to the norm is less than a preset texture threshold;

[0041] If so, Gaussian processing is applied to the prediction mask and the image frame, and the weighted sum between the Gaussian-processed prediction mask and the image frame is determined as a local Gaussian-enhanced image for the target region.

[0042] Furthermore, to achieve the above objectives, the present invention also provides a UAV ground target localization system, the UAV ground target localization system comprising: a memory, a processor, and a UAV ground target localization program based on monocular depth estimation stored in the memory and executable on the processor, wherein when the UAV ground target localization program based on monocular depth estimation is executed by the processor, it implements the steps of the UAV ground target localization method based on monocular depth estimation as described above.

[0043] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a UAV ground target localization program based on monocular depth estimation, wherein the UAV ground target localization program based on monocular depth estimation, when executed by a processor, implements the steps of the UAV ground target localization method based on monocular depth estimation as described above.

[0044] This invention provides a UAV ground target localization method, a UAV ground target localization system, and a computer-readable storage medium based on monocular depth estimation. The method incorporates a hybrid architecture consisting of multiple visual transformers and a single convolutional neural network (CNN) within a pre-defined network architecture to enhance the receptive domain of the deep learning network for remote sensing videos. An attitude network and a scene reconstruction guidance network are also included in the pre-defined network architecture to adapt to the shooting angles when the UAV acquires remote sensing videos at various attitudes during flight. A heterogeneous feature interaction module is introduced into the pre-defined network architecture to blend the feature maps of the visual transformers and the CNN, thereby improving the depth estimation performance of the locally perceptualized CNN on the UAV. A local texture smoothing module is used in the pre-defined network architecture to reduce the texture appearance level in specific regions, thereby improving the accuracy of the UAV in estimating the depth of objects in remote sensing images. These features overcome the performance deficiencies of traditional network architectures, extracting more easily identifiable depth feature maps, thus facilitating accurate object localization in acquired remote sensing videos by the UAV based on these depth feature maps. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the hardware operating environment of the UAV ground target positioning system according to an embodiment of the present invention;

[0046] Figure 2 This is a flowchart illustrating the first embodiment of the UAV ground target localization method based on monocular depth estimation according to the present invention.

[0047] Figure 3 This is a schematic diagram of the architecture of the SRG network involved in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the architecture of the HFI module involved in an embodiment of the present invention;

[0049] Figure 5 This is a flowchart illustrating a second embodiment of the UAV ground target localization method based on monocular depth estimation according to the present invention.

[0050] Figure 6 This is a flowchart illustrating the third embodiment of the UAV ground target localization method based on monocular depth estimation of the present invention.

[0051] Figure 7 This is a schematic diagram of the evaluation results of the test points involved in an embodiment of the present invention;

[0052] Figure 8 This is a flowchart illustrating the fourth embodiment of the UAV ground target localization method based on monocular depth estimation of the present invention.

[0053] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] This application sets up a hybrid architecture consisting of multiple visual transformers and a single convolutional neural network in the preset network architecture to improve the perceptual domain size of remote sensing videos in deep learning networks.

[0055] An attitude network and a scene reconstruction guidance network are set in the preset network architecture to adapt to the shooting angle when the UAV collects remote sensing video in various attitudes during flight.

[0056] A heterogeneous feature interaction module is introduced into the preset network architecture to mix the feature maps of the visual transformer and the convolutional neural network, thereby improving the depth estimation performance of the convolutional neural network based on local perception on the UAV.

[0057] The local texture smoothing module is used in the preset network architecture to reduce the texture appearance level of specific regions, thereby improving the accuracy of UAV in estimating the depth of objects in remote sensing images.

[0058] The above settings overcome the performance defects of traditional network architectures and extract more easily identifiable depth feature maps, thereby enabling UAVs to accurately locate objects in the acquired remote sensing videos based on the depth feature maps.

[0059] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0060] As one implementation scheme, Figure 1 This is a schematic diagram of the hardware operating environment of the UAV ground target positioning system involved in the embodiments of the present invention.

[0061] like Figure 1As shown, the UAV ground target positioning system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0062] Those skilled in the art will understand that Figure 1 The architecture of the UAV ground target positioning system shown in the figure does not constitute a limitation on the UAV ground target positioning system. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0063] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a UAV ground target localization program based on monocular depth estimation. The operating system is a program that manages and controls the hardware and software resources of the UAV ground target localization system, the UAV ground target localization program based on monocular depth estimation, and the operation of other software or programs.

[0064] exist Figure 1 In the UAV ground target positioning system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the UAV ground target positioning program based on monocular depth estimation stored in the memory 1005.

[0065] In this embodiment, the UAV ground target localization system includes: a memory 1005, a processor 1001, and a UAV ground target localization program based on monocular depth estimation stored in the memory and executable on the processor, wherein:

[0066] When processor 1001 calls the UAV ground target localization program based on monocular depth estimation stored in memory 1005, it performs the following operations:

[0067] Based on a preset sampling interval, sampled image frames are acquired from remote sensing videos captured by the monocular camera of the UAV.

[0068] The sampled image frame is input into a preset network architecture for feature extraction to obtain a depth feature map. The preset network architecture includes multiple visual transformers and a single convolutional neural network. The preset network architecture includes a pose network for outputting translation and rotation matrices, a scene reconstruction guidance network for reconstructing scenes from the sampled image frame, a heterogeneous feature interaction module for mixing the feature maps of the visual transformers and the convolutional neural network, and a local texture smoothing module for reducing the texture representation level of the target region in the sampled image frame.

[0069] The location of the target object in the remote sensing video is determined based on the depth feature map.

[0070] When processor 1001 calls the UAV ground target localization program based on monocular depth estimation stored in memory 1005, it performs the following operations:

[0071] Based on a preset convolutional layer, the sampled image frame is cropped to a preset size;

[0072] The sampled image cropped to the preset scale is input into a multi-scale path to extract local features of multiple different pixels, wherein patch labels of different scales are embedded in parallel in the multi-scale path.

[0073] Each of the local features is input in parallel to the multipath transformer layer to obtain the processing results based on the outputs of multiple visual transformers and a single convolutional neural network in the multipath transformer layer. The processing results are then connected by skip connections to obtain a feature map. In the visual transformer, the input local features are globally modeled using a multi-head self-attention strategy to adjust the focus of different regions in the local features.

[0074] The feature map is input into the heterogeneous feature interaction module to enhance the global structural information of the feature map and obtain an interactive feature map.

[0075] The interactive feature map is input into the decoder layer, and the features output by the decoder layer are obtained as the deep feature map. The decoder layer includes a convolutional layer and an activation function layer.

[0076] When processor 1001 calls the UAV ground target localization program based on monocular depth estimation stored in memory 1005, it performs the following operations:

[0077] Calculate the normal map corresponding to the depth feature map;

[0078] Determine the ground reference point based on the orientation of the pixels in the normal map;

[0079] Calculate the relative flight altitude of the UAV based on the ground reference point;

[0080] Obtain the current flight altitude of the drone, and determine the scaling factor based on the ratio of the current flight altitude to the relative flight altitude;

[0081] The absolute depth value of a pixel is determined by multiplying the scaling factor by the relative depth value of each pixel in the depth feature map.

[0082] The current latitude and longitude of the UAV are obtained, and the positioning result of the target object is determined based on the absolute depth value and the current latitude and longitude.

[0083] When processor 1001 calls the UAV ground target localization program based on monocular depth estimation stored in memory 1005, it performs the following operations:

[0084] Obtain a preset combination of loss functions;

[0085] The training loss result of the preset network architecture is constrained by the preset loss function combination, wherein the preset loss function combination includes image reconstruction loss function, image gradient discrimination loss function, matching guidance loss function, minimum luminosity loss function and total loss function;

[0086] The step of constraining the training loss result of the preset network architecture based on the preset loss function combination includes:

[0087] Based on the image reconstruction loss function, the variation error loss of the depth feature map relative to the sampled image frame is constrained; and,

[0088] Based on the image gradient discrimination loss function, the gradient discrimination loss is constrained after performing second-order gradient smoothing on the image region transition of the depth feature map; and...

[0089] Based on the matching-guided loss function, the error loss result is calculated by constraining the maximum scale in the depth feature map; and...

[0090] Based on the minimum photometric loss function, constrain the photometric calculation error loss in the depth feature map; and...

[0091] Based on the total loss function, the change error loss, the gradient discrimination loss, the maximum scale calculation error loss, and the photometric calculation error loss are averaged to determine the overall loss result of the preset network architecture.

[0092] Based on the hardware architecture of the UAV ground target localization system based on computer vision technology described above, an embodiment of the UAV ground target localization method based on monocular depth estimation of the present invention is proposed.

[0093] Reference Figure 2 In the first embodiment, the UAV ground target localization method based on monocular depth estimation includes the following steps:

[0094] Step S10: Based on a preset sampling interval, acquire sampled image frames from the remote sensing video captured by the monocular camera of the UAV.

[0095] In this embodiment, firstly, based on a preset sampling interval, a single frame sampled image from the remote sensing video captured by the monocular camera set on the UAV is obtained as a sampled image frame.

[0096] Step S20: Input the sampled image frame into a preset network architecture for feature extraction to obtain a depth feature map;

[0097] Step S30: Determine the target object location result in the remote sensing video based on the depth feature map.

[0098] In this embodiment, the sampled image frames are input into a preset deep network model architecture for feature extraction. The deep network model extracts the features of the selected target object in the sampled image frames to form a depth feature map, thereby perceiving the relative position of the target object relative to the UAV based on the depth feature map, and thus realizing the localization of the target object in the remote sensing video collected by the UAV.

[0099] In this embodiment, the preset network architecture mainly includes five parts:

[0100] I. Compared with the traditional deep network architecture based purely on convolutional neural network layers, a hybrid architecture consisting of multiple visual transformers and a single convolutional neural network is proposed. This allows the network to extract image features in parallel from the visual transformers and convolutional neural network layers through multiple paths, explore global and local information in a hierarchical manner, and improve the perceptual domain size of remote sensing videos in deep learning networks.

[0101] 2. An attitude network with multiple degrees of freedom for outputting translation and rotation matrices to adapt to the shooting angle when the UAV is collecting remote sensing video in various attitudes during flight.

[0102] Specifically, the pose network adopts the ResNet-18 network architecture. The pose network loads preset network weights and performs estimated self-motion transformation on the sampled image frames to transform multiple sampled image frames into the translation matrix and the rotation matrix.

[0103] It should be noted that self-motion transformation refers to the movement or motion of the drone itself relative to its surrounding environment within its perception system.

[0104] Thirdly, due to the limitations of pose networks based purely on convolutional neural network layers, there is still room for improvement in acquiring long-term dependency information in complex environments. Therefore, this embodiment introduces a Scene Reconstruction Guided (SRG) network based on graph neural networks, referred to as the SRG network. It estimates the self-motion of the entire image based on the correlation of feature matching across the entire image. Similar to pose networks, this network takes a combination of a reference image and neighboring frames as input and outputs a multi-degree-of-freedom pose vector as a training guide for the pose network.

[0105] For example, the architecture of an SRG network is as follows: Figure 3 As shown, the network consists of two key parts: a feature point extraction and descriptor construction module, and an optimal feature point matching module. The feature point extraction and descriptor construction module includes an interest point decoder for obtaining feature points at specific locations and their confidence levels, and a descriptor decoder for adjusting the channels of the feature map and normalizing the feature map to obtain feature point descriptors. The optimal feature point matching module is used to obtain the mixed features of the outputs of the interest point decoder and the descriptor decoder, and uses self-attention and cross-attention mechanisms to determine the attention map corresponding to the mixed features, determine the optimal feature points between the attention maps of adjacent frames, and perform scene reconstruction based on the optimal feature points.

[0106] Specifically, the feature point extraction and descriptor construction module uses the VGG-16 model to extract image features, forming a feature map. Then, the feature map is feature-mapped and simultaneously input into the interest point decoder and descriptor decoder. The features pass through multiple convolutional layers in the interest point decoder and are then activated with softmax to obtain specific points of interest and their confidence levels. Simultaneously, the descriptor decoder uses 3×3 convolutional kernels to adjust the channels of the feature map and then normalizes the features to obtain feature point descriptors.

[0107] Specifically, the optimal feature points can be determined by using a sine angle algorithm to find feature point matching between adjacent sampled image frames.

[0108] It should be noted that, considering the generalization and scale ambiguity of monocular depth estimation methods, SRG networks cannot be used alone, nor can they be used as a replacement for pose networks. In most cases, during the training phase, the overall architecture uses the rotation matrix generated by this network to guide the reconstruction of the reference image.

[0109] Fourth, traditional methods introduce different attention mechanisms into the decoder part of deep networks, only considering the interaction of information between internal channels. This weakens the overall spatial structure of the target and leads to limitations in local optimization and capturing long-range dependencies. In this embodiment, a Heterogeneous Feature Interaction (HFI) module is introduced to mix the feature maps of the visual transformer and the convolutional neural network, thereby improving the depth estimation performance of the convolutional neural network based on local perception on the UAV.

[0110] For example, the architecture of the HFI module is as follows: Figure 4 As shown. Specifically, the heterogeneous feature interaction module has multiple preset strategies for aggregating heterogeneous information. After selecting one as the target strategy for aggregating heterogeneous information, at least two parallel one-dimensional convolutions along the X-axis and at least two parallel one-dimensional convolutions along the Y-axis are used to extract the feature map. The extracted results are then concatenated and input into the subsequent convolution module.

[0111] 5. Since the pixel color and spatial coordinates of the sampled image frames change continuously during the flight of the UAV, it is difficult for the deep network to learn accurate structural features directly from color cues for areas with texture changes (or areas with similar surrounding pixels). Therefore, in this embodiment, the Local Texture Smoothing (LTS) module is used to reduce the texture appearance level of specific areas, thereby improving the accuracy of the UAV in estimating the depth of objects in remote sensing images.

[0112] Optionally, the processing steps of the local texture smoothing module include: calculating the sum of the absolute values ​​of the gradients in two vertical directions of the sampled image frame based on a parallel computation model, and calculating the norm of the sum of the absolute values ​​of the gradients;

[0113] To determine whether a pixel is located in a region with texture similarity, it is determined whether the texture threshold corresponding to the norm is less than a preset texture threshold. If so, it means that the pixel is located in the region. At the same time, Gaussian processing is performed on the prediction mask and the image frame. The prediction mask and the image frame after Gaussian processing are weighted and summed. The result obtained after weighted summing is determined as the local Gaussian enhanced image of the target region in the sampled image frame.

[0114] In this embodiment, after the preset network architecture extracts the depth feature map, the UAV can determine the relative distance between the target object in the remote sensing video and the UAV itself based on the depth feature map, thereby determining the positioning result of the target object.

[0115] In the technical solution provided in this embodiment, a hybrid architecture consisting of multiple visual transformers and a single convolutional neural network is set in the preset network architecture to improve the perceptual domain size of the deep learning network for remote sensing videos. An attitude network and a scene reconstruction guidance network are set in the preset network architecture to adapt to the shooting angles when the UAV acquires remote sensing videos in various attitudes during flight. A heterogeneous feature interaction module is introduced into the preset network architecture to mix the feature maps of the visual transformers and the convolutional neural network, thereby improving the depth estimation performance of the local perception-based convolutional neural network on the UAV. A local texture smoothing module is used in the preset network architecture to reduce the texture appearance level of specific regions, thereby improving the accuracy of the UAV in estimating the depth of objects in remote sensing images. These settings overcome the performance defects of traditional network architectures, extract more easily identifiable depth feature maps, and facilitate the UAV in accurately locating objects in the acquired remote sensing videos based on the depth feature maps.

[0116] Reference Figure 5 In the second embodiment, based on the first embodiment, step S20 includes:

[0117] Step S21: Based on a preset convolutional layer, crop the sampled image frame to a preset scale;

[0118] Step S22: Input the sampled image cropped to the preset scale into the multi-scale path to extract local features of multiple different pixels, wherein patch labels of different scales are embedded in parallel in the multi-scale path;

[0119] Step S23: Input each of the local features in parallel into the multipath transformer layer, obtain the processing results in the multipath transformer layer based on the outputs of multiple visual transformers and a single convolutional neural network, and perform skip connections on the processing results to obtain feature mapping. In the visual transformer, the input local features are globally modeled through a multi-head self-attention strategy to adjust the focus degree of different regions in the local features.

[0120] Step S24: Input the feature map into the heterogeneous feature interaction module to enhance the global structural information of the feature map through the heterogeneous feature interaction module, thereby obtaining an interactive feature map;

[0121] Step S25: Input the interactive feature map into the decoder layer and obtain the features output by the decoder layer as the deep feature map, wherein the decoder layer includes a convolutional layer and an activation function layer.

[0122] As an optional embodiment, this embodiment describes the extraction of features within a preset network architecture to generate a deep feature map.

[0123] First, based on a preset convolutional layer, the sampled image frame is cropped to a preset size. Specifically, two 3×3 convolutional layers can be used as the preset convolutional layers to crop the sampled image frame.

[0124] Furthermore, the sampled image cropped to a preset scale is input into the multi-scale path in the preset network architecture. Patch labels of different scales are embedded in parallel in the multi-scale path to extract local features of multiple different pixels in the sampled image.

[0125] Furthermore, the various local features are input in parallel into the multipath transformer layer in the network architecture. After the multipath transformer layer processes the multiple visual transformers and the single convolutional neural network, it performs skip connections based on the processed output to obtain the feature map.

[0126] In the visual transformer, the input local features are globally modeled using a multi-head self-attention strategy to adjust the degree of focus in different regions of the local features.

[0127] The specific formula is as follows:

[0128]

[0129] In the formula, Q, K′, and V represent query, key, and value, respectively, C represents the number of embedding channels, and softmax is the activation function.

[0130] Furthermore, feature maps from multiple skip connections of different sizes are received, and the feature maps are input into the heterogeneous feature interaction module to balance feature heterogeneity through the heterogeneous feature interaction module, thereby enhancing the global structural information of the feature maps. The enhanced global structural information yields a fused interactive feature map, which is then upsampled and input into the next decoder layer.

[0131] Finally, the interactive feature map is input into the decoder layer to obtain the features output by the decoder layer as the deep feature map. The decoder layer includes convolutional layers and activation function layers. Specifically, the activation function can be the sigmoid activation function, and the convolutional layers can be 3×3 convolutional layers.

[0132] In the technical solution provided in this embodiment, the sampling image frame is processed in five stages to obtain a depth feature map, thereby improving the dependence of the traditional deep network architecture on local perception and increasing the size of the receptive field.

[0133] Reference Figure 6 In the third embodiment, based on any embodiment, step S30 includes:

[0134] Step S31: Calculate the normal map corresponding to the depth feature map;

[0135] Step S32: Determine the ground reference point based on the orientation of the pixels in the normal map;

[0136] Step S33: Calculate the relative flight altitude of the UAV based on the ground reference point;

[0137] Step S34: Obtain the current flight altitude of the UAV, and determine the scaling factor based on the ratio of the current flight altitude to the relative flight altitude;

[0138] Step S35: The product of the scaling factor and the relative depth value of each pixel in the depth feature map is determined as the absolute depth value of the pixel.

[0139] Step S36: Obtain the current latitude and longitude of the UAV, and determine the positioning result of the target object based on the absolute depth value and the current latitude and longitude.

[0140] As an optional embodiment, this embodiment specifically illustrates how to determine a relatively accurate object localization result from the deep feature map obtained from the trained preset network architecture.

[0141] First, calculate the normal map corresponding to the depth feature map.

[0142] Specifically, the least squares method can be used to calculate the normal map n:

[0143]

[0144] Where n = [n_x, n_y, n_z] represents the normal map formed by the normal vectors on the tangent plane formed by the pixels in the 8 neighboring regions of any valid pixel; A represents the matrix formed by the three-dimensional coordinates of the projection of any valid pixel.

[0145] Furthermore, ground reference points are determined based on the orientation of pixels in the normal map.

[0146] Specifically, assuming the angle between the normal direction of a pixel at an ideal ground reference point and the angle perpendicular to the ground plane upwards is less than 5 degrees, and considering that the UAV's field of view is typically 45° downwards, the normal direction of the ground plane can be empirically determined as follows:

[0147]

[0148] Next, the similarity between the normal direction of each effective pixel in the entire image and the normal direction of the ground plane reference point is calculated based on the cosine function, and points with an angle difference of less than 5 degrees are marked as ground reference points.

[0149] Furthermore, the relative flight altitude of the UAV is calculated based on the ground reference point.

[0150] Specifically, by projecting the depth map back into 3D space, the length of the normal vector magnitude of the pixels of the ground reference point in 3D space can be calculated, and the estimated relative flight altitude of the UAV can be obtained by taking the median of the Gaussian distribution.

[0151] Furthermore, the current flight altitude H of the drone is obtained. UAV Based on the current flight altitude and the relative flight altitude H estimate The ratio is used to determine the scaling factor SF:

[0152]

[0153] Furthermore, the product of the scaling factor SF and the relative depth value of each pixel in the depth feature map D_rel is determined as the absolute depth value D of the pixel. abs :

[0154] D abs =SF*D_rel

[0155] Furthermore, the current latitude and longitude of the drone are obtained, and the positioning result of the target object is determined based on the absolute depth value and the current latitude and longitude.

[0156] In this step, the latitude and longitude of the target object in the world coordinate system are determined based on the absolute depth value and the current latitude and longitude, and then the latitude and longitude of the target object in the world coordinate system are used as the positioning result of the target object.

[0157] Specifically, since the z-axis in the camera coordinate system coincides with the camera's optical axis and points outwards, the transformation between the camera coordinate system and the world coordinate system is as follows: First, rotate the camera coordinate system around the positive x-axis by an angle Ang_x1, so that the z-axis is perpendicular to the horizontal plane. Then, rotate the coordinate system around the y-axis by an angle, which we denote as Ang_y2. Finally, rotate the coordinate system around the z-axis by an angle of (270° - Ang_z3) to ensure that the camera coordinate system and the world coordinate system maintain the same orientation.

[0158] Rotation matrix R x1y2z3 The calculation is as follows:

[0159] R x1y2z3 =R X (90°+Ang_x1)×R Y (Ang_y2)R Z (270°-Ang_z3)

[0160] Assume the origin O of the camera coordinate system camThe location coincides with that of the drone, and the origin of the world coordinate system is denoted as O. world O world O cam The coordinates of the UAV in the world coordinate system are represented as (X... oc Y oc Z oc The conversion method for each value is as follows:

[0161] X oc =(N G +H UAV )×cosAng_y2×cosAng_z3

[0162] Y oc =(N G +H UAV )×cosAng_y2×sinAng_z3

[0163]

[0164]

[0165]

[0166] The radius of the Earth's equator is denoted as R. CHI The radius of the Earth's North and South Poles is denoted as R. NB The elevation information of the UAV is denoted as N. G .

[0167] Finally, the ground targets in the image are converted to coordinates in the world coordinate system, calculated as follows: [X world ,Y world Z world ] T =R x1y2z3 ×inv_K×[U,V,1] T ×D abs +O world O cam

[0168] Where X world ,Y world Z world Indicates the coordinates of the ground target in the world coordinate system, and inv_K represents the inverse of the intrinsic parameter matrix of the UAV camera.

[0169] In addition, to prove the accuracy of this positioning result, refer to Figure 7 The test point evaluation results shown indicate that the overall positioning error has been improved.

[0170] The technical solution provided in this embodiment specifically illustrates how to determine a relatively accurate object localization result from the depth feature map obtained from the trained preset network architecture, thereby improving the localization accuracy of locating remote sensing images of target objects collected by UAVs.

[0171] Reference Figure 8 In the fourth embodiment, based on any embodiment, before step S30, the method further includes:

[0172] Step S40: Obtain a preset combination of loss functions;

[0173] Step S50: The training loss result of the preset network architecture is constrained based on the preset loss function combination, wherein the preset loss function combination includes image reconstruction loss function, image gradient discrimination loss function, matching guidance loss function, minimum luminosity loss function and total loss function;

[0174] As an optional embodiment, in the self-supervised depth estimation task of the preset network architecture, image synthesis is a key step. The reconstruction of the input frame is used as the supervision signal for training. Therefore, in this embodiment, several loss terms are designed to constrain the training of the remote sensing scene network framework.

[0175] Step S50 includes:

[0176] Step S51, based on the image reconstruction loss function, constrain the change error loss of the depth feature map relative to the sampled image frame; and,

[0177] For example, the image reconstruction loss function L ir The constraints are as follows:

[0178]

[0179] in, I t (p) represents the sampled image frame, I t+n→t (p) is the depth feature map, T is the sampling period, L1 is the Manhattan distance, and SSIM() is the structural similarity function.

[0180] Step S52, based on the image gradient discrimination loss function, constrain the gradient discrimination loss after performing second-order gradient smoothing on the image region transition of the depth feature map; and,

[0181] In this embodiment, an image gradient recognition loss function is used to improve the predicted depth map. A second-order gradient derivative is added to the first-order derivative to smooth the transitions between image regions.

[0182] For example, the image gradient discriminative loss function L igd The constraints are as follows:

[0183]

[0184] in, and These represent the first and second derivatives of the image coordinates, respectively.

[0185] Step S53: Based on the matching guided loss function, calculate the error loss result by constraining the maximum scale in the depth feature map; and,

[0186] In this embodiment, in order to obtain semantically accurate deep representations and thus accurate image reconstruction results to guide the training of the pose network, a matching-guided loss function L is designed. mg Constrain the scene reconstruction guidance network.

[0187] For example, the matching guidance loss function L mg The constraints are as follows:

[0188]

[0189] Where ψ is the number of sampled image frames, For I t+n→t (p) The output P based on the PoseNet network after processing by the local texture smoothing module. t+n→t ξ is a constant term, ξ = 10 -3 .

[0190] Step S54: Based on the minimum photometric loss function, constrain the photometric calculation error loss in the depth feature map; and,

[0191] In this embodiment, minimum photometric error calculation is used to reduce the reprojection interference between consecutive sampled image frames caused by obstructions.

[0192] For example, the minimum photometric loss function L mp The constraints are as follows:

[0193] L mp (p)=min(L ir (I t (p), I t+n (p)),L ir (I t (p), I t+n→t (p)))

[0194] Among them, I t+n (p) represents adjacent sampling frames, I t+n→t (p) represents the composite target frame between adjacent sampling frames.

[0195] Step S55: Based on the total loss function, the change error loss, the gradient discrimination loss, the maximum scale calculation error loss, and the photometric calculation error loss are averaged to determine the overall loss result of the preset network architecture.

[0196] In this embodiment, the various loss results obtained above are averaged and combined with the matching guided loss calculated only on the error loss λ at a maximum scale to obtain the overall loss result.

[0197] For example, the total loss function L total The constraints are as follows:

[0198]

[0199] Where ν, τ, and κ are preset constants.

[0200] In the technical solution provided in this embodiment, several loss terms are designed to constrain the training of the remote sensing scene network framework in the self-supervised depth estimation task of the preset network architecture, thereby improving the training accuracy of the preset network architecture.

[0201] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the XX system to implement the process steps of the embodiments of the above methods.

[0202] Therefore, the present invention also provides a computer-readable storage medium storing a UAV ground target localization program based on monocular depth estimation, wherein the UAV ground target localization program based on monocular depth estimation, when executed by a processor, implements the various steps of the XX method as described in the above embodiments.

[0203] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0204] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0205] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0207] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0208] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0209] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0210] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0211] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A UAV ground target localization method based on monocular depth estimation, characterized in that, The method includes the following steps: Based on a preset sampling interval, sampled image frames are acquired from remote sensing videos captured by the monocular camera of the UAV. The sampled image frame is input into a preset network architecture for feature extraction to obtain a depth feature map. The preset network architecture includes multiple visual transformers and a single convolutional neural network. The preset network architecture includes a pose network for outputting translation and rotation matrices, a scene reconstruction guidance network for reconstructing scenes from the sampled image frame, a heterogeneous feature interaction module for mixing the feature maps of the visual transformers and the convolutional neural network, and a local texture smoothing module for reducing the texture representation level of the target region in the sampled image frame. The target object localization result in the remote sensing video is determined based on the depth feature map; Before the step of determining the target object localization result in the remote sensing video based on the depth feature map, the method further includes: Obtain a preset combination of loss functions; The training loss result of the preset network architecture is constrained by the preset loss function combination, wherein the preset loss function combination includes image reconstruction loss function, image gradient discrimination loss function, matching guidance loss function, minimum luminosity loss function and total loss function; The step of constraining the training loss result of the preset network architecture based on the preset loss function combination includes: Based on the image reconstruction loss function, the variation error loss of the depth feature map relative to the sampled image frame is constrained; and, Based on the image gradient discrimination loss function, the gradient discrimination loss is constrained after performing second-order gradient smoothing on the image region transition of the depth feature map; and... Based on the matching-guided loss function, the error loss result is calculated by constraining the maximum scale in the depth feature map; and... Based on the minimum photometric loss function, constrain the photometric calculation error loss in the depth feature map; and... Based on the total loss function, the change error loss, the gradient discrimination loss, the maximum scale calculation error loss, and the photometric calculation error loss are averaged to determine the overall loss result of the preset network architecture; The scene reconstruction guidance network includes a feature point extraction and descriptor construction module, as well as an optimal feature point matching module; The feature point extraction and descriptor construction module includes an interest point decoder for obtaining feature points at specific locations and their confidence levels, and a descriptor decoder for adjusting the channel of the feature map and normalizing the feature map to obtain feature point descriptors. The optimal feature point matching module is used to obtain the mixed features of the outputs of the interest point decoder and the descriptor decoder, and to determine the attention map corresponding to the mixed features using a self-attention mechanism and a cross-attention mechanism, to determine the optimal feature points between the attention maps of adjacent frames, and to perform scene reconstruction based on the optimal feature points.

2. The method as described in claim 1, characterized in that, The step of inputting the sampled image frame into a preset network architecture for feature extraction to obtain a depth feature map includes: Based on a preset convolutional layer, the sampled image frame is cropped to a preset size; The sampled image cropped to the preset scale is input into a multi-scale path to extract local features of multiple different pixels, wherein patch labels of different scales are embedded in parallel in the multi-scale path. Each of the local features is input in parallel to the multipath transformer layer to obtain the processing results based on the outputs of multiple visual transformers and a single convolutional neural network in the multipath transformer layer. The processing results are then connected by skip connections to obtain a feature map. In the visual transformer, the input local features are globally modeled using a multi-head self-attention strategy to adjust the focus of different regions in the local features. The feature map is input into the heterogeneous feature interaction module to enhance the global structural information of the feature map and obtain an interactive feature map. The interactive feature map is input into the decoder layer, and the features output by the decoder layer are obtained as the deep feature map. The decoder layer includes a convolutional layer and an activation function layer.

3. The method as described in claim 1, characterized in that, The step of determining the target object localization result in the remote sensing video based on the depth feature map includes: Calculate the normal map corresponding to the depth feature map; Determine the ground reference point based on the orientation of the pixels in the normal map; Calculate the relative flight altitude of the UAV based on the ground reference point; Obtain the current flight altitude of the drone, and determine the scaling factor based on the ratio of the current flight altitude to the relative flight altitude; The absolute depth value of a pixel is determined by multiplying the scaling factor by the relative depth value of each pixel in the depth feature map. The current latitude and longitude of the UAV are obtained, and the positioning result of the target object is determined based on the absolute depth value and the current latitude and longitude.

4. The method as described in claim 1, characterized in that, The attitude network adopts the ResNet-18 network architecture. The attitude network calls preset network weights to perform self-motion transformation on the sampled image frames, so as to transform multiple sampled image frames into the translation matrix and the rotation matrix.

5. The method as described in claim 1, characterized in that, The heterogeneous feature interaction module uses a variety of preset heterogeneous information aggregation strategies, employs at least two parallel one-dimensional convolutions along the X-axis and at least two parallel one-dimensional convolutions along the Y-axis to extract feature maps, and then concatenates the extracted results before inputting them into the subsequent convolution module.

6. The method as described in claim 1, characterized in that, The processing steps of the local texture smoothing module include: Based on the parallel computation model, the sum of the absolute values ​​of the gradients in the two vertical directions of the sampled image frame is calculated, and the norm of the sum of the absolute values ​​of the gradients is calculated. Determine whether the texture threshold corresponding to the norm is less than a preset texture threshold; If so, Gaussian processing is applied to the prediction mask and the image frame, and the weighted sum between the Gaussian-processed prediction mask and the image frame is determined as a local Gaussian-enhanced image for the target region.

7. A ground target positioning system for unmanned aerial vehicles (UAVs), characterized in that, The UAV ground target localization system includes: a memory, a processor, and a UAV ground target localization program based on monocular depth estimation stored in the memory and executable on the processor. When the UAV ground target localization program based on monocular depth estimation is executed by the processor, it implements the steps of the UAV ground target localization method based on monocular depth estimation as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a UAV ground target localization program based on monocular depth estimation, which, when executed by a processor, implements the steps of the UAV ground target localization method based on monocular depth estimation as described in any one of claims 1 to 6.

Images