A low-altitude unmanned aerial vehicle multi-view perception method based on a transformer

Abstract

The application discloses a low-altitude unmanned aerial vehicle multi-view perception method based on a Transformer, and belongs to the technical field of image processing, and comprises the following steps: acquiring images shot by multiple cameras; taking Depth Anything as a basic monocular depth estimation model to obtain a relative depth map corresponding to each image, and correcting the relative depth map by adopting a visual angle self-supervised depth correction mechanism VAST; based on the corrected relative depth map, a dynamic scale fusion model TDSF driven by a Transformer is used to fuse visual depth and navigation information through the Transformer, an optimal scale factor is dynamically estimated, and an absolute depth map is obtained based on the optimal scale factor; and based on the absolute depth map, a sparse visual angle BEV fusion model DSSA using dynamic spatial self-attention is used to convert into a BEV representation to obtain a final BEV feature map. The method improves the accuracy of three-dimensional perception.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a multi-view perception method for low-altitude unmanned aerial vehicles based on Transformer. Background Technology

[0002] The application of low-altitude unmanned aerial vehicles in urban logistics, disaster relief and other scenarios is increasing, which puts forward higher requirements for real-time perception of the surrounding environment.

[0003] Due to limitations in airborne load and energy consumption, constructing dense 3D maps using multiple cameras or LiDAR is often impractical. A more common approach is to acquire images using a single or a small number of cameras, then estimate the depth and convert it into a BEV representation. However, existing depth estimation models such as RSA, Hybrid Transformer, and DynaDepth suffer from scale uncertainty and poor geometric consistency. The depth pre-training model Depth Anything V2, trained on large-scale synthetic data and pseudo-labels, improves prediction accuracy. DynaDepth enhances scale robustness by fusing image feature IMUs; however, these methods lack adaptive correction for aircraft attitude errors, making it difficult to meet the depth estimation requirements of low-altitude aircraft in complex attitudes. Regarding BEV representation, Lift-Splat-Shoot (LSS) projects image features onto a bird's-eye view plane, but suffers from low spatial resolution and cannot handle dynamic scenes. Methods such as SDG-OCC and BEVFusion improve occupancy prediction accuracy through multimodal fusion or 3D convolution, but incur high computational costs. While these methods are effective in autonomous driving, they still suffer from dense computation and insufficient real-time performance in low-altitude UAV scenarios.

[0004] Therefore, designing a multi-view perception framework that can adapt to the attitude changes of low-altitude aircraft while taking into account scale consistency and real-time performance remains a challenge. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this application provides a Transformer-based multi-view perception method for low-altitude UAVs, which solves the problems of large relative depth prediction errors caused by the characteristics of low-altitude flight scenarios, scale ambiguity in monocular methods, and geometric deviations and low information utilization in bird's-eye view (BEV) fusion under sparse perspectives.

[0006] To achieve the aforementioned objectives, the technical solution adopted in this application is as follows:

[0007] This application provides a Transformer-based multi-view perception method for low-altitude unmanned aerial vehicles, including:

[0008] S1: Acquire images captured by multiple cameras;

[0009] S2: Using Depth Anything as the basic monocular depth estimation model, the relative depth map corresponding to each image is obtained, and the view self-supervised depth correction mechanism VAST is used to correct the relative depth map.

[0010] S3: Based on the corrected relative depth map, the Transformer-driven dynamic scale fusion model TDSF is used to fuse visual depth and navigation information through Transformer, dynamically estimate the optimal scale factor, and obtain the absolute depth map based on the optimal scale factor.

[0011] S4: Based on the absolute depth map, the sparse view BEV fusion model DSSA with dynamic spatial self-attention is used to convert it into a BEV representation, and the final BEV feature map is obtained.

[0012] Furthermore, the VAST (View-Based Self-Supervised Depth Correction) mechanism for correcting the relative depth map includes:

[0013] A1: Extract intermediate feature maps from the Depth Anything encoder, and use three independent convolutional mappings on the target pixel and its neighboring pixels to obtain the query vector, key vector, and value vector:

[0014]

[0015] in, For query vector, For key vectors, For value vectors, , and These are the linear projection matrices used in the Transformer to generate the query, key, and value, respectively. For the Depth Anything encoder in the camera The intermediate feature map, Image pixel coordinates, for The neighborhood set, For neighboring pixels, For the first One camera, and Represents the pixel coordinates in the horizontal and vertical directions;

[0016] A2: Calculate the similarity between the query vector of the target pixel and the key vector of each pixel in the neighborhood, and obtain the attention weights through normalization:

[0017]

[0018] in, Attention weights, superscript For transpose, The key vector is obtained by linear projection of the neighboring pixels. The normalization factor for the feature dimension. Indicates in neighborhood Other pixels traversed in the middle;

[0019] A3: The corrected relative depth value is obtained based on attention weight aggregation, and the corrected relative depth value is:

[0020]

[0021] in, for Corrected relative depth value This represents the original relative depth.

[0022] A4: Construct the photometric reprojection error as the photometric loss function, and introduce a gradient smoothing regularization term as the smoothing loss function. Smooth and suppress noise on the corrected relative depth values ​​to obtain the total loss function. :

[0023]

[0024]

[0025]

[0026] in, Let be the photometric loss function. Let slip loss function, The weight hyperparameters are used to smooth the loss. For the set of valid pixels, Indicates time The images in The value at that location, Indicates time An image is a function of pixel coordinates to pixel values. For the camera provided by the navigation sensor at any time arrive The pose change matrix, This is a transformation function that backprojects pixels onto 3D and then onto the coordinate system of the next frame image according to the corrected depth. Describing the L1 norm, and These represent the difference operators in the horizontal and vertical directions, respectively.

[0027] Furthermore, based on the corrected relative depth map, the Transformer-driven Dynamic Scale Fusion Model (TDSF) is used to fuse visual depth and navigation information through a Transformer, dynamically estimating the optimal scale factor, and obtaining the absolute depth map based on the optimal scale factor, including:

[0028] S301: Obtain the initial scale factor using the known absolute altitude of the aircraft:

[0029]

[0030] in, This is the initial scale factor. This refers to the actual altitude of the aircraft. It is the set of pixels representing the ground region in a top-down camera image. This is the relative depth value corrected for a top-down camera view. Represents the average value;

[0031] S302: Calculate the geometric scale factor based on the relative and absolute displacements of the navigation sensors. :

[0032]

[0033] in, This is relative displacement. The absolute displacement vector provided for the navigation sensor. Let be the magnitude of the vector;

[0034] S303: Based on the initial scale factor, the Transformer-driven dynamic scale fusion model TDSF is used to fuse multimodal temporal information, refine and smooth the geometric scale factor, and output the final optimal scale factor.

[0035] S304: Multiply the relative depth map by the optimal scale factor by pixels to obtain the absolute depth map.

[0036] Furthermore, the relative displacement of the navigation sensor includes:

[0037] B1: Acquire two consecutive frames of images, the corrected depth map, and the pose transformation and absolute displacement vector from the navigation sensor;

[0038] B2: Assume that the set of matching feature points in two consecutive frames is... The corrected relative depth is and Using the intrinsic parameter matrix and attitude transformation The pixels are back-projected into three-dimensional space to obtain the projected pixels:

[0039]

[0040] in, and This indicates that the same scene point matched in two consecutive frames is at time [time value missing]. and pixel coordinates, To match the number of feature pairs, and They are respectively and The corresponding corrected relative depth, and The coordinates are homogeneous pixels, with superscript. Represents the camera intrinsic parameter matrix The inverse matrix, and They are respectively and The corresponding projected pixels;

[0041] B3: Relative displacement calculated using depth matching based on the projected pixels. It can be obtained from the difference in coordinates between the two points:

[0042]

[0043] in, and It is the rotation and translation from the navigation sensor.

[0044] Furthermore, based on the initial scale factor, the Transformer-driven Dynamic Scale Fusion Model (TDSF) is used to fuse multimodal temporal information to refine and smooth the geometric scale factor, outputting the final optimal scale factor, including:

[0045] C1: Constructing visual feature vectors and navigation feature vectors The two are then concatenated and projected onto a unified dimension through a fully connected layer to obtain the input labels for the Transformer:

[0046]

[0047] in, and For learnable matrices, Input markers for the Transformer;

[0048] C2: Employs the Transformer-driven dynamic scale fusion model TDSF's multi-head self-attention mechanism for historical data. Frame input token sequence Perform feature interaction:

[0049]

[0050] in, For querying the matrix, The key matrix, For value matrices, For history The first frame in the input token sequence Frame input marker, This is the output of the multi-head self-attention operation;

[0051] C3: Based on the input labeled sequence after feature interaction, the Transformer outputs the scale increment predicted by the feedforward neural network FFN. :

[0052]

[0053] in, For the first Scale factor after frame fusion;

[0054] C4: Based on geometric scale factor and scale increment Update the scale factor for the next time step. :

[0055]

[0056] C5: Based on the scale factor, and using the reprojection error of multiple frames as supervision, a loss function is constructed to optimize the Transformer parameters, thereby obtaining the optimal scale factor. The loss function is:

[0057]

[0058] in, For projection function, For loss function, Indicates time Three-dimensional rotation operator, For a moment The three-dimensional translation vector, For feature points at the th Pixel coordinates of a frame This represents the L2 norm.

[0059] Furthermore, the sparse-view BEV fusion model DSSA based on the absolute depth map and utilizing dynamic spatial self-attention is transformed into a BEV representation to obtain the final BEV feature map, including:

[0060] S401: For the first The corrected pixel and absolute depth maps from each camera are used to calculate the corresponding 3D point coordinates based on the camera's intrinsic and extrinsic parameters. :

[0061]

[0062] in, For the first The intrinsic parameter matrix of each camera, and For the first External parameters of a camera Absolute depth;

[0063] S402: Convert 3D point coordinates Top-down projection yields BEV mesh elements. Network coordinates and intermediate feature maps By weight Aggregation yields the initial BEV feature representation ,in, and Represents network coordinates in the horizontal and vertical directions;

[0064] S403: Define the BEV query vector To characterize the state of BEV network cells, the visible field of view is considered. The system determines whether a BEV network unit is visible based on the camera's view frustum, and calculates the position of the BEV network unit on the camera. Attention weights in :

[0065]

[0066] in, For learnable matrices, For the first The feature vector of the camera mapped to the BEV spatial cell. Represents the set of all visible cameras. For the first Feature vectors of the camera on the BEV grid cell;

[0067] S404: Define the geometric deviation term, taking into account the projection deviation between different cameras. :

[0068]

[0069] in, The coordinates of the projection point, For the center of gravity of the grid, These are learnable bias weights;

[0070] S405: Combines attention weights and geometric bias terms to fuse multi-camera features into a BEV query. :

[0071]

[0072] S406: Query the BEV of all grids The final BEV feature map is formed.

[0073] The beneficial effects of this application are:

[0074] This application presents a Transformer-based multi-view perception method for low-altitude UAVs, achieving end-to-end optimization of visual perception tasks for low-altitude aircraft. The VAST module utilizes the Transformer's self-attention mechanism to correct monocular depth under unlabeled conditions, combining attitude compensation and viewpoint transformation to obtain accurate oblique depth. The TDSF module, based on geometric scale recovery, uses Transformer to fuse visual and navigation information, dynamically estimating the scale factor, alleviating the scale ambiguity problem of monocular depth and achieving real-time performance. The DSSA module uses depth-guided sparse mapping and spatial self-attention fusion to achieve efficient BEV representation under sparse viewpoints, overcoming the shortcomings of traditional LSS pipelines in depth accuracy and computational efficiency. The overall framework design fully utilizes navigation sensor and multi-view information, improving the accuracy and robustness of 3D perception while maintaining end-to-end trainability of the algorithm, laying a solid foundation for autonomous obstacle avoidance and path planning for low-altitude aircraft. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0076] Figure 1 This is a flowchart illustrating a multi-view perception method for low-altitude unmanned aerial vehicles based on Transformer, provided in an embodiment of this application.

[0077] Figure 2 The design framework diagram of the three algorithm models of depth estimation, scale fusion and spatial fusion provided in the embodiments of this application is shown.

[0078] Figure 3 A schematic diagram of rolling compensation provided for an embodiment of this application.

[0079] Figure 4 This is a structural diagram of the TDSF provided in an embodiment of this application. Detailed Implementation

[0080] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0081] Example 1:

[0082] Related information:

[0083] Monocular depth estimation can be divided into self-supervised and supervised methods. RSA models utilize linguistic descriptions to help resolve scale ambiguity; Hybrid Transformer combines Vision Transformer and convolutional neural networks to perform hybrid feature modeling; Depth Anything V2 trains its depth estimation model using 595,000 precisely annotated synthetic images and 62 million pseudo-annotated real images, outperforming version V1 in detail prediction; DynaDepth integrates image feature IMU dynamic information to address scale drift. However, these methods lack adaptive correction for aircraft attitude errors, making it difficult to meet the depth estimation requirements of low-altitude aircraft under complex attitudes.

[0084] Lift-Splat-Shoot (LSS) backprojects 2D features to 3D and then projects them onto the BEV plane, but loses 50% of depth pixels. SDG-OCC improves BEV occupancy prediction within the LSS framework using semantic and depth guidance; BEVFusion employs a unified multi-task, multi-sensor fusion architecture, fusing LiDAR and multi-camera features into the BEV; SurroundOcc improves BEV representation through 2D-3D pre-occupancy prediction followed by 3D reconstruction; OccTransformer introduces a Transformer based on BEVFormer (bird's-eye view transformer) to improve 3D occupancy prediction. These methods are effective in autonomous driving, but still suffer from dense computation and insufficient real-time performance in low-altitude drone scenarios.

[0085] Based on this, this application provides a Transformer-based multi-view perception method for low-altitude UAVs, which can be found in [reference needed]. Figure 1 This method proposes a new algorithm design space from three aspects: depth estimation, scale fusion, and spatial fusion. (See [link to relevant documentation]). Figure 2 , Figure 1The diagram shown is a flowchart illustrating a Transformer-based multi-view perception method for low-altitude unmanned aerial vehicles (UAVs) according to an embodiment of this application, including:

[0086] S1: Acquire images captured by multiple cameras.

[0087] S2: Using Depth Anything as the basic monocular depth estimation model, the relative depth map corresponding to each image is obtained, and the view self-supervised depth correction mechanism VAST is used to correct the relative depth map.

[0088] In one embodiment of this application, Depth Anything is used as the basic monocular depth estimation model. This model, pre-trained with large-scale real and synthetic images, exhibits strong generalization ability across different scenes. Compared to version V1, Depth Anything V2 significantly improves the accuracy and robustness of depth prediction by using synthetic data to replace real annotations, expanding the capacity of the teacher model, and training the student model with large-scale pseudo-annotated data. Given the lack of large-scale labeled data for low-altitude flight scenarios, this application uses this model as the base network and further adapts it for flight perspectives.

[0089] Camera setup: Two opposing fisheye cameras and one top-down wide-angle camera simultaneously acquire images. Assume the... Images captured by the camera are Its output relative depth map is ,in For pixel coordinates, and Represents the pixel coordinates in the horizontal and vertical directions.

[0090] Adaptation strategy: For images with large pitch angles and obvious fisheye distortion, simulation data is added during training, and data augmentation is used to simulate different attitudes, angular velocities, dynamic lighting, etc., to further improve the model's adaptability to flight scenarios.

[0091] While Depth Anything exhibits good generalization ability, it still generates systematic errors in scenes with large pitch angles and distortions. Therefore, this application proposes a View-Adaptive Self-supervised Transformer (VAST) depth correction mechanism. This module utilizes the Transformer's self-attention mechanism and leverages the photometric consistency between adjacent frames to construct a self-supervised signal, enabling dynamic correction of the depth map without requiring additional ground truth depth annotations. Specifically, it includes the following:

[0092] Feature representation: The intermediate features output by the Depth Anything encoder are denoted as... For pixel position Take its neighboring area The features within are used as key-value pairs. Query vector Key vector and value vector Obtained by linear projection, it is the penultimate feature layer (256 channels) of the Depth Anything encoder through three independent... The feature dimension is obtained by convolution mapping. As shown in the formula:

[0093]

[0094] in, The height of the feature map, The width of the feature map. This represents the number of channels in the intermediate feature map. , and These are the linear projection matrices used in the Transformer to generate the query, key, and value, respectively. for Take one of the surrounding local neighborhood sets. The square window can capture local structure while controlling the amount of computation. The three-dimensional coordinates used to distinguish matching feature points in two frames of images For the first There are two cameras. The features output by the attention layer are mapped to depth increments through a feedforward network (two fully connected layers), and then added to the original depth to obtain the corrected depth.

[0095] Self-attention correction: for each pixel Calculate its attention weights:

[0096]

[0097] in, Attention weights are used to measure neighboring pixel pairs. Contribution, superscript For transpose, These are the key vectors obtained by linear projection of the neighboring pixels, used to calculate and query the vector. similarity, The normalization factor for the feature dimension. Indicates in neighborhood The other pixels traversed are used to normalize the weights of all neighboring pixels.

[0098] The corrected depth value is obtained through attention convergence:

[0099]

[0100] in, for Corrected relative depth value This represents the original relative depth.

[0101] Self-supervised loss: To avoid relying on ground truth depth, this study employs a photometric consistency constraint between two consecutive frames and designs photometric reprojection error as the photometric loss function.

[0102]

[0103] in, For the set of valid pixels, Indicates time The images in The value at that location, Indicates time An image is a function of pixel coordinates to pixel values. For the camera provided by the navigation sensor at any time arrive The pose change matrix, This is a transformation function that backprojects pixels onto 3D and then onto the coordinate system of the next frame image according to the corrected depth. This represents the L1 norm.

[0104] Smoothing Regularization: To alleviate noise and ensure local smoothness of the depth map, a gradient smoothing regularization term is added as a smoothing loss function.

[0105]

[0106] in, and These represent the difference operators in the horizontal and vertical directions, respectively.

[0107] Total loss function: The training optimization objective of the VAST module is:

[0108]

[0109] in, To smooth the weight hyperparameters of the loss, the self-supervised loss employs photometric reprojection error and a first-order gradient smoothing regularization term, with a weight ratio of... Photometric consistency is calculated within a three-frame sliding window to enhance temporal constraints. This loss encourages the corrected depth to maintain geometric consistency across consecutive frames.

[0110] In one embodiment of this application, the low-altitude aircraft exhibits significant roll and pitch motions, which can cause an angular deviation between the image coordinate system and the actual horizontal plane. Therefore, attitude compensation is required within the pixel plane to ensure that subsequent depth and viewpoint calculations are performed in a unified reference frame.

[0111] Roll compensation: Let the original pixel coordinates (with the image center as the origin) be... The roll angle of the aircraft is (The positive direction is the clockwise rotation of the machine body along the longitudinal axis, in radians), such as Figure 3 As shown. Compensated coordinates. By rotation matrix Transformation:

[0112]

[0113] Viewpoint offset calculation: for the compensated pixels Horizontal offset angle and vertical offset angle They are defined as follows:

[0114]

[0115] in, and This represents the compensated pixel coordinates in the horizontal and vertical directions. This is the camera focal length (in pixels, which can be obtained by conversion from the intrinsic parameter matrix).

[0116] True slant depth calculation: Depth output by the network The depth component, which is parallel to the image plane, represents the actual distance from the optical center to the point. Both horizontal and vertical offset angles need to be considered:

[0117]

[0118] in, This represents the actual tangential depth.

[0119] S3: Based on the corrected relative depth map, the Transformer-driven dynamic scale fusion model TDSF is used to fuse visual depth and navigation information through Transformer, dynamically estimate the optimal scale factor, and obtain the absolute depth map based on the optimal scale factor.

[0120] In one embodiment of this application, an inherent drawback of monocular depth estimation is the inability to directly obtain the absolute scale; that is, the predicted depth only reflects relative distances. Existing technologies indicate that scale ambiguity and scale inconsistency severely limit the application of depth estimation in tasks such as obstacle avoidance and autonomous landing in UAV applications. To obtain absolute depth, this application draws upon the geometric scale recovery concept from existing technologies: utilizing the feature correspondence between two consecutive frames and the pose information provided by the navigation sensor, the ratio of relative displacement to absolute displacement is calculated as a scale factor. Then, the relative depth map is multiplied by this scale factor pixel by pixel to obtain the absolute depth map. This method requires only a monocular camera and a common navigation sensor, does not rely on a calibration board or ground constraints, and its robustness to sensor noise has been verified on the public dataset Mid-Air.

[0121] To enhance the real-time performance and robustness of scale recovery, this study designs a Transformer-driven Dynamic Scale Fusion (TDSF) module based on geometric recovery. This module fuses visual depth, image features, and navigation information (IMU, GPS) via a Transformer to dynamically estimate the scale factor. Figure 4 As shown, the process of visual features and navigation features being projected and used as input to the Transformer, with the Transformer layer outputting the scale increment, is illustrated.

[0122] Visual feature vectors It includes statistics such as the mean, standard deviation, and maximum value of the corrected depth map (4 dimensions in total) and a segment of image texture features after convolutional encoding (32 dimensions); navigation feature vector. It consists of the average and standard deviation of IMU triaxial acceleration and angular velocity (6 dimensions each) and displacement information (3 dimensions) provided by GPS. After concatenation, a fully connected layer projects the result onto a 64-dimensional array, which serves as the input token for the Transformer. A multi-head self-attention mechanism with four heads (each 16-dimensional) is employed, and two Transformer layers are stacked. (History sequence length...) Choosing a value of 5 considers both recent scale changes and avoids the computational burden of excessively long sequences. The output scale increment is projected as a scalar through the feedforward network and then compared with the geometric scale. Adding them together yields the updated scale. The smoothing factor is updated through a moving average window (length 10) to reduce the impact of IMU noise.

[0123] Initial scale estimation: First, the initial scale factor is obtained using the known absolute altitude of the aircraft. Let the set of pixels in the ground region of the overhead camera image be... The corresponding correction depth is The actual altitude of the aircraft is Then the initial scaling factor for:

[0124]

[0125] in, This represents the average value.

[0126] This operation overcomes the scale uncertainty of monocular depth by using the ratio of the average relative depth to the true height as the initial scale.

[0127] Geometric scale correction: Considering two consecutive frames and The absolute displacement vector provided by the navigation sensor is denoted as... The relative displacement is obtained by matching feature points in the depth map. Let the set of matched feature points in the two images be... The corrected depths are respectively and Using the intrinsic parameter matrix and attitude transformation Projecting the pixels back into three-dimensional space, we have:

[0128]

[0129] in, and This indicates that the same scene point matched in two consecutive frames is at time [time value missing]. and pixel coordinates, To match the number of feature pairs, and They are respectively and The corresponding corrected depth, and The coordinates are homogeneous pixels, with superscript. Represents the camera intrinsic parameter matrix The inverse matrix, and They are respectively and The corresponding projected pixels.

[0130] Relative displacement calculated by depth matching It can be obtained from the difference in coordinates between the two points:

[0131]

[0132] in, and It is the rotation and translation from the navigation sensor.

[0133] Final geometric scale factor The values ​​are obtained from the ratio of absolute to relative displacement:

[0134]

[0135] in, Let the magnitude of the vector be . This is relative displacement. The absolute displacement vector provided to the navigation sensor is used to achieve smooth updates of the scale factor through a sliding window or exponential averaging, thereby suppressing the influence of noise.

[0136] Transformer Fusion Update: Geometric methods provide a coarse estimate of scale, but in dynamic flight scenarios, navigation sensor noise and matching errors can cause scale factors to become unstable. Drawing on the successful application of the vision-inertial fusion Transformer in attitude estimation, this application introduces a Transformer module for cross-modal fusion and dynamic updating of scale.

[0137] Feature construction: Let's assume at time... The scaling factor is Define visual feature vectors It consists of depth map statistics (such as mean and variance), image texture features, and geometric scale factors; navigation feature vector. This includes IMU three-axis acceleration, angular velocity, and GPS displacement. The two data points are then stitched together and projected onto a unified dimension through a linear layer.

[0138]

[0139] in, and For learnable matrices, For the input markers of the Transformer, It is the set of real numbers.

[0140] Cross-modal self-attention: Employs a multi-head self-attention mechanism to process historical data. Frame input token sequence Perform feature interaction:

[0141]

[0142] in, For querying the matrix, The key matrix, The value matrix is ​​obtained by projecting the input labels through different linear projections. For history The first frame in the token sequence Frame input marker, This is the output of the multi-head self-attention operation.

[0143] The scale increment of the Transformer output is predicted by the feedforward neural network FFN. :

[0144]

[0145] in, For the first Scale factor after frame fusion The scale increment predicted by the Transformer. It is a feedforward neural network.

[0146] Scale Update: Integrate the geometric scaling factor and the increment predicted by the Transformer to update the scale for the next time step. :

[0147]

[0148] Training objective: Using the reprojection error of multiple frames as supervision, optimize the Transformer parameters so that the estimated scale factor can minimize the projection error of 3D points in each frame.

[0149]

[0150] in, For projection function, For loss function, Indicates time Three-dimensional rotation operator, For a moment The three-dimensional translation vector, For feature points at the th Pixel coordinates of a frame This represents the L2 norm.

[0151] S4: Based on the absolute depth map, the sparse view BEV fusion model DSSA with dynamic spatial self-attention is used to convert it into a BEV representation, and the final BEV feature map is obtained.

[0152] In one embodiment of this application, in multi-camera bird's-eye view fusion, the common Lift-Splat-Shoot (LSS) pipeline projects the depth distribution of each pixel into the BEV space by discretizing it. This method is widely used in lightweight models, but existing technologies point out that LSS has two main problems: first, the depth estimation error is large, and it cannot fully utilize geometric and semantic information; second, the BEV space utilization is low, with only about 50% of the mesh being effective, resulting in a large amount of redundant computation. Especially for sparse view aircraft systems, over-reliance on dense meshes will significantly increase computational overhead and reduce real-time performance.

[0153] To this end, this application proposes a Dynamic Spatial Self-Attention (DSSA) module, which achieves efficient BEV representation from a sparse perspective by fusing explicit depth-guided feature mapping and self-attention under visible domain constraints.

[0154] 1. Deeply Guided 2D-3D Mapping

[0155] Coordinate transformation: for the first Corrected pixels in each camera and absolute depth Based on the camera's intrinsic parameter matrix and external references Calculate the corresponding 3D point coordinates :

[0156]

[0157] in, For the first The intrinsic parameter matrix of each camera, and For the first External parameters of a camera This is the absolute depth, and the three-dimensional point is represented in the aircraft coordinate system. Its top-view projection... For locating BEV mesh cells, a BEV mesh with a resolution of 0.5m is selected, where, and Represents the network coordinates in the horizontal and vertical directions; the weights of each pixel are adjusted according to the depth confidence during projection, and the confidence is given by the inverse variance of the deep network output.

[0158] Feature projection: Projecting intermediate feature maps Multiply by the depth-guided weight and project onto the corresponding BEV mesh cell. The weights are determined by depth confidence or semantic information to emphasize reliable spatial structure. For each grid cell, features from different camera projections are summed or averaged, and intermediate feature maps are used. By weight Aggregation yields the initial BEV feature representation. .

[0159] 2. Dynamic Spatial Self-Attention Fusion

[0160] BEV Query Definition: Define the BEV query vector. Characterizing mesh cells The state. The feature vector of each camera mapped to the BEV space is denoted as... Define a query for each BEV mesh cell. (Dimension 64), Features of each camera projected onto this grid (Dimension 64) is mapped to keys and values ​​through a linear layer.

[0161] Attention weights: considering the field of view (i.e., camera) (The set of meshes observable in BEV) determines whether a mesh is visible based on the camera's view frustum, and calculates the mesh. In the camera Attention weights in :

[0162]

[0163] in, For learnable matrices, For the first The feature vectors mapped from the camera to the BEV space, Represents the set of all visible cameras. For the first The feature vector of the camera on the BEV grid cell.

[0164] Geometric deviation correction: Considering the projection deviation between different cameras, a geometric deviation term is defined. :

[0165]

[0166] in, The coordinates of the projection point, For the center of gravity of the grid, The bias weights are learnable.

[0167] Geometric deviation term The distance between the camera projection point and the grid centroid is calculated using a single-layer MLP (outputting 1D weights) and used as an additive bias for the attention weights. The attention mechanism employs single-head self-attention with an output dimension of 64. This term is used to adjust the attention weights and suppress mismatches caused by parallax.

[0168] Fusion Update: By combining attention weights and geometric biases, features from multiple cameras are fused into a BEV query.

[0169]

[0170] in, For the attention weights after softmax normalization, apply them to all grids. Forming the final BEV feature map .

[0171] The self-attention learning process is optimized using the loss function of downstream tasks (such as occupancy prediction and object detection).

[0172] Example 2:

[0173] To further explain this design scheme, an experiment is designed to illustrate the design method in detail:

[0174] 1. Experimental hardware and software configuration

[0175] Hardware environment: All experiments were conducted on a workstation equipped with an Intel Core i9-13900K processor, 32GB RAM, and an NVIDIA RTX 4090 graphics card with 24GB of GPU memory, capable of simultaneously training deep networks and Transformer models. Sensor experiments used a two-way fisheye camera (1024×1024 resolution, 90 FOV) and an IMU / GPS module mounted on a low-altitude aircraft. The IMU sampling rate was 200Hz, and the GPS frequency was 1Hz.

[0176] Software environment: The operating system is Ubuntu 22.04 LTS; the deep learning framework used is PyTorch 2.2; the Transformer layer is implemented using Hugging Face; tables and charts are drawn using Matplotlib and Seaborn; data generation and experimental code are all implemented using Python 3.10.

[0177] 2. Dataset and Preprocessing

[0178] Public synthetic data: To verify the algorithm's depth estimation and spatial perception capabilities in low-altitude flight scenarios, the Mid-Air dataset, constructed using the AirSim platform (an aeronautical informatics and robotics simulation platform), was selected. This dataset contains 54 low-altitude flight trajectories, each providing RGB images, camera intrinsic and extrinsic parameters, depth maps, and navigation sensor data. The image resolution is 1024×1024 pixels, with a field of view of 90 degrees. This experiment selected 24 trajectories from the "PLE" environment as the training set and 5 trajectories as the test set, using only four weather conditions under spring conditions: sunny, sunset, foggy, and cloudy.

[0179] Custom AirSim trajectories: To verify the applicability of the model under different speed conditions, this application designed two additional flight paths in the AirSim open-source scenario "Landscape Mountains":

[0180] Uniform velocity trajectory: The speed is constant at 8m / s, and a total of 845 frames of images are collected;

[0181] Variable speed trajectory: The speed varied between 4-12 m / s, and a total of 856 frames of images were collected.

[0182] These two trajectories have the same resolution and field of view as the Mid-Air dataset and are used to test the adaptability of the dynamic scale fusion module to changes in motion.

[0183] Real-world UAV scenarios: In addition, this application collected three real outdoor low-altitude flight videos, using a fisheye camera and IMU / GPS to record data, to verify the generalization ability of the algorithm. The real data underwent distortion correction and synchronization processing, and the true depth was estimated based on GPS altitude and flight logs.

[0184] 3. Comparison Algorithm

[0185] To comprehensively evaluate the effectiveness of the proposed method, representative methods in depth estimation and BEV perception from the past three years were selected for comparison.

[0186] Table 1 Comparison Algorithms

[0187]

[0188] Continued table

[0189]

[0190] All comparison algorithms were trained and tested on the Mid-Air dataset using the authors' open-source code and default settings. Where supported by the original paper, this application also fine-tunes or directly evaluates the models on custom trajectories and real UAV data to ensure fair comparisons.

[0191] 4. Evaluation Indicators

[0192] For depth estimation tasks, the following metrics are used to measure error and accuracy:

[0193] Absolute relative error (Abs Rel): .

[0194] Squared relative error (Sq Rel): .

[0195] Root Mean Square Error (RMSE): .

[0196] Root mean square error (RMSE log): .

[0197] accuracy / / :satisfy The pixel ratio, where , The number of effective pixels used in the evaluation. For the first The predicted depth at each valid pixel. For the first The true depth at each valid pixel.

[0198] In the BEV occupancy prediction task, the accuracy of space occupancy is evaluated using mean inter-intersection over union (mIoU), precision, and recall. Furthermore, the frame rate (FPS) for each method is reported to measure real-time performance.

[0199] 5. Experimental Results and Analysis

[0200] Depth Estimation Performance Comparison: Table 2 presents the depth estimation results of this application and eight comparative algorithms on the Mid-Air test set. For ease of comparison, all relative depth predictions were converted to absolute scale using standard "median scaling" and then compared with the absolute depth output by the viewpoint correction and dynamic scale fusion employed in this application. Smaller numbers indicate lower errors. The larger the value, the higher the precision.

[0201] Table 2. Depth estimation results of this application and eight comparative algorithms on the Mid-Air test set.

[0202]

[0203] As shown in Table 2, the method in this application outperforms the comparative algorithms on all error metrics. In particular, AbsRel and Sq Rel achieve approximately 14%-20% lower accuracy compared to DynaDepth, which has the closest performance. The accuracy was improved to 0.98. This is due to: ① The VAST module significantly reduced viewpoint distortion through self-supervised correction; ② The TDSF module dynamically estimated the scale factor using visual statistics and navigation information to solve the monocular scale blur problem; ③ Attitude compensation and simulation data augmentation enhanced the model's adaptability in scenarios with large pitch angles and fisheye distortion.

[0204] BEV Occupancy Prediction Performance: To evaluate the advantages of DSSA in sparse viewpoint BEV representation, this application compares it with the LSS pipeline and the SDG-OCC deep semantic fusion scheme. All methods use the same backbone visual encoder to output BEV occupancy maps (80×80 grid). Table 3 shows the mIoU, Precision, Recall, and frame rate on the Mid-Air test set.

[0205] Table 3 shows mIoU, Precision, Recall, and Frame Rate in the Mid-Air test set.

[0206]

[0207] The results show that the DSSA module, by utilizing depth confidence and visual field constraints, improves the mIoU of BEV occupancy prediction by approximately 9 percentage points. Furthermore, compared to the LSS method, the method in this application only slightly reduces the frame rate (22 FPS) but significantly improves accuracy, making it suitable for real-time flight control.

[0208] To analyze the contribution of each submodule, this application conducted ablation experiments:

[0209] Table 4 Ablation Experiment

[0210]

[0211] As can be seen, removing any module will lead to a performance decrease, with the removal of VAST having the greatest impact, indicating that view self-supervised correction is particularly critical in low-altitude flight scenarios.

[0212] Parameter and setting analysis:

[0213] Frame interval: Changes the interval between the reference frame and the current frame. exist Frame-by-frame measurement Abs Rel results show When Abs Rel=0.12, The value increased to 0.14 at one time. The value was 0.18, which verifies that the smaller the interval, the more sufficient the feature matching and the more accurate the scale estimation.

[0214] Transformer Head Count and Layer Count: Increasing the number of heads in TDSF from 2 to 8 improved Abs Rel by less than 1%, but increased runtime by about 15%; therefore, the study adopted a balanced configuration of 4 heads and 2 layers.

[0215] Scale fusion sliding window: Increasing the window length from 5 frames to 15 frames resulted in smoother scale estimation, but significantly reduced real-time performance. A length of 10 was ultimately selected.

[0216] This application achieves end-to-end optimization of visual perception tasks for low-altitude aircraft by introducing three modules: VAST, TDSF, and DSSA. The VAST module utilizes the Transformer's self-attention mechanism to correct monocular depth under unlabeled conditions, combining attitude compensation and viewpoint transformation to obtain accurate oblique depth. The TDSF module, based on geometric scale recovery, uses a Transformer to fuse visual and navigation information, dynamically estimating the scale factor, mitigating the scale ambiguity problem of monocular depth and achieving real-time performance. The DSSA module, through depth-guided sparse mapping and spatial self-attention fusion, achieves efficient BEV representation under sparse viewpoints, overcoming the shortcomings of traditional LSS pipelines in depth accuracy and computational efficiency. The overall framework design fully utilizes navigation sensor and multi-view information, improving the accuracy and robustness of 3D perception while maintaining end-to-end trainability of the algorithm, laying a solid foundation for autonomous obstacle avoidance and path planning for low-altitude aircraft.

[0217] It should be noted that those skilled in the art will recognize that the embodiments described herein are for the purpose of helping readers understand the principles of this application, and should be understood as not limiting the scope of protection of this application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this application without departing from the essence of this application, and these modifications and combinations are still within the scope of protection of this application.

Claims

1. A multi-view perception method for low-altitude unmanned aerial vehicles based on Transformer, characterized in that, include: S1: Acquire images captured by multiple cameras; S2: Using Depth Anything as the basic monocular depth estimation model, the relative depth map corresponding to each image is obtained, and the view self-supervised depth correction mechanism VAST is used to correct the relative depth map. S3: Based on the corrected relative depth map, the Transformer-driven dynamic scale fusion model TDSF is used to fuse visual depth and navigation information through Transformer, dynamically estimate the optimal scale factor, and obtain the absolute depth map based on the optimal scale factor. S4: Based on the absolute depth map, the sparse view BEV fusion model DSSA with dynamic spatial self-attention is used to convert it into a BEV representation, and the final BEV feature map is obtained.

2. The Transformer-based multi-view perception method for low-altitude UAVs according to claim 1, characterized in that, The method of using the viewpoint self-supervised depth correction mechanism (VAST) to correct the relative depth map includes: A1: Extract intermediate feature maps from the Depth Anything encoder, and use three independent convolutional mappings on the target pixel and its neighboring pixels to obtain the query vector, key vector, and value vector: in, For query vector, For key vectors, For value vectors, , and These are the linear projection matrices used in the Transformer to generate the query, key, and value, respectively. For the Depth Anything encoder in the camera The intermediate feature map, Image pixel coordinates, for The neighborhood set, For neighboring pixels, For the first One camera, and Represents the pixel coordinates in the horizontal and vertical directions; A2: Calculate the similarity between the query vector of the target pixel and the key vector of each pixel in the neighborhood, and obtain the attention weights through normalization: in, Attention weights, superscript For transpose, The key vector is obtained by linear projection of the neighboring pixels. The normalization factor for the feature dimension. Indicates in neighborhood Other pixels traversed in the middle; A3: The corrected relative depth value is obtained based on attention weight aggregation, and the corrected relative depth value is: in, for Corrected relative depth value This represents the original relative depth. A4: Construct the photometric reprojection error as the photometric loss function, and introduce a gradient smoothing regularization term as the smoothing loss function. Smooth and suppress noise on the corrected relative depth values ​​to obtain the total loss function. : in, Let be the photometric loss function. Let slip loss function, The weight hyperparameters are used to smooth the loss. For the set of valid pixels, Indicates time The images in The value at that location, Indicates time An image is a function of pixel coordinates to pixel values. For the camera provided by the navigation sensor at any time arrive The pose change matrix, This is a transformation function that backprojects pixels onto 3D and then onto the coordinate system of the next frame image according to the corrected depth. Describing the L1 norm, and These represent the difference operators in the horizontal and vertical directions, respectively.

3. The Transformer-based multi-view perception method for low-altitude UAVs according to claim 2, characterized in that, The method, based on the corrected relative depth map, utilizes the Transformer-driven Dynamic Scale Fusion Model (TDSF) to fuse visual depth and navigation information via Transformer, dynamically estimating the optimal scale factor, and then obtaining the absolute depth map based on the optimal scale factor, including: S301: Obtain the initial scale factor using the known absolute altitude of the aircraft: in, This is the initial scale factor. This refers to the actual altitude of the aircraft. It is the set of pixels representing the ground region in a top-down camera image. This is the relative depth value corrected for a top-down camera view. Represents the average value; S302: Calculate the geometric scale factor based on the relative and absolute displacements of the navigation sensors. : in, This is relative displacement. The absolute displacement vector provided for the navigation sensor. Let be the magnitude of the vector; S303: Based on the initial scale factor, the Transformer-driven dynamic scale fusion model TDSF is used to fuse multimodal temporal information, refine and smooth the geometric scale factor, and output the final optimal scale factor. S304: Multiply the relative depth map by the optimal scale factor by pixels to obtain the absolute depth map.

4. The Transformer-based multi-view perception method for low-altitude UAVs according to claim 3, characterized in that, The relative displacement of the navigation sensor includes: B1: Acquire two consecutive frames of images, the corrected depth map, and the pose transformation and absolute displacement vector from the navigation sensor; B2: Assume that the set of matching feature points in two consecutive frames is... The corrected relative depth is and Using the intrinsic parameter matrix and attitude transformation The pixels are back-projected into three-dimensional space to obtain the projected pixels: in, and This indicates that the same scene point matched in two consecutive frames is at time [time value missing]. and pixel coordinates, To match the number of feature pairs, and They are respectively and The corresponding corrected relative depth, and The coordinates are homogeneous pixels, with superscript. Represents the camera intrinsic parameter matrix The inverse matrix, and They are respectively and The corresponding projected pixels; B3: Relative displacement calculated using depth matching based on the projected pixels. It can be obtained from the difference in coordinates between the two points: in, and It is the rotation and translation from the navigation sensor.

5. The Transformer-based multi-view perception method for low-altitude UAVs according to claim 4, characterized in that, The process involves refining and smoothing the geometric scale factor based on the initial scale factor, using the Transformer-driven Dynamic Scale Fusion Model (TDSF) to fuse multimodal temporal information, and outputting the final optimal scale factor. This includes: C1: Constructing visual feature vectors and navigation feature vectors The two are then concatenated and projected onto a unified dimension through a fully connected layer to obtain the input labels for the Transformer: in, and For learnable matrices, Input markers for the Transformer; C2: Employs the Transformer-driven dynamic scale fusion model TDSF's multi-head self-attention mechanism for historical data. Frame input token sequence Perform feature interaction: in, For querying the matrix, The key matrix, For value matrices, For history The first frame in the input token sequence Frame input marker, This is the output of the multi-head self-attention operation; C3: Based on the input labeled sequence after feature interaction, the Transformer outputs the scale increment predicted by the feedforward neural network FFN. : in, For the first Scale factor after frame fusion; C4: Based on geometric scale factor and scale increment Update the scale factor for the next time step. : C5: Based on the scale factor, and using the reprojection error of multiple frames as supervision, a loss function is constructed to optimize the Transformer parameters, thereby obtaining the optimal scale factor. The loss function is: in, For projection function, For loss function, Indicates time Three-dimensional rotation operator, For a moment The three-dimensional translation vector, For feature points at the th Pixel coordinates of a frame This represents the L2 norm.

6. The Transformer-based multi-view perception method for low-altitude UAVs according to claim 5, characterized in that, The absolute depth map-based sparse-view BEV fusion model DSSA, utilizing dynamic spatial self-attention, is used to transform the data into a BEV representation, resulting in the final BEV feature map, which includes: S401: For the first The corrected pixel and absolute depth maps from each camera are used to calculate the corresponding 3D point coordinates based on the camera's intrinsic and extrinsic parameters. : in, For the first The intrinsic parameter matrix of each camera. and For the first External parameters of a camera Absolute depth; S402: Convert 3D point coordinates Top-down projection yields BEV mesh elements. Network coordinates and intermediate feature maps By weight Aggregation yields the initial BEV feature representation ,in, and Represents network coordinates in the horizontal and vertical directions; S403: Define the BEV query vector To characterize the state of BEV network cells, the visible field of view is considered. The system determines whether a BEV network unit is visible based on the camera's view frustum, and calculates the position of the BEV network unit on the camera. Attention weights in : in, For learnable matrices, For the first The feature vector of the camera mapped to the BEV spatial cell. Represents the set of all visible cameras. For the first Feature vectors of the camera on the BEV grid cell; S404: Define the geometric deviation term, taking into account the projection deviation between different cameras. : in, The coordinates of the projection point, For the center of gravity of the grid, These are learnable bias weights; S405: Combines attention weights and geometric bias terms to fuse multi-camera features into a BEV query. : S406: Query the BEV of all grids The final BEV feature map is formed.

Images