A visual navigation method, system, terminal and storage medium for a six-rotor full-drive drone

Abstract

The present invention relates to the field of drone perception technology, and discloses a visual navigation method, system, terminal and storage medium for a six-rotor full-drive drone. The present invention constructs a navigation framework that integrates dynamic visual perception and full-drive control characteristics, uses a visual encoder to extract multi-scale environmental features, and combines a long-short-term memory network to process temporal information. This achieves efficient feature extraction and motion trajectory prediction for image sequences under high-speed rotation and drastic posture changes of the drone, solves the problem of feature tracking loss in high-dynamic scenes caused by traditional algorithms, strengthens global correlation modeling of environmental features through an attention mechanism, and combines control instruction generation with dynamic constraints. This significantly improves the navigation accuracy and robustness of the drone in complex dynamic environments, without relying on high-precision inertial navigation compensation. This provides an efficient and reliable navigation solution for the application of six-rotor full-drive drones in scenarios such as emergency rescue and high-speed inspection.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) perception technology, and in particular to a visual navigation method, system, terminal, and computer-readable storage medium for a six-rotor fully-driven UAV. Background Art

[0002] As one of the core technologies for autonomous drone navigation, visual navigation has made significant progress in recent years in algorithm optimization and hardware integration. Mainstream solutions include feature-point-based simultaneous localization and mapping (SLAM), deep learning-based end-to-end navigation, and multi-sensor fusion. For example, the Large-Scale Direct Monocular SLAM (LSD-SLAM) algorithm achieves real-time pose estimation using a direct method and has proven its effectiveness in quadrotor drones.

[0003] However, existing technologies have significant drawbacks. Traditional visual SLAM algorithms (such as ORB-SLAM3) rely on sparse feature point matching, which can easily lead to feature tracking loss or map drift in highly dynamic motion scenarios, such as the rapid rotation and acceleration of hexacopter drones. For example, when a drone rotates at high angular velocities, the feature point extraction speed cannot keep up with the speed of motion, resulting in excessive positioning latency.

[0004] Current research on hexacopter navigation primarily focuses on path planning and control algorithms (such as model predictive control), while specialized research on visual navigation is limited. For example, some hexacopter drones use lidar for obstacle avoidance, but their visual navigation capabilities only support simple target recognition and are unable to achieve autonomous positioning. Furthermore, the decoupled position control of fully driven platforms has not yet been fully integrated with visual navigation, resulting in insufficient navigation accuracy.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0006] The main purpose of the present invention is to provide a visual navigation method, system, terminal and computer-readable storage medium for a six-rotor full-drive UAV, aiming to solve the problems in the existing technology of visual navigation technology, such as insufficient adaptability in high-dynamic motion scenes, high complexity of multi-sensor fusion, limited computing power consumption and load, and insufficient consideration of the dynamic characteristics of the six-rotor full-drive UAV.

[0007] To achieve the above object, the present invention provides a visual navigation method for a six-rotor fully-driven UAV, the visual navigation method for the six-rotor fully-driven UAV comprising the following steps:

[0008] Collecting pixel depth maps, real-time pose quaternions, and forward speeds during the flight of a six-rotor full-drive drone, performing invalid pixel filling and pixel value normalization on the pixel depth maps to obtain a preprocessed depth map sequence, and transmitting the depth map sequence, the real-time pose quaternions, and the forward speeds to an ARM processor;

[0009] The depth map sequence is input into a visual encoder. Each frame is divided into a sequence of patches of preset pixel sizes through an image embedding operation. The global obstacle spatial correlation is captured by a self-attention layer, and a multi-scale feature vector is output. The multi-scale feature vectors of multiple consecutive frames are input into a bidirectional LSTM network to model the temporal dependencies of the drone during high-speed motion and generate an environment state vector including the time dimension.

[0010] The environmental state vector, the real-time pose quaternion and the forward velocity are input into the MPC controller. With the goal of minimizing trajectory tracking error and energy consumption, the three-dimensional velocity command is optimized within a preset control cycle. Based on the full-drive dynamics model of the six-rotor UAV, the optimized three-dimensional velocity command is converted into the independent rotational speeds of the six rotors. The fuselage acceleration and attitude angle changes are matched through a nonlinear mapping function to achieve motion decoupling control.

[0011] Optionally, the visual navigation method for the six-rotor full-drive UAV, wherein the pixel depth map, real-time pose quaternion and forward velocity of the six-rotor full-drive UAV are collected during flight, and invalid pixel filling and pixel value normalization are performed on the pixel depth map to obtain a pre-processed depth map sequence, specifically comprising:

[0012] The depth camera on the six-rotor full-drive drone collects 60×90 pixel depth maps at a preset frequency, and the real-time position quaternion of the drone is obtained synchronously through the IMU. and forward velocity ;

[0013] The pixel depth map is filled with invalid pixels through the FPGA hardware module, and the pixel values ​​of the pixel depth map are normalized to the range of [0, 1] to obtain a preprocessed depth map sequence.

[0014] Optionally, the visual navigation method for the six-rotor full-drive drone, wherein the depth map sequence is input into a visual encoder, each frame of the image is divided into a patch sequence of preset pixel sizes through an image embedding operation, the global obstacle spatial correlation is captured through a self-attention layer, a multi-scale feature vector is output, and the multi-scale feature vectors of multiple consecutive frames are input into a bidirectional LSTM network to model the temporal dependency of the drone during high-speed motion and generate an environment state vector containing a time dimension, specifically including:

[0015] Input the depth map sequence into the visual encoder, divide the pixel depth map input by the depth camera into a sequence of 16×16 pixel patches through the ViT feature extraction module, calculate the global correlation between the patches through the visual encoder, and obtain global environmental features through the ViT self-attention mechanism;

[0016] For each patch sequence, the corresponding query vector, key vector, and value vector are generated in combination with the learnable weight matrix, and the raw similarity score matrix is ​​calculated:

[0017] The row-wise scores are normalized using the softmax function to obtain a weight matrix that reflects the strength of patch association.

[0018] Perform weighted summation of the value vector according to the attention weight to generate a patch feature that contains the global environment;

[0019] All patch feature sequences are aggregated through a multi-layer perceptron, and a multi-scale feature vector containing global environment features is finally output;

[0020] Connect to the bidirectional LSTM network, perform time series modeling on the multi-scale feature vectors of three consecutive frames, and output the time-dependent environment state vector :

[0021] ;

[0022] in, represents the multi-scale features of the first frame, Represents the multi-scale features of the second frame, Represents the multi-scale features of the 3rd frame.

[0023] Optionally, in the visual navigation method for the six-rotor full-drive drone, the ViT feature extraction module includes an image embedding layer, a self-attention layer and a multi-layer perceptron.

[0024] Optionally, the visual navigation method for the six-rotor fully-driven UAV, wherein the environmental state vector, the real-time pose quaternion, and the forward velocity are input into an MPC controller, and the three-dimensional velocity instruction is optimized within a preset control period with the goal of minimizing trajectory tracking error and energy consumption, specifically includes:

[0025] Construct a six-rotor UAV full-drive dynamics model. If the speed command The mapping to actual acceleration is a linear relationship:

[0026] ;

[0027] in, represents the actual acceleration, represents the target acceleration output by the MPC controller, represents the acceleration gain matrix, Indicates three-dimensional speed instruction, represents the forward speed;

[0028] Construct a cost function to minimize tracking error and energy consumption:

[0029] ;

[0030] in, represents the cost function, Indicates the number of control cycles, Represents a certain control cycle, represents the expected speed, represents the weight coefficient, Indicates the Three-dimensional speed instructions for a control cycle, Indicates the The expected speed of a control cycle, Indicates the The target acceleration output by the MPC controller in each control cycle;

[0031] In each control cycle, based on the current state ( , ) and global environmental characteristics , get the future through MPC controller The control sequence of the step { },in, Indicates the Three-dimensional speed instructions for a control cycle, Indicates the Three-dimensional speed instructions for a control cycle, Indicates the Three-dimensional speed command for a control cycle.

[0032] Optionally, the visual navigation method for the six-rotor full-drive UAV, wherein, based on the six-rotor full-drive UAV dynamics model, the optimized three-dimensional velocity command is converted into the independent rotational speeds of the six rotors, and the fuselage acceleration and attitude angle changes are matched through a nonlinear mapping function to achieve motion decoupling control, specifically includes:

[0033] Calculate the number of rotors per hexacopter Generate thrust and counter torque:

[0034] ;

[0035] ;

[0036] in, represents the thrust coefficient, represents the torque coefficient, Indicates the The angular velocity of each rotor;

[0037] Based on the six degrees of freedom of the six-rotor UAV, a nonlinear mapping relationship between the rotor speed and the fuselage acceleration and attitude angle is established:

[0038] ;

[0039] in, Represents a nonlinear mapping relationship, represents the linear acceleration along the x-axis of the inertial coordinate system, represents the linear acceleration along the y-axis of the inertial coordinate system, represents the linear acceleration along the z-axis of the inertial coordinate system, represents the roll angular velocity, represents the pitch angular velocity, represents the yaw angular velocity;

[0040] The contribution of each rotor is combined into the generalized force on the drone:

[0041] ;

[0042] in, Represents the components of force in three directions in the body coordinate system, Represents the steady-state torque of the UAV in three directions, represents the moment of inertia, Indicates the quality of the drone;

[0043] The rotor speed is related to the generalized control force through the structure of the six-rotor drone:

[0044] ;

[0045] in, represents the relationship between rotor speed and generalized control force, represents the allocation matrix, represents the rotational angular velocity of the first rotor, represents the rotational angular velocity of the second rotor, represents the rotational angular velocity of the third rotor, represents the rotational angular velocity of the fourth rotor, represents the rotational angular velocity of the fifth rotor, Indicates the rotational angular velocity of the sixth rotor;

[0046] The three-dimensional speed command Convert to expected acceleration:

[0047] ;

[0048] in, represents the expected acceleration, represents the control period, Represents the gravitational acceleration converted to the body coordinate system;

[0049] The required generalized control forces are calculated using the inverse dynamics model:

[0050] ;

[0051] Use the pseudo-inverse of the allocation matrix to solve the square speed of each rotor:

[0052] ;

[0053] The actual speed is obtained by taking the square root:

[0054] .

[0055] Optionally, in the visual navigation method for the six-rotor full-drive UAV, the ARM processor is used to perform dynamic visual feature extraction and time series modeling operations.

[0056] In addition, to achieve the above-mentioned purpose, the present invention further provides a visual navigation system for a six-rotor fully-driven UAV, wherein the visual navigation system for the six-rotor fully-driven UAV comprises:

[0057] A data acquisition and preprocessing module is used to collect pixel depth maps, real-time pose quaternions, and forward speeds during the flight of the six-rotor full-drive UAV, perform invalid pixel filling and pixel value normalization on the pixel depth maps to obtain a preprocessed depth map sequence, and transmit the depth map sequence, the real-time pose quaternions, and the forward speed to the ARM processor;

[0058] A dynamic visual feature extraction and temporal modeling module is used to input the depth map sequence into a visual encoder, divide each frame of the image into a patch sequence of preset pixel sizes through an image embedding operation, capture the global obstacle spatial correlation through a self-attention layer, output a multi-scale feature vector, input the multi-scale feature vector of multiple consecutive frames into a bidirectional LSTM network, model the temporal dependency of the drone during high-speed motion, and generate an environment state vector including the time dimension;

[0059] The navigation control coordination and instruction generation module is used to input the environmental state vector, the real-time pose quaternion and the forward velocity into the MPC controller, optimize the three-dimensional velocity instruction within a preset control cycle with the goal of minimizing trajectory tracking error and energy consumption, and convert the optimized three-dimensional velocity instruction into the independent speeds of the six rotors based on the full-drive dynamics model of the six-rotor UAV. The nonlinear mapping function is used to match the fuselage acceleration and attitude angle changes to achieve motion decoupling control.

[0060] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a visual navigation program for a six-rotor full-drive drone stored in the memory and runnable on the processor. When the visual navigation program for the six-rotor full-drive drone is executed by the processor, the steps of the visual navigation method for the six-rotor full-drive drone as described above are implemented.

[0061] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a visual navigation program for a six-rotor full-drive drone, and when the visual navigation program for the six-rotor full-drive drone is executed by a processor, the steps of the visual navigation method for the six-rotor full-drive drone as described above are implemented.

[0062] In the present invention, pixel depth maps, real-time pose quaternions and forward speeds are collected during the flight of a six-rotor full-drive UAV, invalid pixel filling processing and pixel value normalization processing are performed on the pixel depth maps to obtain a pre-processed depth map sequence, and the depth map sequence, the real-time pose quaternion and the forward speed are transmitted to an ARM processor; the depth map sequence is input into a visual encoder, and each frame of the image is divided into a patch sequence of pixels of a preset size through an image embedding operation. The global obstacle spatial correlation is captured through a self-attention layer, and a multi-scale feature vector is output. The multi-scale feature vectors of multiple consecutive frames are input into a bidirectional LSTM The network models the temporal dependencies of a drone during high-speed motion and generates an environmental state vector containing a time dimension. The environmental state vector, the real-time pose quaternion, and the forward velocity are input into an MPC controller. The controller optimizes the three-dimensional velocity command within a preset control cycle with the goal of minimizing trajectory tracking error and energy consumption. Based on the full-drive dynamics model of the six-rotor drone, the optimized three-dimensional velocity command is converted into the independent rotation speeds of the six rotors. The fuselage acceleration and attitude angle changes are matched via a nonlinear mapping function to achieve motion decoupling control. This invention constructs a navigation framework that integrates dynamic visual perception and full-drive control characteristics. It uses a visual encoder to extract multi-scale environmental features and combines it with a long-short-term memory network to process temporal information. This allows for efficient feature extraction and motion trajectory prediction of image sequences under high-speed rotation and drastic attitude changes of the drone. It uses an attention mechanism to enhance the global correlation modeling of environmental features and combines it with dynamic constraints to generate control commands, significantly improving the navigation accuracy and robustness of the drone in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flow chart of a preferred embodiment of the visual navigation method for a six-rotor full-drive UAV of the present invention;

[0064] Figure 2 This is a block diagram of the principle of the visual perception module in a preferred embodiment of the visual navigation method for a six-rotor full-drive UAV of the present invention;

[0065] Figure 3 1 is a structural diagram of a preferred embodiment of the visual navigation system of a six-rotor full-drive UAV of the present invention;

[0066] Figure 4 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0068] In response to the problems of insufficient adaptability of existing visual navigation technology in high-dynamic motion scenes, high complexity of multi-sensor fusion, limited computing power consumption and load, and insufficient consideration of the dynamic characteristics of six-rotor full-drive UAVs, the present invention proposes an end-to-end visual navigation method based on visual Transformer. By constructing a navigation framework that integrates dynamic visual perception and full-drive control characteristics, using visual Transformer to extract multi-scale environmental features, and combining long short-term memory network (LSTM) to process time series information, efficient feature extraction and motion trajectory prediction of image sequences under high-speed rotation and drastic posture changes of UAVs can be achieved, solving the problem of feature tracking loss of traditional algorithms in high-dynamic scenes.

[0069] This invention strengthens the global correlation modeling of environmental features through the attention mechanism, and combines it with the control instruction generation of dynamic constraints, which significantly improves the navigation accuracy and robustness of drones in complex dynamic environments. It does not need to rely on high-precision inertial navigation compensation, and provides an efficient and reliable navigation solution for the application of six-rotor full-drive drones in scenarios such as emergency rescue and high-speed inspection.

[0070] The visual navigation method of the six-rotor full-drive UAV described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the visual navigation method of the six-rotor full-drive UAV includes the following steps:

[0071] Step S10: Collect pixel depth maps, real-time pose quaternions, and forward speeds during the flight of the six-rotor full-drive drone, perform invalid pixel filling and pixel value normalization on the pixel depth maps to obtain a preprocessed depth map sequence, and transmit the depth map sequence, the real-time pose quaternions, and the forward speed to an ARM processor.

[0072] Specifically, if Figure 2 As shown in the figure, the depth camera carried by the six-rotor full-drive drone collects a 60×90 pixel depth map from the start to the entire flight process at a preset frequency (for example, 30Hz), and the real-time position quaternion of the drone is obtained synchronously through the IMU. (Pose represents the position and posture of an object in three-dimensional space, where the position is the coordinate of the object in three-dimensional space, expressed as a triplet ( ) indicates that the attitude is the current rotation direction of the object, expressed as a quaternion ( ) indicates that That is, the quadruple representing the current drone posture in the posture information) and the forward speed ; Then, the pixel depth map is filled with invalid pixels through the FPGA hardware module (each pixel value of the pixel depth map represents the distance between the corresponding point in the scene and the camera. Due to sensor noise, occlusion, out of measurement range, etc., there may be invalid pixels in the pixel depth map, and these areas need to be filled), and the pixel values ​​of the pixel depth map are normalized to the range of [0, 1] (the pixel values ​​of the pixel depth map are normalized from the original range (in is the maximum measurement distance of the sensor) is linearly scaled to [0, 1] so that all pixel values ​​fall between 0 and 1), and the preprocessed depth map sequence is obtained. Then, the preprocessed depth map sequence (3 consecutive frames) and the real-time pose quaternion are converted into and the forward velocity The data is transmitted to the ARM processor to facilitate the use of the computing power of the ARM processor for subsequent dynamic visual feature extraction and timing modeling operations.

[0073] Step S20: Input the depth map sequence into a visual encoder, divide each frame of the image into a patch sequence of preset pixel sizes through an image embedding operation, capture the global obstacle spatial correlation through a self-attention layer, output a multi-scale feature vector, input the multi-scale feature vector of multiple consecutive frames into a bidirectional LSTM network, model the temporal dependency of the drone during high-speed motion, and generate an environment state vector containing a time dimension.

[0074] Specifically, the depth map sequence is input into the visual encoder (visual transformer), and each frame image is divided into a 16×16 pixel patch sequence through image embedding operation. The global obstacle spatial correlation is captured by the self-attention layer, and a multi-scale feature vector is output. Then, the multi-scale feature vectors of three consecutive frames are input into the bidirectional LSTM network (Long Short-Term Memory Network) to model the temporal dependency of the drone during high-speed motion and generate an environment state vector containing the time dimension. , which solves the problem of feature loss of single-frame images in dynamic scenes.

[0075] like Figure 2 As shown in the figure, a visual encoder and LSTM fusion architecture is used to process image sequences (60×90 pixel depth maps) input from a drone's onboard depth camera. The ViT feature extraction module (ViT, Vision Transformer, visual encoder, including image embedding layer, self-attention layer, and multi-layer perceptron) divides the image into a sequence of 16×16 pixel patches, and the self-attention mechanism is used to capture global environmental features.

[0076] The implementation process of each step in the ViT feature extraction module is as follows:

[0077] The three frames of pixel depth maps are concatenated into a 3×60×90 tensor. Each depth map frame is divided into 15 non-overlapping patches of 16×16 pixels, for a total of 45 patches across the three frames. Each patch is flattened into a 256-dimensional vector. The visual encoder then calculates the global correlation between the patches, and the ViT self-attention mechanism is used to obtain global environmental features. The principle of the self-attention mechanism is described as follows:

[0078] ;

[0079] in, represents the query vector, represents the key vector, represents a value vector, represents the dimension of the key vector, Indicates transpose.

[0080] A. For each 256-dimensional patch vector, three learnable weight matrices are used. Multiply to generate the corresponding query vector , key vector Sum value vector , these vectors are used to subsequently calculate the attention scores.

[0081] B. For all and ( j = 1, 2, ..., 45) to calculate the dot product and then divide by the square root scaling factor of the vector dimension to ensure numerical stability (Right now ), and get the original similarity score matrix:

[0082] ;

[0083] C. Utilization The function normalizes the row-wise scores to obtain a weight matrix reflecting the patch association strength. :

[0084] ;

[0085] in, represents the number of key vectors, Indicates the key vectors;

[0086] D. Perform weighted summation of the value vector according to the attention weight to generate patch features containing the global environment :

[0087] ;

[0088] in, Indicates the j A vector of values;

[0089] E. Finally, all patch feature sequences are aggregated through a multi-layer perceptron, and the final output is a visual feature vector containing global environment features.

[0090] Connect to the bidirectional LSTM network, perform time series modeling on the feature sequence of three consecutive frames, and output the environment state vector containing time dependence ,in, represents the multi-scale features of the first frame, Represents the multi-scale features of the second frame, Represents the multi-scale features of the 3rd frame.

[0091] Step S30: Input the environmental state vector, the real-time pose quaternion, and the forward velocity into the MPC controller. With the goal of minimizing trajectory tracking error and energy consumption, optimize the three-dimensional velocity command within a preset control period. Based on the full-drive dynamics model of the six-rotor UAV, convert the optimized three-dimensional velocity command into the independent rotational speeds of the six rotors. Match the fuselage acceleration and attitude angle changes through a nonlinear mapping function to achieve motion decoupling control.

[0092] Specifically, the environment state vector output by the visual perception module is Compared with real-time pose ( , ) is fed into an MPC controller (Model Predictive Control, a control system embedded in the drone’s flight control computer in the form of an algorithm) to optimize the three-dimensional velocity command within a 50ms control cycle with the goal of minimizing trajectory tracking error and energy consumption. ∈ Then, based on the six-rotor UAV full-drive dynamics model, the three-dimensional speed command is converted into the independent speed of the six rotors. , through the nonlinear mapping function Match the body acceleration and attitude angle changes to achieve motion decoupling control.

[0093] The optimization principle is as follows:

[0094] A. Construct a six-rotor UAV full-drive dynamics model. If the speed command The mapping to actual acceleration is a linear relationship:

[0095] ;

[0096] in, Indicates the actual acceleration (speed derivative with respect to time), represents the target acceleration output by the MPC controller, Represents the acceleration gain matrix, which is used to adjust the tracking response speed. Indicates three-dimensional speed instruction, that is, in the spatial coordinate system Speed ​​in three dimensions, Represents the forward speed (the goal is to optimize this instruction sequence).

[0097] B. Construct a cost function to minimize tracking error and energy consumption:

[0098] ;

[0099] in, represents the cost function, Indicates the number of control cycles, Represents a certain control cycle, represents the expected speed, represents the weight coefficient, Indicates the Three-dimensional speed instructions for a control cycle, Indicates the The expected speed of a control cycle, Indicates the The target acceleration output by the MPC controller during each control cycle.

[0100] C. In each control cycle, based on the current state ( , ) and global environmental characteristics , get the future through MPC controller The control sequence of the step { },in, Indicates the Three-dimensional speed instructions for a control cycle, Indicates the Three-dimensional speed instructions for a control cycle, Indicates the Three-dimensional speed command for a control cycle.

[0101] D. Only execute the speed instruction of the first cycle , and the process repeats in the next cycle.

[0102] The speed command is then converted into independent speeds of the six rotors through the full-drive thrust distribution algorithm. ,The transformation derivation process is as follows:

[0103] A. Build a dynamic model. The six-rotor UAV is a fully driven system. Calculate the dynamics of each rotor of the six-rotor UAV. Generate thrust and counter torque:

[0104] ;

[0105] ;

[0106] in, represents the thrust coefficient, represents the torque coefficient, Indicates the The angular velocity of the rotor.

[0107] Based on the six degrees of freedom of the six-rotor UAV, a nonlinear mapping relationship between the rotor speed and the fuselage acceleration and attitude angle is established:

[0108] ;

[0109] in, Represents a nonlinear mapping relationship, ( i =1, 2, ..., 6) means the The angular velocity of the rotor, It represents the linear acceleration in the x-axis direction of the inertial coordinate system, and represents the accelerated motion of the fuselage along the horizontal lateral direction. It represents the linear acceleration in the y-axis direction of the inertial coordinate system, and represents the accelerated motion of the fuselage along the horizontal longitudinal direction. It represents the linear acceleration in the z-axis direction of the inertial coordinate system, and represents the accelerated motion of the fuselage in the vertical direction. represents the roll angular velocity, that is, the rotation angular velocity of the fuselage around the x-axis, represents the pitch angular velocity, that is, the rotation angular velocity of the fuselage around the y-axis, It represents the yaw angular velocity, that is, the rotational angular velocity of the fuselage around the z-axis.

[0110] B. Combine the contributions of each rotor into the generalized force on the drone:

[0111] ;

[0112] in, Represents the components of force in three directions in the body coordinate system, Represents the steady-state torque of the drone in three directions (ignoring angular acceleration, suitable for fast thrust distribution), represents the moment of inertia, Indicates the quality of the drone.

[0113] C. Correlate rotor speed with generalized control force through the structure of the hexacopter:

[0114] ;

[0115] in, represents the relationship between rotor speed and generalized control force, represents the allocation matrix, represents the rotational angular velocity of the first rotor, represents the rotational angular velocity of the second rotor, represents the rotational angular velocity of the third rotor, represents the rotational angular velocity of the fourth rotor, represents the rotational angular velocity of the fifth rotor, Indicates the rotational angular velocity of the sixth rotor.

[0116] D. The three-dimensional speed instruction Convert to expected acceleration:

[0117] ;

[0118] in, represents the expected acceleration, represents the control period, Indicates the gravitational acceleration converted to the body coordinate system.

[0119] E. Calculate the required generalized control force through the dynamic inverse model:

[0120] ;

[0121] F. Use the pseudo-inverse of the allocation matrix to solve the square speed of each rotor:

[0122] ;

[0123] G. Get the actual speed by taking the square root:

[0124] .

[0125] This paper utilizes an ARM+FPGA heterogeneous architecture: the ARM processor runs the ViT-LSTM neural network inference, while the FPGA implements parallel acceleration of the self-attention layer and hardware-based thrust allocation algorithms. Depth map preprocessing is performed by the FPGA, and global environmental features extracted by ViT are transmitted to the ARM via a high-speed bus. The FPGA then generates final control instructions in real time.

[0126] The innovative features of the present invention are as follows:

[0127] (1) End-to-end vision-control collaborative architecture (different from traditional modular navigation): For the first time, the visual Transformer is combined with the long short-term memory network to build a dynamic visual perception module, which directly extracts the environment state vector containing global spatial correlation and temporal motion features from the depth image sequence, replacing the traditional method that relies on sparse feature point matching or manually designed features.

[0128] (2) Model Predictive Control (MPC) for Full-Drive Dynamics Adaptation: Design an MPC closed-loop control algorithm based on the full-drive characteristics of the six-rotor propeller, directly map the posture information output by the visual navigation to the independent speed instructions of the six rotors, and use the nonlinear dynamic model to Realize motion decoupling and break the decoupling design of traditional navigation and control modules such as "positioning first and then separately planning the trajectory".

[0129] (3) Lightweight heterogeneous computing hardware architecture: Using ARM+FPGA heterogeneous computing solutions, FPGA accelerates the ViT self-attention layer and thrust allocation algorithm in parallel, while ARM handles high-level reasoning, reducing overall power consumption and weight, and solving the load and endurance bottlenecks of existing visual navigation systems.

[0130] The beneficial effects brought by the present invention are:

[0131] (1) Improved adaptability to dynamic environments: The self-attention mechanism captures the global obstacle distribution, improving the long-range feature recognition rate compared to traditional CNN methods such as ResNet-18. LSTM processes temporal information, improving the feature tracking success rate in high-speed rotation scenes and avoiding navigation failures caused by image blur. At the same time, ViT's global modeling capabilities and LSTM's temporal memory capabilities solve the problem of dynamic scene feature loss in existing technologies.

[0132] (2) Optimization of full-drive control accuracy: The pose error output by visual navigation is reduced, and end-to-end learning eliminates the cumulative error caused by the decoupling of navigation and control, thereby shortening the trajectory tracking delay of the UAV during high-speed steering. In addition, the optimization of the full-drive dynamics model and MPC directly improves the feasibility of control instructions, avoiding the problem of mismatch between control instructions and the physical characteristics of the UAV in traditional methods.

[0133] Alternative embodiments and variations of the present invention:

[0134] (1) Alternative sensor solutions: In addition to RGB-D (RGB Image and Depth Image) cameras, lidar can also be used to measure distance information. However, the cost of lidar is much higher than that of depth cameras, and it is large in size and has high power consumption, making it unsuitable for lightweight drones. On the contrary, depth cameras have a compact structure, low power consumption, and are easy to deploy in embedded systems.

[0135] (2) Network structure transformation: The ViT encoder can be replaced with a traditional convolutional neural network (CNN). However, ViT directly captures long-distance dependencies across regions through a self-attention mechanism, avoiding the limitations of CNNs that indirectly model global information through multi-layer stacking or global pooling. It is more effective for tasks that require global semantic understanding, such as image classification and long-tail distribution scenarios.

[0136] (3) Visual perception uses ORB-SLAM3 or LSD-SLAM for sparse feature point positioning, replacing the end-to-end feature extraction of ViT-LSTM; the control module uses a classic PID controller to adjust the speed of the six-rotor, and solves the thrust distribution through an independent kinematic model to avoid MPC and dynamic mapping functions. Joint design.

[0137] Furthermore, if Figure 3 As shown, based on the above-mentioned visual navigation method of the six-rotor full-drive UAV, the present invention also provides a visual navigation system of the six-rotor full-drive UAV, wherein the visual navigation system of the six-rotor full-drive UAV includes:

[0138] The data acquisition and preprocessing module 51 is used to collect pixel depth maps, real-time pose quaternions, and forward speeds during the flight of the six-rotor full-drive UAV, perform invalid pixel filling and pixel value normalization on the pixel depth maps to obtain a preprocessed depth map sequence, and transmit the depth map sequence, the real-time pose quaternions, and the forward speeds to the ARM processor;

[0139] The dynamic visual feature extraction and temporal modeling module 52 is configured to input the depth map sequence into a visual encoder, divide each frame of the image into a sequence of patches of preset pixel sizes through an image embedding operation, capture global obstacle spatial correlations through a self-attention layer, output a multi-scale feature vector, input the multi-scale feature vectors of multiple consecutive frames into a bidirectional LSTM network, model the temporal dependencies of the drone during high-speed motion, and generate an environmental state vector including a time dimension;

[0140] The navigation control coordination and instruction generation module 53 is used to input the environmental state vector, the real-time pose quaternion and the forward velocity into the MPC controller, with the goal of minimizing trajectory tracking error and energy consumption, optimize the three-dimensional velocity instruction within a preset control cycle, and convert the optimized three-dimensional velocity instruction into the independent speed of the six rotors based on the full-drive dynamics model of the six-rotor UAV, and match the fuselage acceleration and attitude angle changes through a nonlinear mapping function to achieve motion decoupling control.

[0141] Furthermore, if Figure 4 As shown, based on the visual navigation method and system of the above-mentioned six-rotor full-drive UAV, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 4 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0142] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard drive or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card, etc. equipped on the terminal. Furthermore, the memory 20 may include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software installed on the terminal and various types of data, such as program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is about to be output. In one embodiment, the memory 20 stores a visual navigation program 40 for a six-rotor full-drive drone. The visual navigation program 40 for the six-rotor full-drive drone can be executed by the processor 10, thereby implementing the visual navigation method for the six-rotor full-drive drone in this application.

[0143] In some embodiments, the processor 10 can be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the visual navigation method of the six-rotor full-drive drone.

[0144] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The processor 10, memory 20, and display 30 of the terminal communicate with each other via a system bus.

[0145] In one embodiment, when the processor 10 executes the visual navigation program 40 for the six-rotor full-drive UAV in the memory 20 , the steps of visual navigation of the six-rotor full-drive UAV described above are implemented.

[0146] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a visual navigation program for a six-rotor full-drive UAV, and when the visual navigation program for the six-rotor full-drive UAV is executed by a processor, the steps of the visual navigation method for the six-rotor full-drive UAV as described above are implemented.

[0147] In summary, the present invention provides a visual navigation method, system, terminal and storage medium for a six-rotor full-drive UAV. The method comprises: collecting a pixel depth map, a real-time pose quaternion and a forward velocity during the flight of the six-rotor full-drive UAV, performing invalid pixel filling processing and pixel value normalization processing on the pixel depth map to obtain a preprocessed depth map sequence, transmitting the depth map sequence, the real-time pose quaternion and the forward velocity to an ARM processor; inputting the depth map sequence into a visual encoder, dividing each frame of the image into a patch sequence of pixels of a preset size through an image embedding operation, capturing the global obstacle spatial correlation through a self-attention layer, outputting a multi-scale feature vector, and inputting the multi-scale feature vectors of multiple consecutive frames into a bidirectional LSTM The network models the temporal dependencies of a drone during high-speed motion and generates an environmental state vector containing a time dimension. The environmental state vector, the real-time pose quaternion, and the forward velocity are input into an MPC controller. The controller optimizes the three-dimensional velocity command within a preset control cycle with the goal of minimizing trajectory tracking error and energy consumption. Based on the full-drive dynamics model of the six-rotor drone, the optimized three-dimensional velocity command is converted into the independent rotation speeds of the six rotors. The fuselage acceleration and attitude angle changes are matched via a nonlinear mapping function to achieve motion decoupling control. This invention constructs a navigation framework that integrates dynamic visual perception and full-drive control characteristics. It uses a visual encoder to extract multi-scale environmental features and combines it with a long-short-term memory network to process temporal information. This allows for efficient feature extraction and motion trajectory prediction of image sequences under high-speed rotation and drastic attitude changes of the drone. It uses an attention mechanism to enhance the global correlation modeling of environmental features and combines it with dynamic constraints to generate control commands, significantly improving the navigation accuracy and robustness of the drone in complex dynamic environments.

[0148] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0149] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When executed, the program can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0150] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A visual navigation method for a six-rotor full-drive UAV, characterized in that: The visual navigation method of the six-rotor full-drive UAV includes: Collecting pixel depth maps, real-time pose quaternions, and forward speeds during the flight of a six-rotor full-drive drone, performing invalid pixel filling and pixel value normalization on the pixel depth maps to obtain a preprocessed depth map sequence, and transmitting the depth map sequence, the real-time pose quaternions, and the forward speeds to an ARM processor; The depth map sequence is input into a visual encoder. Each frame is divided into a sequence of patches of preset pixel sizes through an image embedding operation. The global obstacle spatial correlation is captured by a self-attention layer, and a multi-scale feature vector is output. The multi-scale feature vectors of multiple consecutive frames are input into a bidirectional LSTM network to model the temporal dependencies of the drone during high-speed motion and generate an environment state vector including the time dimension. The environmental state vector, the real-time pose quaternion and the forward velocity are input into the MPC controller. With the goal of minimizing trajectory tracking error and energy consumption, the three-dimensional velocity command is optimized within a preset control cycle. Based on the full-drive dynamics model of the six-rotor UAV, the optimized three-dimensional velocity command is converted into the independent rotational speeds of the six rotors. The fuselage acceleration and attitude angle changes are matched through a nonlinear mapping function to achieve motion decoupling control.

2. The visual navigation method for a six-rotor full-drive UAV according to claim 1, characterized in that: The method collects pixel depth maps, real-time pose quaternions, and forward speeds during the flight of the six-rotor full-drive drone, performs invalid pixel filling processing and pixel value normalization processing on the pixel depth maps, and obtains a pre-processed depth map sequence, specifically including: The depth camera on the six-rotor full-drive drone collects 60×90 pixel depth maps at a preset frequency, and the real-time position quaternion of the drone is obtained synchronously through the IMU. and forward velocity ; The pixel depth map is filled with invalid pixels through the FPGA hardware module, and the pixel values ​​of the pixel depth map are normalized to the range of [0, 1] to obtain a preprocessed depth map sequence.

3. The visual navigation method for a six-rotor full-drive UAV according to claim 2, characterized in that: The depth map sequence is input into the visual encoder, and each frame of the image is divided into a patch sequence of preset pixel sizes through an image embedding operation. The global obstacle spatial correlation is captured by the self-attention layer, and a multi-scale feature vector is output. The multi-scale feature vectors of multiple consecutive frames are input into a bidirectional LSTM network to model the temporal dependency of the drone during high-speed motion and generate an environment state vector containing a time dimension. Specifically, the following steps are performed: Input the depth map sequence into the visual encoder, divide the pixel depth map input by the depth camera into a sequence of 16×16 pixel patches through the ViT feature extraction module, calculate the global correlation between the patches through the visual encoder, and obtain global environmental features through the ViT self-attention mechanism; For each patch sequence, the corresponding query vector, key vector, and value vector are generated in combination with the learnable weight matrix, and the raw similarity score matrix is ​​calculated: The row-wise scores are normalized using the softmax function to obtain a weight matrix that reflects the strength of patch association. Perform weighted summation of the value vector according to the attention weight to generate a patch feature that contains the global environment; All patch feature sequences are aggregated through a multi-layer perceptron, and a multi-scale feature vector containing global environment features is finally output; Connect to the bidirectional LSTM network, perform time series modeling on the multi-scale feature vectors of three consecutive frames, and output the time-dependent environment state vector : ； in, represents the multi-scale features of the first frame, Represents the multi-scale features of the second frame, Represents the multi-scale features of the 3rd frame.

4. The visual navigation method for a six-rotor full-drive UAV according to claim 3, characterized in that: The ViT feature extraction module includes an image embedding layer, a self-attention layer, and a multi-layer perceptron.

5. The visual navigation method for a six-rotor full-drive UAV according to claim 3, characterized in that: The step of inputting the environmental state vector, the real-time pose quaternion, and the forward velocity into an MPC controller, optimizing the three-dimensional velocity command within a preset control period with the goal of minimizing trajectory tracking error and energy consumption, specifically includes: Construct a six-rotor UAV full-drive dynamics model. If the speed command The mapping to actual acceleration is a linear relationship: ； in, represents the actual acceleration, represents the target acceleration output by the MPC controller, represents the acceleration gain matrix, Indicates three-dimensional speed instruction, represents the forward speed; Construct a cost function to minimize tracking error and energy consumption: ； in, represents the cost function, Indicates the number of control cycles, Represents a certain control cycle, represents the expected speed, represents the weight coefficient, Indicates the Three-dimensional speed instructions for a control cycle, Indicates the The expected speed of a control cycle, Indicates the The target acceleration output by the MPC controller in each control cycle; In each control cycle, based on the current state ( , ) and global environmental characteristics , get the future through MPC controller The control sequence of the step { },in, Indicates the Three-dimensional speed instructions for a control cycle, Indicates the Three-dimensional speed instructions for a control cycle, Indicates the Three-dimensional speed command for a control cycle.

6. The visual navigation method for a six-rotor full-drive UAV according to claim 5, characterized in that: Based on the full-drive dynamics model of the six-rotor UAV, the optimized three-dimensional speed command is converted into the independent speeds of the six rotors, and the fuselage acceleration and attitude angle changes are matched through a nonlinear mapping function to achieve motion decoupling control, specifically including: Calculate the number of rotors per hexacopter Generate thrust and counter torque: ； ； in, represents the thrust coefficient, represents the torque coefficient, Indicates the The angular velocity of each rotor; Based on the six degrees of freedom of the six-rotor UAV, a nonlinear mapping relationship between the rotor speed and the fuselage acceleration and attitude angle is established: ； in, Represents a nonlinear mapping relationship, represents the linear acceleration along the x-axis of the inertial coordinate system, represents the linear acceleration along the y-axis of the inertial coordinate system, represents the linear acceleration along the z-axis of the inertial coordinate system, represents the roll angular velocity, represents the pitch angular velocity, represents the yaw angular velocity; The contribution of each rotor is combined into the generalized force on the drone: ； in, Represents the components of force in three directions in the body coordinate system, Represents the steady-state torque of the UAV in three directions, represents the moment of inertia, Indicates the quality of the drone; The rotor speed is related to the generalized control force through the structure of the six-rotor drone: ； in, represents the relationship between rotor speed and generalized control force, represents the allocation matrix, represents the rotational angular velocity of the first rotor, represents the rotational angular velocity of the second rotor, represents the rotational angular velocity of the third rotor, represents the rotational angular velocity of the fourth rotor, represents the rotational angular velocity of the fifth rotor, Indicates the rotational angular velocity of the sixth rotor; The three-dimensional speed command Convert to expected acceleration: ； in, represents the expected acceleration, represents the control period, Represents the gravitational acceleration converted to the body coordinate system; The required generalized control forces are calculated using the inverse dynamics model: ； Use the pseudo-inverse of the allocation matrix to solve the square speed of each rotor: ； The actual speed is obtained by taking the square root: 。 7. The visual navigation method for a six-rotor full-drive UAV according to claim 1, characterized in that: The ARM processor is used to perform dynamic visual feature extraction and time series modeling operations.

8. A visual navigation system for a six-rotor fully-driven UAV, characterized in that: The visual navigation system of the six-rotor full-drive UAV includes: A data acquisition and preprocessing module is used to collect pixel depth maps, real-time pose quaternions, and forward speeds during the flight of the six-rotor full-drive UAV, perform invalid pixel filling and pixel value normalization on the pixel depth maps to obtain a preprocessed depth map sequence, and transmit the depth map sequence, the real-time pose quaternions, and the forward speed to the ARM processor; A dynamic visual feature extraction and temporal modeling module is used to input the depth map sequence into a visual encoder, divide each frame of the image into a patch sequence of preset pixel sizes through an image embedding operation, capture the global spatial correlation of obstacles through a self-attention layer, output a multi-scale feature vector, and input the multi-scale feature vector of multiple consecutive frames into a bidirectional LSTM network to model the temporal dependencies of the drone during high-speed motion and generate an environmental state vector including a time dimension; The navigation control coordination and instruction generation module is used to input the environmental state vector, the real-time pose quaternion and the forward velocity into the MPC controller, optimize the three-dimensional velocity instruction within a preset control cycle with the goal of minimizing trajectory tracking error and energy consumption, and convert the optimized three-dimensional velocity instruction into the independent speeds of the six rotors based on the full-drive dynamics model of the six-rotor UAV. The nonlinear mapping function is used to match the fuselage acceleration and attitude angle changes to achieve motion decoupling control.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a visual navigation program for a six-rotor full-drive UAV stored in the memory and runnable on the processor. When the visual navigation program for the six-rotor full-drive UAV is executed by the processor, the steps of the visual navigation method for the six-rotor full-drive UAV as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a visual navigation program for a six-rotor full-drive drone. When the visual navigation program for the six-rotor full-drive drone is executed by a processor, the steps of the visual navigation method for the six-rotor full-drive drone as described in any one of claims 1 to 7 are implemented.

Images