Unmanned aerial vehicle dynamic platform autonomous landing method based on nmpc

Abstract

The application provides an unmanned aerial vehicle dynamic platform autonomous landing method based on multi-sensor fusion of NMPC. The position information obtained by the unmanned aerial vehicle in combination with an RTK-GPS and image information obtained by a camera are used, special identification codes laid in the unmanned ship landing area are used for cooperative planning and navigation, UKF is used to predict the dynamic platform position, and on the basis of the cooperative accurate positioning of the two platforms, an anti-interference NMPC algorithm is used to optimize the unmanned aerial vehicle tracking trajectory to realize autonomous landing control of the unmanned ship carrying unmanned aerial vehicle. Simulation experiment and actual experiment results show that the algorithm and the autonomous landing control method provided in the paper are feasible and effective, and an effective solution is provided for autonomous landing of a quadrotor aircraft on any dynamic platform.

Description

Technical Field

[0001] This invention belongs to the field of autonomous collaborative control of mother-daughter heterogeneous systems based on UAVs / ships / vehicles, and involves the design of a new method for autonomous landing of UAV mobile platforms. Background Technology

[0002] Unmanned Aerial Vehicles (UAVs) have become one of the most eye-catching "keywords" in the technology field over the past decade. Among them, heterogeneous mother-daughter collaborative robot systems, represented by unmanned vehicles and UAVs, are one of the important means for future geographical surveys, disaster relief, and military operations. Due to the inherent advantages of UAVs, such as low cost, high maneuverability, and strong stealth capabilities, many countries have conducted various researches based on UAV platforms, especially quadcopter UAVs, which have seen extremely widespread application. In these mission scenarios, the heterogeneous collaboration between UAVs and unmanned vehicles / unmanned surface vessels can compensate for the deficiencies between different types of equipment, combine the perception capabilities of multiple devices, expand the reconnaissance range, and improve the overall efficiency of the equipment. In these heterogeneous collaborative missions, the entire process from takeoff to operation and landing of UAVs places higher demands on the degree of autonomy of UAVs. Achieving autonomous landing of UAVs in complex environments involves research fields such as computer vision, communication transmission, and UAV control. Among these, effective estimation of the pose information of each subsystem in the heterogeneous system, rapid and accurate identification of markers through computer vision, and position control during the UAV landing process are the three most important parts of the landing process. Summary of the Invention

[0003] This invention addresses the problems in existing landing systems where UAVs are easily limited by distance and environmental interference when tracking and autonomously landing on a mobile platform. In environments lacking point features, the pose estimation accuracy relying solely on visual odometry is insufficient. Furthermore, traditional linear path planning solvers and controllers cannot meet the rapid and safe requirements under the nonlinear and strongly coupled characteristics of cooperative landing systems. This invention provides a new method for autonomous landing of UAVs on a mobile platform.

[0004] An autonomous landing method for a drone platform based on NMPC multi-sensor fusion includes: Positioning based on RTK-GPS and image information fusion: precise positioning of each subsystem independently based on RTK-GPS at long distances outside the field of view, and pose calculation by combining visual information near the landing end; Jerk smooth safe landing trajectory planning based on kinematic equations: The kinematic model of the landing platform is established, and the pose estimation and prediction of the unmanned vehicle are based on UKF. At the same time, in order to ensure the real-time performance of the system, unlike the traditional UAV-based calculation method, the onboard part of the unmanned vehicle is used as the computing unit to realize the minimum Jerk trajectory landing trajectory planning based on multi-sensor data fusion. Anti-interference trajectory optimization prediction and tracking based on improved NMPC: UKF is used to compensate for the influence of crosswinds and predict the position of the UAV in real time, so that the UAV can autonomously calculate the optimal tracking trajectory path based on the acquired trajectory information, and achieve the most accurate trajectory tracking. A host computer system that autonomously switches between different flight states and control logics throughout the entire drone landing process.

[0005] In some disclosures, the positioning based on RTK-GPS and image information fusion includes precise positioning of each subsystem independently based on RTK-GPS at long distances outside the field of view, and pose calculation by combining visual information at the near landing end; at long distances, dual-end synchronized RTK-GPS information is used as the source of UAV position estimation information and target point position information, and the detected identification information is fused with existing information during the process of the UAV moving platform entering the UAV's line of sight until landing; When the identification information is complete, the combined identification code information based on Aruco is directly used as the data input; the weighted fusion is performed using the following formula: in, The final relative pose matrix in the x-direction is used to combine the identifiers. To determine whether the location identifier has been identified, a flag matrix is ​​used. This is a matrix showing the coordinate relationships between each identifier in the combined identifier and its center point. in, To obtain the relative distance in the x-direction, This represents the actual number of identifiers captured. Define the drone's own coordinates as The coordinates obtained through vision are The fused UAV world coordinates were obtained by using a weighted average method. as follows, in .

[0006] When the identification code is detected to be distorted, the space of the circle of the identification is used as the region of interest (ROI) for subsequent image calculation. The distorted image information is repaired based on the current UAV attitude and the extracted information within the ROI in order to estimate the pose information of the target point.

[0007] In some publications, the Jerk smooth safe landing trajectory planning based on kinematic equations includes establishing a kinematic model of the landing platform, and estimating and predicting the pose of the unmanned vehicle based on UKF. At the same time, to ensure the real-time performance of the system, unlike the traditional UAV-based onboard solution method, the onboard part of the unmanned vehicle is used as the computing unit to realize the minimum Jerk trajectory landing trajectory planning based on multi-sensor data fusion.

[0008] In some disclosures, the feature is that the anti-interference trajectory optimization prediction tracking based on improved NMPC: the NMPC control method based on a nonlinear UAV model minimizes the cost function considering both control and reference trajectories, under the premise of considering hard input constraints and soft state constraints. The constraints are satisfied: By utilizing UKF to compensate for the effects of crosswinds and predict the position of the UAV in real time, the UAV can autonomously calculate the optimal tracking trajectory path based on the acquired trajectory information, thereby achieving the most accurate trajectory tracking. It also provides good assurance of the real-time performance of the calculation, the influence of external interference such as wind, and the attitude maintenance under the influence of other UAVs and other dynamic obstacles.

[0009] In some publicly available descriptions, the host computer system that autonomously switches between different flight states and control logic throughout the entire UAV landing process uses a Jetson Nano onboard computer for logic reasoning and calculation, communicates with the flight controller via MAVLINK2, and interacts with subsystems via LoRa pass-through. On the software side, the ROS system is used to implement one-click takeoff and landing, target tracking, and autonomous landing. Furthermore, submodules of each system are built in Gazebo for joint simulation, and the simulation results are ported to the hardware system to verify the actual working effect and landing accuracy.

[0010] The beneficial effects of this invention are: By employing a fusion positioning method and an improved preprocessing Hough gradient algorithm, positioning accuracy is enhanced. The anti-interference NMPC algorithm is used to optimize the UAV tracking trajectory to achieve precise control of the autonomous landing of the unmanned vehicle-mounted UAV. Attached Figure Description

[0011] The invention will now be further described with reference to the accompanying drawings.

[0012] Figure 1 This is a flowchart of the application; Figure 2 The combined logo design drawing for this application Figure 3 The image shows the physical hardware of the drone used in this application. The components corresponding to each number in the diagram are as follows: Detailed Implementation The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The main innovative points of this invention are as follows: 1) A method for UAV pose estimation and navigation based on multi-sensor fusion.

[0014] This paper proposes a method for calculating the current pose of a UAV and the relative pose between the UAV and the landing platform using data fusion of RTK-GPS and image information, while incorporating data from UAV-borne sensors for compensation and correction. Regarding image information calculation, a target region extraction method based on OpenCV with contour preprocessing is proposed, along with a novel cooperative label design based on Aruco.

[0015] 2) Based on the predicted values ​​of heterogeneous data, a control method for NMPC that resists crosswind interference is proposed.

[0016] Under conditions of crosswind interference, the UKF (Uniform Kinematics Function) is used to compensate for the crosswind effect and predict the UAV's position in real time. The predicted positions of the UAV and unmanned vehicle, along with obstacle information, are used to generate the UAV's landing trajectory in real time using the Jerk optimal trajectory planning method. The NMPC control method based on a nonlinear UAV model minimizes the cost function considering both control and reference trajectories, while taking into account hard input constraints and soft state constraints, to achieve the most accurate trajectory tracking. The flowchart of this invention is as follows: 1) To address the insufficient positioning performance of a single visual navigation mode in environments with limited feature coverage, a positioning method based on RTK-GPS and image information fusion is proposed. This method mainly includes independent, precise RTK-GPS-based positioning of each subsystem at long distances outside the field of view, and pose fusion calculation combining visual information near the landing end. At long distances, dual-end synchronized RTK-GPS information is used as the source of UAV position estimation and target point location information. During the process from the UAV moving platform entering line-of-sight until landing, detected identification information is fused with existing information. When the identification information is complete, the combined identification code information based on Aruco is directly used as the data input. Weighted fusion is performed using the following formula: in, The final relative pose matrix in the x-direction is used to combine the identifiers. To determine whether the location identifier has been identified, a flag matrix is ​​used. This is a matrix showing the coordinate relationships between each identifier in the combined identifier and its center point.

[0017] in, To obtain the relative distance in the x-direction, This represents the actual number of identifiers captured.

[0018] Define the drone's own coordinates as The coordinates obtained through vision are The fused UAV world coordinates were obtained by using a weighted average method. as follows, in .

[0019] When the identification code is distorted, after detecting the ring of the identification code, the space of the ring is used as the region of interest (ROI) for subsequent image calculation. Based on the current UAV attitude and the extracted information within the ROI, the distorted image information is repaired to estimate the target point pose information. The designed identification is as follows: Figure 2 Combination logo design 2) To address the low landing safety issue of traditional path planning control methods under non-uniform linear motion modes of the landing platform, a Jerk smooth safe landing trajectory planning method based on kinematic equations is proposed. The main steps include establishing the kinematic model of the landing platform and estimating and predicting the unmanned vehicle's pose based on UKF. To ensure real-time performance, unlike traditional UAV-based onboard calculation methods, the onboard components of the unmanned vehicle are used as the computational unit to achieve a minimum Jerk trajectory landing trajectory planning method based on multi-sensor data fusion.

[0020] 3) To address the poor disturbance suppression capability of linear control methods under nonlinear and strongly coupled characteristics, an anti-interference trajectory optimization prediction and tracking method based on improved NMPC is proposed. The NMPC control method based on a nonlinear UAV model minimizes the cost function considering both control and reference trajectories, while taking into account both hard input constraints and soft state constraints. The constraints are satisfied: By utilizing UKF (Uniform Kinematics Function) to compensate for the effects of crosswinds and predict the UAV's position in real time, the UAV can autonomously calculate the optimal tracking trajectory based on the acquired trajectory information, achieving the highest possible accuracy in trajectory tracking. Furthermore, it provides good assurance regarding the real-time performance of the calculation, the impact of external disturbances such as wind, and the attitude maintenance under the influence of other UAVs and other dynamic obstacles.

[0021] 4) To improve the experimental verification capability of the UAV autonomous landing control algorithm, a host computer system design method is proposed, enabling autonomous switching of different flight states and control logic throughout the UAV landing process. On the hardware side, the host computer uses a Jetson Nano onboard processor for logic reasoning and calculation, communicates with the flight controller via MAVLINK2, and interacts between subsystems using LoRa pass-through. On the software side, the ROS system is used to implement one-click takeoff and landing, target tracking, and autonomous landing. Submodules of each system are built in Gazebo for joint simulation, and the simulation results are ported to the hardware system to verify the actual working effect and landing accuracy. The physical UAV is shown below. Figure 3 .

[0022] In the description of this invention, it should be understood that the terms "opening", "upper", "lower", "thickness", "top", "middle", "length", "inner", "around", etc., which indicate orientation or positional relationship, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting this invention.

[0023] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0024] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for autonomous landing of an unmanned aerial vehicle (UAV) platform based on NMPC multi-sensor fusion, characterized in that, include: Positioning based on RTK-GPS and image information fusion: precise positioning of each subsystem independently based on RTK-GPS at long distances outside the field of view, and pose calculation by combining visual information near the landing end; Jerk Smooth Safe Landing Trajectory Planning Based on Kinematic Equations: Kinematic Model Establishment of Landing Platform, UKF-Based Autonomous Vehicle Pose Estimation and Prediction; Anti-interference trajectory optimization prediction tracking based on improved NMPC: UKF is used to compensate for the influence of crosswinds and predict the position of the UAV in real time, so that the UAV can autonomously calculate the optimal tracking trajectory path based on the acquired trajectory information. A host computer system that autonomously switches between different flight states and control logics throughout the entire drone landing process; The positioning based on RTK-GPS and image information fusion includes independent RTK-GPS-based precise positioning of each subsystem at long distances outside the field of view, and pose calculation by combining visual information at the near landing end; at long distances, dual-end synchronized RTK-GPS information is used as the source of UAV position estimation information and target point position information; during the process of the UAV moving platform entering the UAV line of sight until landing, the detected identification information is fused with existing information. When the identification information is complete, the combined identification code information based on Aruco is directly used as the data input; the weighted fusion is performed using the following formula: in, The final relative pose matrix in the x-direction is used to combine the identifiers. A matrix of flag bits indicating whether a location identifier has been identified. This is a matrix showing the coordinate relationships between each identifier in the combined identifier and its center point. in, To obtain the relative distance in the x-direction, This represents the actual number of identifiers captured. Define the drone's own coordinates as The coordinates obtained through vision are The fused UAV world coordinates were obtained by using a weighted average method. as follows, in ; When the identification code is detected to be distorted, the space of the ring is used as the region of interest (ROI) for subsequent image calculations after the ring of the identification is detected. The distorted image information is repaired based on the current UAV attitude and the extracted information within the ROI in order to estimate the pose information of the target point. The Jerk smooth safe landing trajectory planning based on kinematic equations includes the establishment of a kinematic model of the landing platform, and the estimation and prediction of the unmanned vehicle's pose based on UKF. At the same time, in order to ensure the real-time performance of the system, unlike the traditional solution method based on UAV onboard, the onboard part of the unmanned vehicle is used as the computing unit to realize the minimum Jerk trajectory landing trajectory planning based on multi-sensor data fusion. The anti-interference trajectory optimization prediction tracking based on improved NMPC: The NMPC control method based on a nonlinear UAV model minimizes the cost function considering both control and reference trajectories, under the premise of considering hard input constraints and soft state constraints. The constraints are satisfied: 。 2. The autonomous landing method for a UAV dynamic platform based on NMPC multi-sensor fusion according to claim 1, characterized in that, The host computer system, which autonomously switches between different flight states and control logic throughout the entire UAV landing process, uses a Jetson Nano onboard computer for logic reasoning and calculation in terms of hardware. It communicates with the flight controller via MAVLINK2, and the subsystems interact with each other using LoRa pass-through. In terms of software, the ROS system is used to realize one-click take-off and landing, target tracking, and autonomous landing. The sub-modules of each system are built in Gazebo for joint simulation. The simulation results are then ported to the hardware system to verify the actual working effect and landing accuracy.

3. An autonomous landing device for a UAV platform based on NMPC multi-sensor fusion, characterized in that, include: The positioning module is used for positioning based on RTK-GPS and image information fusion: precise positioning of each subsystem independently based on RTK-GPS at long distances outside the field of view, and pose calculation by combining visual information near the landing end; at long distances, dual-end synchronized RTK-GPS information is used as the source of UAV position estimation information and target point position information; during the process of the UAV moving platform entering the UAV's line of sight until landing, the detected identification information is fused with existing information. When the identification information is complete, the combined identification code information based on Aruco is directly used as the data input; the weighted fusion is performed using the following formula: in, The final relative pose matrix in the x-direction is used to combine the identifiers. A matrix of flag bits indicating whether a location identifier has been identified. This is a matrix showing the coordinate relationships between each identifier in the combined identifier and its center point. in, To obtain the relative distance in the x-direction, This represents the actual number of identifiers captured. Define the drone's own coordinates as The coordinates obtained through vision are The fused UAV world coordinates were obtained by using a weighted average method. as follows, in ; When the identification code is detected to be distorted, the space of the ring is used as the region of interest (ROI) for subsequent image calculations after the ring of the identification is detected. The distorted image information is repaired based on the current UAV attitude and the extracted information within the ROI in order to estimate the pose information of the target point. The safe landing trajectory planning module is used to perform Jerk smooth safe landing trajectory planning based on kinematic equations: establishing the kinematic model of the landing platform, and estimating and predicting the pose of the unmanned vehicle based on UKF; at the same time, to ensure the real-time performance of the system, unlike the traditional UAV-based onboard solution method, the onboard part of the unmanned vehicle is used as the computing unit to realize the minimum Jerk trajectory landing trajectory planning based on multi-sensor data fusion. An anti-interference trajectory optimization prediction and tracking module is used for anti-interference trajectory optimization prediction and tracking based on improved NMPC: it utilizes UKF to compensate for the influence of crosswinds and predicts the UAV's position in real time, enabling the UAV to autonomously calculate the optimal tracking trajectory path based on the acquired trajectory information; and it uses an NMPC control method based on a nonlinear UAV model to minimize the cost function considering both control and reference trajectories, while taking into account hard input constraints and soft state constraints. The constraints are satisfied: ； The host computer module is used to autonomously switch between different flight states and control logic throughout the entire landing process of the drone.

4. The autonomous landing device for a UAV platform based on NMPC multi-sensor fusion as described in claim 3, characterized in that, The host computer module includes: on the hardware side, the host computer uses Jetson Nano onboard for logic reasoning and calculation, communicates with the flight controller via MAVLINK2, and interacts with subsystems via LoRa pass-through; on the software side, the ROS system is used to realize one-click take-off and landing, target tracking and autonomous landing; and the sub-modules of each system are built in Gazebo for joint simulation, the simulation results are ported to the hardware system and the actual working effect and landing accuracy are verified.

Images