Flight navigation method and device, electronic equipment and storage medium
By integrating inertial measurement sensors and optical flow detection data onto the UAV, and combining them with the Kalman filtering algorithm, the navigation error problem of the UAV when satellite signals are weakened or interrupted is solved, and precise flight control in complex terrain is achieved.
Patent Information
- Application Number
- CN202410992277.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
When drones are in mountainous areas or other locations where satellite signals are weak or interrupted, existing navigation systems are prone to errors, making it impossible for them to fly normally.
By utilizing built-in inertial measurement sensors and optical flow detection data, combined with Kalman filtering algorithms, the current position of the drone is estimated, and flight navigation is performed based on the estimated position to avoid errors caused by signal interruption or weakening.
This improves the reliability and accuracy of UAV flight navigation in complex terrain, ensuring mission completion.
Smart Images

Figure CN118936449A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a flight navigation method, device, electronic device and storage medium. Background Art
[0002] At present, drones are being used more and more widely. The attitude estimation of drones is not only crucial for the attitude control of drones, but also for the positioning and navigation of drones. Drones are usually used in some application scenarios that require close tracking of ground features, such as geological exploration, agricultural operations, environmental monitoring, etc. These operations often need to be carried out in mountainous conditions. The challenge of mountainous terrain for autonomous navigation of drones lies in the weakening of satellite signals, and sometimes even the absence of satellite signals. The current navigation system is highly dependent on the Global Positioning System (GPS), which is prone to errors due to interruption or weakening of GPS signals, resulting in the inability of drones to complete flight missions normally. Summary of the invention
[0003] The embodiments of the present application provide a flight navigation system, method, electronic device, and storage medium, which can avoid errors caused by signal interruption or weakening, resulting in the inability of the drone to complete the flight mission normally, thereby improving the reliability of flight navigation.
[0004] In a first aspect, an embodiment of the present application provides a flight navigation method, including:
[0005] Perform flight missions based on the built-in positioning system;
[0006] When it is detected that the signal strength corresponding to the positioning system does not meet the preset conditions, acquiring inertial measurement sensor data and optical flow detection data;
[0007] Estimate the current position of the drone based on the preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data;
[0008] Based on the current position and the flight route corresponding to the flight mission, the drone is controlled to perform flight navigation.
[0009] Optionally, in some embodiments of the present application, estimating the current position of the drone according to a preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data includes:
[0010] Determine a first initial Kalman filter corresponding to the inertial measurement sensor and a second initial Kalman filter corresponding to the optical flow detection sensor;
[0011] estimating a local state of the drone according to a preset Kalman filter algorithm, a first initial Kalman filter, and inertial measurement sensor data;
[0012] estimating the local state of the drone according to a preset Kalman filter algorithm, a second initial Kalman filter, and optical flow detection sensor data;
[0013] The estimated local states are integrated to obtain the current position of the UAV.
[0014] Optionally, in some embodiments of the present application, estimating the local state of the drone according to a preset Kalman filter algorithm, a first initial Kalman filter, and inertial measurement sensor data includes:
[0015] Extracting acceleration data, magnetic data, gyroscope data, and air pressure data from the inertial measurement sensor data;
[0016] Constructing a first observation model corresponding to the drone based on the acceleration data, magnetic data, gyroscope data, and air pressure data;
[0017] According to a preset Kalman filter algorithm and the first observation model, the local state of the UAV is estimated.
[0018] Optionally, in some embodiments of the present application, estimating the local state of the drone according to a preset Kalman filter algorithm, a second initial Kalman filter, and optical flow detection sensor data includes:
[0019] Determining three-dimensional motion information of the drone based on the optical flow detection sensor data and camera parameters corresponding to the optical flow detection sensor;
[0020] Constructing a second observation model corresponding to the UAV according to the three-dimensional motion information;
[0021] According to a preset Kalman filter algorithm and the second observation model, the local state of the UAV is estimated.
[0022] Optionally, in some embodiments of the present application, determining the three-dimensional motion information of the drone based on the optical flow detection sensor data and camera parameters corresponding to the optical flow detection sensor includes:
[0023] Dividing each frame of optical flow image of the optical flow detection sensor data into a plurality of image blocks;
[0024] Determine matching image blocks in the optical flow images after two adjacent frames are divided;
[0025] According to the matching results between the image blocks, the two-dimensional motion vector corresponding to the drone is determined, and based on the two-dimensional motion vector and the camera parameters corresponding to the optical flow detection sensor, the three-dimensional motion information of the drone is determined.
[0026] Optionally, in some embodiments of the present application, estimating the current position of the drone according to a preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data includes:
[0027] Performing anomaly detection on the inertial measurement sensor data and the optical flow detection data;
[0028] According to the anomaly detection results, the sensor weight corresponding to the sensor is output;
[0029] The current position of the drone is estimated based on the sensor weights, a preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data.
[0030] Optionally, in some embodiments of the present application, controlling the drone to perform flight navigation based on the current position and the flight route corresponding to the flight mission includes:
[0031] obtaining a flight route from the flight mission;
[0032] Comparing the current position with the flight route to obtain a deviation between the UAV and the flight route;
[0033] The drone is controlled to perform flight navigation according to the flight status of the drone and the deviation between the drone and a flight route.
[0034] In a second aspect, an embodiment of the present application provides a flight navigation device, including:
[0035] An execution module, used to execute flight missions based on the built-in positioning system;
[0036] An acquisition module, used to acquire inertial measurement sensor data and optical flow detection data when it is detected that the signal strength corresponding to the positioning system does not meet a preset condition;
[0037] The estimation module is used to estimate the current position of the drone based on the preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data;
[0038] The control module is used to control the UAV to perform flight navigation based on the current position and the flight route corresponding to the flight mission.
[0039] Correspondingly, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of any of the above methods when executing the program.
[0040] The present application also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements any of the above methods when the computer program is executed by a processor.
[0041] The present application provides a flight navigation method, device, electronic device and storage medium. When performing a flight mission based on a built-in positioning system, when it is detected that the signal strength corresponding to the positioning system does not meet the preset conditions, inertial measurement sensor data and optical flow detection data are obtained, and then, according to the preset Kalman filter algorithm, inertial measurement sensor data and optical flow detection data, the current position of the drone is estimated. Finally, based on the current position and the flight route corresponding to the flight mission, the drone is controlled to perform flight navigation. In the flight navigation scheme provided by the present application, when it is detected that the signal strength corresponding to the positioning system does not meet the preset conditions, inertial measurement sensor data and optical flow detection data are obtained, and then, according to the preset Kalman filter algorithm, inertial measurement sensor data and optical flow detection data, the current position of the drone is estimated to avoid errors caused by signal interruption or weakening, which causes the drone to be unable to complete the flight mission normally. After determining the current position of the drone, the drone is controlled to perform flight navigation in combination with the flight route corresponding to the flight mission, thereby improving the reliability of flight navigation. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0043] Figure 1 is a flowchart of a flight navigation method provided by an embodiment of the present application;
[0044] Figure 2 is a schematic diagram of block matching in the flight navigation method provided in an embodiment of the present application;
[0045] Figure 3 is a schematic diagram of the structure of a flight navigation device provided in an embodiment of the present application;
[0046] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of systems and methods consistent with the examples detailed in the attached claims, or with some aspects of the present application.
[0048] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover descriptions such as non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.
[0049] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0050] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.
[0051] Embodiments of the present application provide a flight navigation method, device, electronic device, and storage medium.
[0052] The flight navigation device can be deployed in a drone. A drone (Unmanned Aerial Vehicle, UAV), also known as a remote-controlled aircraft or an unmanned aircraft system (Unmanned Aircraft System, UAS), is an aircraft that does not require a pilot to operate in the cabin. The drone can communicate with the server via wired or wireless means. The server can include an independently operated server or a distributed server, or a server cluster composed of multiple servers. The terminal can include a mobile phone, a tablet computer or a personal computer (PC, Personal Computer).
[0053] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.
[0054] The present application provides a flight navigation method, including: executing a flight mission based on a built-in positioning system; when it is detected that the signal strength corresponding to the positioning system does not meet a preset condition, acquiring inertial measurement sensor data and optical flow detection data; estimating the current position of the drone according to a preset Kalman filter algorithm, inertial measurement sensor data and optical flow detection data; and controlling the drone to perform flight navigation based on the current position and a flight route corresponding to the flight mission.
[0055] See also Figure 1 , Figure 1 : is a flowchart of the flight navigation method provided by the present application, and the flight navigation method specifically includes the following process:
[0056] 101. Perform flight missions based on the built-in positioning system.
[0057] The drone uses its onboard GPS or other satellite navigation system to determine its position and navigate according to the predetermined flight path. For example, when a drone performs a surveying and mapping task in a mountainous area, its built-in positioning system (such as GPS) is its main navigation tool. When the drone is started, it first uses its built-in GPS system to obtain initial position and altitude information. While performing the surveying and mapping task, the signal strength of the GOS is continuously detected. When the signal strength does not meet the preset conditions (such as interruption or the signal strength is less than the preset value), step 102 is executed.
[0058] 102. When it is detected that the signal strength corresponding to the positioning system does not meet the preset condition, inertial measurement sensor data and optical flow detection data are obtained.
[0059] An Inertial Measurement Unit (IMU) is a device that integrates multiple sensors to measure and report an object's specific force, angular velocity, and in some cases, the orientation of an object around a magnetic field. IMUs play a vital role in drone navigation, especially when GPS signals are limited or unavailable. IMUs typically include the following sensors: accelerometers: measure the acceleration of an object in each axis; gyroscopes: measure the angular velocity of an object, i.e., the rate and direction of rotation; and magnetometers: detect the direction of the magnetic field, thereby determining the orientation of an object relative to the Earth's magnetic field.
[0060] Optical flow sensors estimate the motion of objects by analyzing pixel changes between consecutive image frames. This approach can provide information about the drone's motion relative to the ground or other background.
[0061] It can be understood that the inertial measurement sensor data includes accelerometer data, gyroscope data, and magnetometer data, among which, accelerometer data: used to measure the linear acceleration of the drone, helping to determine the speed and position changes of the drone. Gyroscope data: provides angular velocity information of the drone, helping to determine the rotation and attitude changes of the drone. Magnetometer data: combined with accelerometer and gyroscope data, provides information about the direction of the drone, especially when GPS signals are not available. Optical flow refers to the change in pixel intensity patterns caused by object motion or camera movement in an image sequence. It provides information about the motion of the object relative to the observer. Optical flow detection data is a technique for estimating the motion of an object by analyzing the changes between consecutive image frames, and is particularly suitable for drone navigation and control. Optical flow: Optical flow refers to the change in pixel intensity patterns caused by object motion or camera movement in an image sequence. It provides information about the motion of the object relative to the observer. Optical flow detection: estimates the speed and direction of an object's movement by calculating the changes in feature points in an image sequence over time. The steps of optical flow detection are as follows
[0062] Image acquisition: Use a camera to continuously capture images to form an image sequence.
[0063] Feature point detection: Detect feature points in an image that are identifiable and trackable in an image sequence.
[0064] Feature point tracking: Track these feature points in consecutive image frames and record their position changes.
[0065] Optical flow calculation: Calculate the speed and direction of an object based on changes in the position of feature points.
[0066] Optionally, in some embodiments of the present application, a COMS-based camera with a resolution of 752×480 is installed on the bottom of the drone to capture ground images. The optical flow algorithm uses these images for flow measurement and calculates the optical flow between two consecutive frames to achieve sub-pixel accuracy. The drone uses an onboard gyroscope for angular rate compensation to improve the accuracy of optical flow measurement. The block matching algorithm (BM algorithm) is commonly used in optical flow algorithms, which is widely used for motion estimation in video coding. The image is divided into blocks, and the algorithm attempts to search for the best matching block in the previous frame, with the goal of minimizing the sum of absolute differences within the search window.
[0067] Specifically, a COMS-based camera with a resolution of 752×480 is placed at the bottom of the drone. The optical flow algorithm uses the images collected by the camera for flow measurement. The optical flow between two consecutive frames is calculated and refined to achieve sub-pixel accuracy. Angular rate compensation uses the onboard gyroscope. Finally, the laser ranging sonar provides accurate height from the ground; it is used to scale the optical flow readings to obtain precise metric motion details. The ground point is D[Xp,Yp,Zp]. The focal length f can give the camera coordinate system: Zp is the height given by the laser ranging sensor.
[0068] The commonly used optical flow algorithm is the block matching algorithm (BM algorithm). The BM algorithm is widely used in motion estimation in video coding. In the BM algorithm, the image is divided into blocks, such as Figure 2 As shown. For each block, the algorithm tries to search for the best matching block in the previous frame. The goal is to minimize the sum of absolute differences within the search window. That is:
[0069]
[0070] The motion vector MV in (a,b) is defined as follows:
[0071]
[0072]
[0073] Among them, SAD (Sum of Absolute Differences): Sum of Absolute Differences is a metric used in optical flow detection to evaluate the difference between two image blocks. The motion vector MV is defined as the (a, b) value that minimizes SAD. This means that a match is found where the difference between the image blocks of the current frame and the previous frame is the smallest, g i-1 represents the image block of the previous frame, g i represents the image block of the current frame. The search range R is defined as the set of (a, b) that satisfies certain conditions: a and b must be in the range [-W, W], where W is the size of the search window. It must be a valid position in the previous frame. The calculation of the motion vector MV is completed by finding the (a, b) value that minimizes the SAD within the search range R. a and b represent the horizontal and vertical components of the motion vector (MV), which are used to describe the direction and distance that a block in an image moves between two consecutive frames in optical flow detection. Specifically:
[0074] a: The amount of movement in the horizontal direction, which indicates how many pixels the image block is moved horizontally from its original position.
[0075] b: The amount of movement in the vertical direction, which indicates how many pixels the image block has moved vertically from its original position. In the optical flow detection algorithm, the motion vector (a, b) is usually determined by minimizing a specific error function, which measures the difference between the current frame image block and the previous frame image block under the assumed amount of movement.
[0076] 103. Estimate the current position of the drone based on the preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data.
[0077] The Kalman filter algorithm is an efficient recursive filter that can estimate the state of a dynamic system from a series of measurements containing noise. This algorithm has a wide range of applications in drone navigation, control systems, signal processing and other fields.
[0078] The Kalman filter implements state estimation in two main steps: prediction and update.
[0079] predict:
[0080] Based on the system's dynamic model and control input, predict the state at the next moment.
[0081] At the same time, the state covariance of the next moment is predicted, which indicates the uncertainty of the predicted state.
[0082] renew:
[0083] Using the new measurement data and the observation model, the state estimate is updated.
[0084] Calculate the Kalman gain, which is a coefficient that weighs prediction error and measurement error.
[0085] Update state estimates and state covariance to reduce prediction uncertainty.
[0086] Kalman filter algorithms are used for UAV navigation, especially in mountainous conditions where GPS signals may be unstable or unavailable. The Extended Kalman Filter (EKF) is used to fuse IMU data and optical flow detection data for state estimation. The Extended Kalman Filter is an algorithm used to handle state estimation of nonlinear systems. It linearizes the nonlinear model so that the standard Kalman filter can be applied to a wider range of systems. The Extended Kalman Filter handles nonlinear systems by linearizing the nonlinear model at each time step. This is usually achieved by performing a Taylor series expansion around the current estimated state, thereby converting the nonlinear problem into a linear problem.
[0087] First, get the initial position, velocity, and attitude of the drone, then, the initial covariance matrix: initialize the uncertainty of the state estimate (covariance matrix). State prediction: use IMU data and nonlinear system model to predict the state of the drone at the next moment. Image processing: use the camera to capture the ground image and calculate the optical flow between two consecutive frames. Angular rate compensation: use gyroscope data to perform angular rate compensation to improve the accuracy of optical flow measurement. Distance scaling: combined with the height information from the ground provided by the laser ranging sonar, scale the optical flow readings to obtain accurate metric motion details. Observation equation: establish an observation model to associate the state of the drone with the optical flow detection data. Finally, output the current position estimate of the drone to provide a basis for navigation and control.
[0088] Optionally, in some embodiments of the present application, the step of “estimating the current position of the drone according to a preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data” may specifically include:
[0089] Determine a first initial Kalman filter corresponding to the inertial measurement sensor and a second initial Kalman filter corresponding to the optical flow detection sensor;
[0090] The local state of the UAV is estimated based on the preset Kalman filter algorithm, the first initial Kalman filter and the inertial measurement sensor data;
[0091] The local state of the UAV is estimated based on the preset Kalman filter algorithm, the second initial Kalman filter and the optical flow detection sensor data;
[0092] The estimated local states are integrated to obtain the current position of the drone.
[0093] First, determine the initial Kalman filter (i.e., the first initial Kalman filter and the second initial Kalman filter), the first initial Kalman filter: the initial Kalman filter corresponding to the inertial measurement sensor (IMU) data. The IMU includes an accelerometer, a gyroscope, and a magnetometer, which provide the attitude, speed, and direction information of the drone. The second initial Kalman filter: the initial Kalman filter corresponding to the optical flow detection sensor data. Optical flow detection estimates the motion of the drone relative to the ground by analyzing the changes between consecutive image frames. Use the first initial Kalman filter: use the IMU data to estimate the local state of the drone through the Kalman filter algorithm. Predict the position, speed, and attitude of the drone, taking into account the system dynamics and control inputs, as well as the uncertainty of the predicted state.
[0094] Use the second initial Kalman filter: Use the optical flow detection data to estimate the local state of the drone through the Kalman filter algorithm. Optical flow data processing: Use the camera to capture the ground image, calculate the optical flow between two consecutive frames, and estimate the movement of the drone relative to the ground. Angular rate compensation: Use gyroscope data to perform angular rate compensation to improve the accuracy of optical flow measurement. Distance scaling: Combine the height information from the ground provided by the laser ranging sonar to scale the optical flow readings to obtain accurate metric motion details. Fusion of the estimation results based on IMU and optical flow detection data to obtain the final position estimate.
[0095] Optionally, in some embodiments of the present application, the step of “estimating the local state of the drone according to a preset Kalman filter algorithm, a first initial Kalman filter, and inertial measurement sensor data” may specifically include:
[0096] Extract acceleration data, magnetic data, gyroscope data, and air pressure data from inertial measurement sensor data;
[0097] Constructing a first observation model corresponding to the drone based on the acceleration data, the magnetic data, the gyroscope data, and the air pressure data;
[0098] According to the preset Kalman filter algorithm and the first observation model, the local state of the UAV is estimated.
[0099] Based on the acceleration data, magnetic data, gyroscope data and air pressure data, an observation model of the drone is constructed. This model will be used in the Kalman filter algorithm to associate the sensor data with the actual state of the drone. State vector: Define the state vector of the drone, which usually includes position, velocity, attitude (pitch angle, roll angle, yaw angle), etc. Observation equation: Build the observation equation to describe how to calculate the observation value from the state vector. For example, use the accelerometer data to calculate the velocity, and use the magnetometer and gyroscope data to calculate the attitude.
[0100] Prediction steps:
[0101] Predict the state of the UAV using nonlinear system models and control inputs.
[0102] Update the predicted state covariance, accounting for process noise.
[0103] Update steps:
[0104] Using the observation model and IMU data, update the drone's state estimate. Calculate the Kalman gain to weigh prediction error and measurement error. Update the state estimate and state covariance.
[0105] Local state estimation: The local state of the drone is estimated by combining the Kalman filter algorithm with the first observation model. This includes the estimation of position, velocity and attitude. State update: At each time step, the state estimate of the drone is updated using the latest IMU data to ensure the accuracy and real-time performance of the estimate.
[0106] Multi-sensor data fusion: Combine IMU data with optical flow detection data and use Kalman filter algorithm for data fusion to improve the accuracy of drone state estimation. Finally, the current position estimate of the drone is output to provide a basis for navigation and control.
[0107] Optionally, in some embodiments of the present application, the step of “estimating the local state of the drone according to a preset Kalman filter algorithm, a second initial Kalman filter, and optical flow detection sensor data” may specifically include:
[0108] Determine the three-dimensional motion information of the drone based on the optical flow detection sensor data and the camera parameters corresponding to the optical flow detection sensor;
[0109] Constructing a second observation model corresponding to the UAV according to the three-dimensional motion information;
[0110] According to the preset Kalman filter algorithm and the second observation model, the local state of the UAV is estimated.
[0111] The optical flow image is divided into multiple image blocks, and matching image blocks are determined in two adjacent frames. Based on the matching image blocks, the two-dimensional motion vector of the drone relative to the ground is calculated. Combined with the camera parameters corresponding to the optical flow detection sensor (such as focal length, resolution, etc.), the two-dimensional motion vector is converted into three-dimensional motion information.
[0112] Based on the 3D motion information, a second observation model of the drone is constructed. The state vector of the drone is defined, which usually includes position, velocity, and attitude. An observation equation is established to describe how to calculate the observation value from the state vector, for example, using optical flow data to estimate the relative velocity and displacement of the drone.
[0113] Prediction steps:
[0114] Predict the state of the UAV using nonlinear system models and control inputs.
[0115] Update the predicted state covariance, accounting for process noise.
[0116] Update steps:
[0117] Using the observation model and optical flow detection data, the state estimate of the drone is updated.
[0118] Calculate the Kalman gain to weigh the prediction error and measurement error.
[0119] Update state estimates and state covariance.
[0120] The local state of the drone is estimated through the Kalman filter algorithm combined with the second observation model. This includes the estimation of position, velocity and attitude. At each time step, the latest optical flow detection data is used to update the state estimate of the drone to ensure the accuracy and real-time performance of the estimate. The optical flow detection data is combined with the IMU data and the Kalman filter algorithm is used for data fusion to improve the accuracy of the drone state estimation. Finally, the current position estimate of the drone is output to provide a basis for navigation and control.
[0121] Through these steps, the drone can estimate its local state in real time, thereby achieving precise navigation and control.
[0122] Optionally, in some embodiments of the present application, the step of “determining the three-dimensional motion information of the drone based on the optical flow detection sensor data and the camera parameters corresponding to the optical flow detection sensor” may specifically include:
[0123] Divide each frame of optical flow image of the optical flow detection sensor data into multiple image blocks;
[0124] Determine matching image blocks in the optical flow images after two adjacent frames are divided;
[0125] According to the matching results between the image blocks, the two-dimensional motion vector corresponding to the UAV is determined, and based on the two-dimensional motion vector and the camera parameters corresponding to the optical flow detection sensor, the three-dimensional motion information of the UAV is determined.
[0126] For example, in two consecutive frames of images, find the corresponding feature point pairs, that is, the matching image blocks. Based on the matching image blocks, calculate the two-dimensional motion vector of the drone relative to the ground. These vectors represent the horizontal and vertical motion of the drone on the image plane. The two-dimensional motion vector is converted into three-dimensional motion information using the camera parameters corresponding to the optical flow detection sensor, such as focal length, resolution, etc. Combined with the height information from the ground provided by the laser ranging sonar or other sensors, the two-dimensional motion vector is expanded in the vertical direction to obtain the three-dimensional motion information of the drone.
[0127] Optionally, in some embodiments of the present application, the step of “estimating the current position of the drone according to a preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data” may specifically include:
[0128] Perform anomaly detection on inertial measurement sensor data and optical flow detection data;
[0129] According to the anomaly detection results, the sensor weight corresponding to the sensor is output;
[0130] Estimate the current position of the drone based on sensor weights, a preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data.
[0131] IMU data (accelerometer, gyroscope, magnetometer, barometer) and optical flow detection data are monitored and analyzed in real time to detect whether there are outliers or deviations. Based on the results of anomaly detection, a weight is assigned to each sensor. The weight reflects the credibility and accuracy of the sensor data. The weight is dynamically adjusted based on the quality and reliability of the sensor data to optimize navigation performance.
[0132] First observation model: Build the dynamic model of the drone based on IMU data (acceleration, angular velocity, magnetic field direction, and air pressure).
[0133] Second observation model: Build the UAV’s visual observation model based on optical flow detection data and camera parameters.
[0134] Predict the state of the drone using the system dynamics model and control inputs. Update the predicted state covariance to account for process noise. Update the drone's state estimate using the observation model and sensor data. Calculate the Kalman gain to weigh prediction error and measurement error. Combine the IMU data with the optical flow detection data and use the Kalman filter algorithm for data fusion. Estimate the drone's current position using the Kalman filter algorithm, combined with the sensor weights and observation model. At each time step, update the drone's state estimate using the latest sensor data to ensure the accuracy and real-time nature of the estimate. Implement fault detection features to dynamically reduce the impact of noisy sensor outputs. Based on the fault detection results, adjust the sensor weights in the Kalman filter algorithm to improve navigation accuracy. : Finally, output the drone's current position estimate to provide a basis for navigation and control.
[0135] Loop: Repeat the prediction, update, fault detection, and dynamic adjustment steps until the drone completes the mission.
[0136] Through these steps, the drone is able to estimate its position in real time using IMU data and optical flow detection data, thereby achieving precise navigation and control. This method is particularly suitable for environments where GPS signals are limited or unavailable, such as complex terrain such as mountainous areas. By dynamically adjusting sensor weights, the system can better handle uncertainty and noise in sensor data, improving navigation accuracy and robustness.
[0137] 104. Based on the current position and the flight route corresponding to the flight mission, control the drone to perform flight navigation.
[0138] For example, extract the flight route from the preset flight mission. The flight route may include a series of waypoints, paths, and expected flight parameters (such as speed, altitude, etc.). Use IMU data, optical flow detection data, and Kalman filtering algorithm to estimate the current position of the drone. Among them, the real-time position of the drone is continuously updated through sensor data and filtering algorithms. Compare the current position of the drone with the preset flight route to determine the position of the drone on the route. Calculate the deviation between the current position of the drone and the nearest waypoint on the flight route, which includes horizontal deviation and vertical deviation. Monitor the flight status of the drone, including speed, altitude, attitude, etc. According to the flight status of the drone and the deviation from the flight route, adjust the flight control parameters of the drone to achieve flight navigation.
[0139] Optionally, in some embodiments of the present application, the step of “controlling the drone to perform flight navigation based on the current position and the flight route corresponding to the flight mission” may specifically include:
[0140] Get flight routes from flight missions;
[0141] Compare the current position with the flight path to obtain the deviation between the drone and the flight path;
[0142] According to the flight status of the drone and the deviation between the drone and the flight route, the drone is controlled to perform flight navigation.
[0143] For example, a drone needs to perform a mapping mission in a mountainous area. The flight route has been preset and includes multiple waypoints and paths. The drone obtains the preset flight route from the control system, which includes a series of waypoints from the take-off point to the target area. After the drone takes off, the IMU and optical flow detection sensors start working, providing the real-time attitude of the drone and the motion information relative to the ground. The Kalman filter algorithm is used to combine these sensor data to estimate the current position of the drone. During the flight of the drone, the system continuously compares the current position with the preset flight route. Assume that when the drone approaches the first waypoint, it is found that there is a certain horizontal deviation. The system calculates the deviation distance to be 10 meters. Based on the deviation information and the current flight status of the drone (such as speed, altitude, attitude), the control system generates adjustment instructions. The flight controller of the drone receives the instructions and adjusts the flight attitude and speed to make the drone approach the predetermined waypoint. The drone continues to fly, and the system continues to monitor its position and flight status, and adjusts the flight path in real time to keep it on the predetermined route. When encountering changes in mountain terrain or wind speed, the system will dynamically adjust the flight parameters to ensure that the drone completes the mission safely and accurately.
[0144] Through this process, the drone can achieve precise path tracking and flight control in complex environments, especially in mountainous conditions where GPS signals are limited, using IMU and optical flow detection data for assisted navigation to improve navigation accuracy and robustness.
[0145] As can be seen from the above, the embodiment of the present application provides a flight navigation method. When performing a flight mission based on a built-in positioning system, when it is detected that the signal strength corresponding to the positioning system does not meet the preset conditions, inertial measurement sensor data and optical flow detection data are obtained, and then, according to the preset Kalman filter algorithm, inertial measurement sensor data and optical flow detection data, the current position of the drone is estimated. Finally, based on the current position and the flight route corresponding to the flight mission, the drone is controlled to perform flight navigation. In the flight navigation scheme provided by the present application, when it is detected that the signal strength corresponding to the positioning system does not meet the preset conditions, inertial measurement sensor data and optical flow detection data are obtained, and then, according to the preset Kalman filter algorithm, inertial measurement sensor data and optical flow detection data, the current position of the drone is estimated to avoid errors caused by signal interruption or weakening, which causes the drone to be unable to complete the flight mission normally. After determining the current position of the drone, the drone is controlled to perform flight navigation in combination with the flight route corresponding to the flight mission, thereby improving the reliability of flight navigation.
[0146] In order to better implement the flight navigation method of the present application embodiment, the present application embodiment also provides a flight navigation device, wherein the meanings of the terms are the same as those in the above-mentioned flight navigation system, and the specific implementation details can refer to the description in the system embodiment.
[0147] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a flight navigation device provided in an embodiment of the present application, wherein the flight navigation device may specifically include an execution module 201, an acquisition module 202, an estimation module 203, and a control module 204, which may be specifically as follows:
[0148] The execution module 201 is used to execute the flight mission based on the built-in positioning system.
[0149] The acquisition module 202 is used to acquire inertial measurement sensor data and optical flow detection data when it is detected that the signal strength corresponding to the positioning system does not meet a preset condition.
[0150] The estimation module 203 is used to estimate the current position of the drone based on a preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data.
[0151] Optionally, in some embodiments of the present application, the estimation module 203 may specifically include:
[0152] A determination unit, used to determine a first initial Kalman filter corresponding to the inertial measurement sensor and a second initial Kalman filter corresponding to the optical flow detection sensor;
[0153] A first estimation unit, configured to estimate a local state of the UAV according to a preset Kalman filter algorithm, a first initial Kalman filter, and inertial measurement sensor data;
[0154] A second estimation unit, used to estimate the local state of the UAV according to a preset Kalman filter algorithm, a second initial Kalman filter and optical flow detection sensor data;
[0155] The fusion unit is used to fuse the estimated local states to obtain the current position of the UAV.
[0156] Optionally, in some embodiments of the present application, the first estimation unit can be specifically used to: extract acceleration data, magnetic data, gyroscope data and air pressure data from inertial measurement sensor data; construct a first observation model corresponding to the drone based on the acceleration data, magnetic data, gyroscope data and air pressure data; and estimate the local state of the drone according to a preset Kalman filter algorithm and the first observation model.
[0157] Optionally, in some embodiments of the present application, the second estimation unit may specifically include:
[0158] A determination subunit, configured to determine three-dimensional motion information of the drone based on optical flow detection sensor data and camera parameters corresponding to the optical flow detection sensor;
[0159] A construction subunit is used to construct a second observation model corresponding to the UAV according to the three-dimensional motion information;
[0160] The estimation subunit is used to estimate the local state of the UAV according to a preset Kalman filter algorithm and a second observation model.
[0161] Optionally, in some embodiments of the present application, the determination subunit is specifically used for:
[0162] Divide each frame of optical flow image of the optical flow detection sensor data into multiple image blocks;
[0163] Determine matching image blocks in the optical flow images after two adjacent frames are divided;
[0164] According to the matching results between the image blocks, the two-dimensional motion vector corresponding to the UAV is determined, and based on the two-dimensional motion vector and the camera parameters corresponding to the optical flow detection sensor, the three-dimensional motion information of the UAV is determined.
[0165] Optionally, in some embodiments of the present application, the estimation module 203 may be specifically used for:
[0166] Perform anomaly detection on inertial measurement sensor data and optical flow detection data;
[0167] According to the anomaly detection results, the sensor weight corresponding to the sensor is output;
[0168] Estimate the current position of the drone based on sensor weights, a preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data.
[0169] The control module 204 is used to control the UAV to perform flight navigation based on the current position and the flight route corresponding to the flight mission.
[0170] Optionally, in some embodiments of the present application, the control module 204 may be specifically used to:
[0171] Get flight routes from flight missions;
[0172] Compare the current position with the flight route to obtain the deviation between the drone and the flight route;
[0173] According to the flight status of the drone and the deviation between the drone and the flight route, the drone is controlled to perform flight navigation.
[0174] The embodiment of the present application provides a flight navigation device, when the execution module 201 performs a flight mission based on the built-in positioning system, when it is detected that the signal strength corresponding to the positioning system does not meet the preset conditions, the acquisition module 202 acquires the inertial measurement sensor data and the optical flow detection data, then the estimation module 203 estimates the current position of the drone according to the preset Kalman filter algorithm, the inertial measurement sensor data and the optical flow detection data, and finally, the control module 204 controls the drone to perform flight navigation based on the current position and the flight route corresponding to the flight mission. In the flight navigation scheme provided by the present application, when it is detected that the signal strength corresponding to the positioning system does not meet the preset conditions, the inertial measurement sensor data and the optical flow detection data are acquired, then, according to the preset Kalman filter algorithm, the inertial measurement sensor data and the optical flow detection data, the current position of the drone is estimated to avoid errors caused by signal interruption or weakening, which causes the drone to be unable to complete the flight mission normally. After determining the current position of the drone, the drone is controlled to perform flight navigation in combination with the flight route corresponding to the flight mission, thereby improving the reliability of flight navigation.
[0175] In addition, the present application also provides an electronic device, such as Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:
[0176] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0177] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.
[0178] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and flight navigation methods by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.
[0179] The electronic device also includes a power supply 303 for supplying power to each component. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions. The power supply 303 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.
[0180] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0181] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby realizing various functions, as follows:
[0182] Based on the historical attitude angle and historical control signal of the UAV at the previous moment, the predicted attitude angle of the UAV is determined; based on the historical covariance matrix corresponding to the previous moment and the historical control matrix corresponding to the historical control signal, the covariance matrix corresponding to the current moment is determined; the current attitude angle of the UAV is obtained, and the predicted attitude angle is updated according to the current attitude angle, the predicted attitude angle and the covariance matrix; based on the updated attitude angle and the current attitude angle, the control signal corresponding to the current moment is generated; in response to the control signal, the current flight attitude is controlled.
[0183] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0184] In the embodiment of the present application, when performing a flight mission based on a built-in positioning system, when it is detected that the signal strength corresponding to the positioning system does not meet the preset conditions, inertial measurement sensor data and optical flow detection data are obtained, and then, according to the preset Kalman filter algorithm, inertial measurement sensor data and optical flow detection data, the current position of the drone is estimated, and finally, based on the current position and the flight route corresponding to the flight mission, the drone is controlled to perform flight navigation. In the flight navigation scheme provided by the present application, when it is detected that the signal strength corresponding to the positioning system does not meet the preset conditions, inertial measurement sensor data and optical flow detection data are obtained, and then, according to the preset Kalman filter algorithm, inertial measurement sensor data and optical flow detection data, the current position of the drone is estimated, so as to avoid errors caused by signal interruption or weakening, which causes the drone to be unable to complete the flight mission normally. After determining the current position of the drone, the drone is controlled to perform flight navigation in combination with the flight route corresponding to the flight mission, thereby improving the reliability of flight navigation.
[0185] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0186] To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the flight navigation methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:
[0187] Execute flight missions based on the built-in positioning system; obtain inertial measurement sensor data and optical flow detection data when it is detected that the signal strength corresponding to the positioning system does not meet the preset conditions; estimate the current position of the drone based on the preset Kalman filter algorithm, inertial measurement sensor data and optical flow detection data; control the drone for flight navigation based on the current position and the flight route corresponding to the flight mission.
[0188] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0189] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0190] Since the instructions stored in the storage medium can execute the steps in any one of the flight navigation methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the flight navigation methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0191] The above is a detailed introduction to a flight navigation method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A flight navigation method, It is characterized in that include: Perform flight missions based on the built-in positioning system; When it is detected that the signal strength corresponding to the positioning system does not meet the preset conditions, acquiring inertial measurement sensor data and optical flow detection data; Estimate the current position of the drone based on the preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data; Based on the current position and the flight route corresponding to the flight mission, the drone is controlled to perform flight navigation.
2. The flight navigation method according to claim 1, It is characterized in that The method of estimating the current position of the drone based on a preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data includes: Determine a first initial Kalman filter corresponding to the inertial measurement sensor and a second initial Kalman filter corresponding to the optical flow detection sensor; estimating a local state of the drone according to a preset Kalman filter algorithm, a first initial Kalman filter, and inertial measurement sensor data; estimating the local state of the drone according to a preset Kalman filter algorithm, a second initial Kalman filter, and optical flow detection sensor data; The estimated local states are integrated to obtain the current position of the UAV.
3. The flight navigation method according to claim 2, It is characterized in that The estimating the local state of the drone according to the preset Kalman filter algorithm, the first initial Kalman filter and the inertial measurement sensor data includes: Extracting acceleration data, magnetic data, gyroscope data, and air pressure data from the inertial measurement sensor data; Constructing a first observation model corresponding to the drone based on the acceleration data, magnetic data, gyroscope data, and air pressure data; According to a preset Kalman filter algorithm and the first observation model, the local state of the UAV is estimated.
4. The flight navigation method according to claim 2, It is characterized in that The method of estimating the local state of the drone according to the preset Kalman filter algorithm, the second initial Kalman filter and the optical flow detection sensor data includes: Determining three-dimensional motion information of the drone based on the optical flow detection sensor data and camera parameters corresponding to the optical flow detection sensor; Constructing a second observation model corresponding to the UAV according to the three-dimensional motion information; According to a preset Kalman filter algorithm and the second observation model, the local state of the UAV is estimated.
5. The flight navigation method according to claim 4, It is characterized in that The determining the three-dimensional motion information of the drone based on the optical flow detection sensor data and the camera parameters corresponding to the optical flow detection sensor includes: Dividing each frame of optical flow image of the optical flow detection sensor data into a plurality of image blocks; Determine matching image blocks in the optical flow images after two adjacent frames are divided; According to the matching results between the image blocks, the two-dimensional motion vector corresponding to the drone is determined, and based on the two-dimensional motion vector and the camera parameters corresponding to the optical flow detection sensor, the three-dimensional motion information of the drone is determined.
6. The flight navigation method according to claim 1, It is characterized in that The method of estimating the current position of the drone based on a preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data includes: Performing anomaly detection on the inertial measurement sensor data and the optical flow detection data; According to the anomaly detection results, the sensor weight corresponding to the sensor is output; The current position of the drone is estimated based on the sensor weights, a preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data.
7. The flight navigation method according to claim 1, It is characterized in that The controlling the UAV to perform flight navigation based on the current position and the flight route corresponding to the flight mission includes: obtaining a flight route from the flight mission; Comparing the current position with the flight route to obtain a deviation between the UAV and the flight route; The drone is controlled to perform flight navigation according to the flight status of the drone and the deviation between the drone and a flight route.
8. A flight navigation device, It is characterized in that include: An execution module, used to execute flight missions based on the built-in positioning system; An acquisition module, used to acquire inertial measurement sensor data and optical flow detection data when it is detected that the signal strength corresponding to the positioning system does not meet a preset condition; The estimation module is used to estimate the current position of the drone based on the preset Kalman filter algorithm, inertial measurement sensor data, and optical flow detection data; The control module is used to control the UAV to perform flight navigation based on the current position and the flight route corresponding to the flight mission.
9. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program according to the steps of the flight navigation method according to any one of claims 1 to 7.
10. A storage medium, It is characterized in that A computer program is stored which can be loaded by a processor and execute the flight navigation method according to any one of claims 1 to 7.
