Visual guidance precise landing method for unmanned aerial vehicle

Abstract

The invention relates to an unmanned aerial vehicle visual guidance precise landing method, and belongs to the technical field of machine vision and unmanned aerial vehicle control. The method comprises the following steps: capturing surrounding environment information through a camera of an unmanned aerial vehicle to obtain operation environment data, and constructing a flight control model through construction parameters of the unmanned aerial vehicle; setting an expected landing point, extracting a target image from the corresponding operation environment data according to the expected landing point, and establishing visual mapping between the unmanned aerial vehicle and the expected landing point based on the target image; the relative position data and the to-be-flied distance of the unmanned aerial vehicle are calculated according to the visual mapping, and a predicted moving trajectory based on the relative position data and the to-be-flied distance is obtained through a flight control model; and an expected landing point is automatically captured and tracked in the landing process, relative pose information between the unmanned aerial vehicle and the expected landing point is solved in real time, and flight parameters of the unmanned aerial vehicle are adjusted in real time according to the relative pose information and the predicted moving trajectory. Visual guidance precise landing of the unmanned aerial vehicle is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision and unmanned aerial vehicle control, and in particular relates to a method for accurate landing of an unmanned aerial vehicle guided by vision. Background Art

[0002] The process of automatic landing of drones is the process of automatic and complete control of the aircraft by the onboard automatic flight to fly and land. Due to the high risk and technical difficulty of drone flight, the requirements for automatic landing are also relatively high. At present, the visual guidance precision landing method still has the following areas for improvement: in the process of capturing images, changes in the natural environment, such as changes in lighting conditions (strong light or backlight), weather conditions (rain, fog, snow), and lack of ground texture (such as solid color ground or overly complex texture) will have a negative impact on the accuracy of the visual system. These factors may lead to a decrease in the quality of image capture, thereby affecting the ability to accurately locate and land; although drones are equipped with visual guidance systems, it is still a challenge to accurately detect and avoid these obstacles in complex environments, especially in the presence of dynamic obstacles (such as other moving drones, birds or crowds) and static obstacles (such as wires, trees or buildings); visual guidance systems require extremely high accuracy and stability when processing images for positioning and navigation. However, maintaining high-precision visual recognition and position will be affected when moving quickly or flying in front of complex backgrounds. Summary of the invention

[0003] In order to solve the above problems existing in the prior art, the present invention provides a method for accurate landing of a UAV under visual guidance;

[0004] The purpose of the present invention can be achieved through the following technical solutions:

[0005] S1: Capturing surrounding environment information through the camera of the drone to obtain operating environment data, obtaining physical property data, power system data, and control system data through the construction parameters of the drone itself, and constructing a flight control model of the drone based on the operating environment data, the physical property data, the power system data, and the control system data;

[0006] S2: setting a desired landing point, extracting a target image from corresponding operating environment data according to the desired landing point, and establishing a visual mapping between the UAV and the desired landing point based on the target image;

[0007] S3: obtaining relative position data and a distance to be flown of the UAV according to the visual mapping calculation, and obtaining a predicted running trajectory based on the relative position data and the distance to be flown through the flight control model;

[0008] S4: Automatically capture and track the expected landing point during the landing process, extract feature information from the target image acquired in real time, combine the UAV's own posture information, calculate the relative posture information between the UAV and the expected landing point in real time, and adjust the UAV flight parameters in real time according to the relative posture information and the predicted running trajectory.

[0009] Specifically, the flight control model is used to determine the flight trajectory curve, and obtain the longitudinal motion parameters of the UAV according to the operating environment data, the physical property data, the power system data, and the control system data. The state quantity, control quantity and output quantity of the UAV are determined by the longitudinal motion parameters; the state quantity includes the flight speed, aerodynamic coefficient, pitch angular velocity, pitch angle, engine power, and flight altitude under horizontal steady-state flight; the control quantity is the flight speed and flight altitude under real-time operating conditions; and the output quantity is the motor power supply and elevator deflection.

[0010] Specifically, the S2 specifically includes:

[0011] Identify the coded mark from the target image and obtain the identified pixel coordinates of the coded mark; calculate the rotation matrix and translation matrix of the center of the expected landing point relative to the camera center according to the known coordinate values ​​of the coded mark corners on the expected landing point relative to the landing center; obtain the calculated pixel coordinates of the coded mark in the image through the rotation matrix and translation matrix and the coordinate value of the center of the coded mark in the landing point center coordinate system; subtract the calculated pixel coordinates from the identified pixel coordinates to obtain the pixel difference; calculate the relative position of the drone and the landing point according to the pixel difference and the drone body coordinate system and the geodetic coordinate system of the landing point, and the calculation formula is:

[0012] P xyz =C[W[T T T y T z ]+(1-W)[I x I y I z ]]+P insxyz ,

[0013] Among them, P xyz is the position of the landing point relative to the earth coordinate system of the UAV, C is the rotation matrix from the UAV body coordinate system to the earth coordinate system, W is the weight for adjusting the visual relative positioning data and the inertial navigation positioning data, [T x T y T z ] is the relative visual position, [I x I y I z ] is the inertial navigation positioning data, P insxyzIt is the error bias data of the camera lens relative to the drone body coordinate system.

[0014] Specifically, the predicted running trajectory is calculated by the visual mapping to obtain the relative position data of the UAV and the distance to be flown to construct an objective function. The objective function comprehensively considers the control amount increment and target detection and tracking error of the UAV running smoothly in a limited time domain, and the calculation formula is:

[0015]

[0016] Among them, y ref (k+i|k) is the output reference value, y(k+i|k) is the predicted output, Δu(k+j|k) is the control increment, J(k) is the objective function, i and j are the time counts, k is the current control time, N p is the prediction time domain, N c For the control time domain, Q is the output constraint matrix, and R is the constraint matrix of the control increment;

[0017] According to the objective function, the error generated by the prediction tracking is minimized to the maximum extent, and the longitudinal displacement information of the predicted running trajectory is obtained based on the relative position data and the distance to be flown. The calculation formula is:

[0018]

[0019] Among them, pn(i) is the longitudinal displacement data in the time domain, pn is the longitudinal displacement data at the current moment, i is the moment count in the predicted time domain, V is the flight speed under horizontal steady-state flight, ΔT is the system sampling time, and o i is the predicted value of the track angle in the prediction time domain, e i+1 、e i It is the tracking error of the UAV at adjacent moments when executing the planned trajectory;

[0020] The tracking error calculation formula is:

[0021] e i =h i -p i -iV△T sin o,

[0022] Among them, e i is the tracking error of the UAV at time i when executing the planned trajectory, h i is the flight altitude, i is the time count in the prediction time domain, V is the flight speed in horizontal steady-state flight, ΔT is the system sampling time, and o is the predicted value of the track angle in the prediction time domain under the inertial system;

[0023] The target image collected by the drone in real time is filtered and binarized, the edge of the block is detected according to the gradient change of the block pixels, and the adjacent pixels are clustered according to the gradient direction of the block edge to achieve semantic segmentation of the block; the desired landing point graphic is screened out according to the characteristics of the block corner points, and the center point and corner points of the image are determined. The mapping transformation based on the image position of the corner points and the actual corner points of the desired landing point is obtained through the visual mapping relationship, and the relative coordinates and relative attitude angle of the center of the desired landing point graphic in the camera coordinate system are obtained according to the mapping transformation;

[0024] The predicted running trajectory is set according to the longitudinal displacement information and the relative coordinates and relative attitude angle of the center of the expected landing point graphic.

[0025] Specifically, the real-time adjustment method is to automatically adjust the UAV flight parameters by comparing the relative posture information with the difference between the UAV position information corresponding to the predicted running trajectory at the same moment.

[0026] The beneficial effects of the present invention are:

[0027] By calibrating the landing point image multiple times and collecting real-time data during the drone landing process, centimeter-level landing accuracy can be achieved, allowing the drone to land accurately in complex environments without signal interference or obstruction; by setting the landing point in advance and then performing trajectory calculations through a visually guided precise landing method, it is highly autonomous and does not rely on external signals. At the same time, it ensures that the drone avoids obstacles during landing, greatly improving flight safety. By establishing visual mapping, the calculation data source is suitable for drones of different scenarios and types, with good universality and easy operation. The drone can start autonomous landing by setting the landing point through a simple interface; by setting the predicted running trajectory and taking into account the differences in tracking requirements for different track segments during terrain following flight, the time domain parameter design is adjusted in real time according to the scenario to meet the tracking control requirements for different paths. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0029] Figure 1 The figure is a flow chart of a method for accurate landing of an unmanned aerial vehicle guided by vision according to the present invention. DETAILED DESCRIPTION

[0030] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0031] See also Figure 1 , a method for accurate landing of a UAV under visual guidance, comprising:

[0032] S1: Capturing surrounding environment information through the camera of the drone to obtain operating environment data, obtaining physical property data, power system data, and control system data through the construction parameters of the drone itself, and constructing a flight control model of the drone based on the operating environment data, the physical property data, the power system data, and the control system data;

[0033] S2: setting a desired landing point, extracting a target image from corresponding operating environment data according to the desired landing point, and establishing a visual mapping between the UAV and the desired landing point based on the target image;

[0034] S3: obtaining relative position data and a distance to be flown of the UAV according to the visual mapping calculation, and obtaining a predicted running trajectory based on the relative position data and the distance to be flown through the flight control model;

[0035] S4: Automatically capture and track the expected landing point during the landing process, extract feature information from the target image acquired in real time, combine the UAV's own posture information, calculate the relative posture information between the UAV and the expected landing point in real time, and adjust the UAV flight parameters in real time according to the relative posture information and the predicted running trajectory.

[0036] Specifically, the flight control model is used to determine the flight trajectory curve, and obtain the longitudinal motion parameters of the UAV according to the operating environment data, the physical property data, the power system data, and the control system data. The state quantity, control quantity and output quantity of the UAV are determined by the longitudinal motion parameters; the state quantity includes the flight speed, aerodynamic coefficient, pitch angular velocity, pitch angle, engine power, and flight altitude under horizontal steady-state flight; the control quantity is the flight speed and flight altitude under real-time operating conditions; and the output quantity is the motor power supply and elevator deflection.

[0037] In this embodiment, a camera device and an image acquisition card are embedded inside the drone, and an image acquisition interval is set in the acquisition card. Under the synergistic effect of the two, a color image acquisition signal is obtained, and on this basis, the image is enhanced to improve the clarity of the image, and the image can also be better processed by computer. Select the Gaussian filter processing method, and select the window size of 3×3. When performing template operation on the image boundary, in order to retain the necessary image information, the boundary is repeatedly expanded as needed; according to the real-time image processing results obtained by visual guidance, landmarks, obstacles and flight runways are identified. The purpose of landmark recognition is to determine the target landing position of the drone; obstacle detection is to ensure that there are no obstacles on the planned landing path; flight runway detection is to determine the constraints of the drone landing path. Extract the contour features in the real-time image, extract all the contour sequences, first judge and screen the contour sequences, find the candidate contours of the landmarks, and then judge them according to the designed landmark features. In this process, the contours of obvious non-landmarks are first eliminated, and then each contour is traversed. According to the shape of the set landmark, it is judged whether the extracted contour shape is consistent with the set landmark shape, and the inconsistent contour features are eliminated. Finally, the area formed by the contour features is calculated. If the error between the solved area and the set landmark area is less than 5%, it is determined that the image represents the recognition result of the landmark target, and the position is continuously tracked. Similarly, the detection and recognition results of obstacles and flight runways can also be obtained.

[0038] Specifically, the S2 specifically includes:

[0039] Identify the coded mark from the target image and obtain the identified pixel coordinates of the coded mark; calculate the rotation matrix and translation matrix of the center of the expected landing point relative to the camera center according to the known coordinate values ​​of the coded mark corners on the expected landing point relative to the landing center; obtain the calculated pixel coordinates of the coded mark in the image through the rotation matrix and translation matrix and the coordinate value of the center of the coded mark in the landing point center coordinate system; subtract the calculated pixel coordinates from the identified pixel coordinates to obtain the pixel difference; calculate the relative position of the drone and the landing point according to the pixel difference and the drone body coordinate system and the geodetic coordinate system of the landing point, and the calculation formula is:

[0040] P xyz =C[W[T x T y T z ]+(1-W)[I x I y I z ]]+P insxyz ,

[0041] Among them, P xyzis the position of the landing point relative to the earth coordinate system of the UAV, C is the rotation matrix from the UAV body coordinate system to the earth coordinate system, W is the weight for adjusting the visual relative positioning data and the inertial navigation positioning data, [T x T y T z ] is the relative visual position, [I x I y I z ] is the inertial navigation positioning data, P insxyz It is the error bias data of the camera lens relative to the drone body coordinate system.

[0042] Specifically, the predicted running trajectory is calculated by the visual mapping to obtain the relative position data of the UAV and the distance to be flown to construct an objective function. The objective function comprehensively considers the control amount increment and target detection and tracking error of the UAV running smoothly in a limited time domain, and the calculation formula is:

[0043]

[0044] Among them, y ref (k+i|k) is the output reference value, y(k+i|k) is the predicted output, Δu(k+j|k) is the control increment, J(k) is the objective function, i and j are the time counts, k is the current control time, N p is the prediction time domain, N c For the control time domain, Q is the output constraint matrix, and R is the constraint matrix of the control increment;

[0045] According to the objective function, the error generated by the prediction tracking is minimized to the maximum extent, and the longitudinal displacement information of the predicted running trajectory is obtained based on the relative position data and the distance to be flown. The calculation formula is:

[0046]

[0047] Among them, pn(i) is the longitudinal displacement data in the time domain, pn is the longitudinal displacement data at the current moment, i is the moment count in the predicted time domain, V is the flight speed under horizontal steady-state flight, ΔT is the system sampling time, and o i is the predicted value of the track angle in the prediction time domain, e i +1, e i It is the tracking error of the UAV at adjacent moments when executing the planned trajectory;

[0048] The tracking error calculation formula is:

[0049] e i =h i -p i -iV△T sin o,

[0050] Among them, e i is the tracking error of the UAV at time i when executing the planned trajectory, h i is the flight altitude, i is the time count in the prediction time domain, V is the flight speed in horizontal steady-state flight, ΔT is the system sampling time, and o is the predicted value of the track angle in the prediction time domain under the inertial system;

[0051] The target image collected by the drone in real time is filtered and binarized, the edge of the block is detected according to the gradient change of the block pixels, and the adjacent pixels are clustered according to the gradient direction of the block edge to achieve semantic segmentation of the block; the desired landing point graphic is screened out according to the characteristics of the block corner points, and the center point and corner points of the image are determined. The mapping transformation based on the image position of the corner points and the actual corner points of the desired landing point is obtained through the visual mapping relationship, and the relative coordinates and relative attitude angle of the center of the desired landing point graphic in the camera coordinate system are obtained according to the mapping transformation;

[0052] The predicted running trajectory is set according to the longitudinal displacement information and the relative coordinates and relative attitude angle of the center of the expected landing point graphic.

[0053] In this embodiment, a strategy of using digital map information to complete the terrain following task is used to achieve precise position control in two-dimensional space, and time domain parameters are used as the main optimization decision variables of the design, and real-time adjustments are made according to the system evolution state and operation trend; an MPC controller that accurately tracks the reference track is used, and visual navigation data is used to assist landing; the operator sends the UAV return command through the ground station, and the UAV debugs the maximum return speed parameters during the return process. After the UAV enters the hovering stage and obtains the cloud measurement operation trajectory, the UAV continues to return and land through the visual assisted navigation equipment, and the landing position and take-off position deviation of the UAV are recorded in real time. During the landing process, the UAV autonomously switches the positioning data and adjusts the control quantity data according to the real-time data output, and adjusts the descent speed according to the adjusted control quantity data, and finally completes the landing.

[0054] Specifically, the real-time adjustment method is to automatically adjust the UAV flight parameters by comparing the relative posture information with the difference between the UAV position information corresponding to the predicted running trajectory at the same moment.

[0055] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for accurate landing of an unmanned aerial vehicle under visual guidance, characterized in that: include: S1: Capturing surrounding environment information through the camera of the drone to obtain operating environment data, obtaining physical property data, power system data, and control system data through the construction parameters of the drone itself, and constructing a flight control model of the drone based on the operating environment data, the physical property data, the power system data, and the control system data; S2: setting a desired landing point, extracting a target image from corresponding operating environment data according to the desired landing point, and establishing a visual mapping between the UAV and the desired landing point based on the target image; S3: obtaining relative position data and a distance to be flown of the UAV according to the visual mapping calculation, and obtaining a predicted running trajectory based on the relative position data and the distance to be flown through the flight control model; S4: Automatically capture and track the expected landing point during the landing process, extract feature information from the target image acquired in real time, combine the UAV's own posture information, calculate the relative posture information between the UAV and the expected landing point in real time, and adjust the UAV flight parameters in real time according to the relative posture information and the predicted running trajectory.

2. The method for accurate landing of an unmanned aerial vehicle under visual guidance according to claim 1, characterized in that: The flight control model is used to determine the flight trajectory curve, and the longitudinal motion parameters of the UAV are obtained according to the operating environment data, the physical property data, the power system data, and the control system data. The state quantity, control quantity and output quantity of the UAV are determined by the longitudinal motion parameters; the state quantity includes the flight speed, aerodynamic coefficient, pitch angular velocity, pitch angle, engine power, and flight altitude under horizontal steady-state flight, the control quantity is the flight speed and flight altitude under real-time operating conditions, and the output quantity is the motor power supply and elevator deflection.

3. The method for accurate landing of an unmanned aerial vehicle under visual guidance according to claim 1, characterized in that: The S2 specifically includes: Identify the coded mark from the target image and obtain the identified pixel coordinates of the coded mark; calculate the rotation matrix and translation matrix of the center of the expected landing point relative to the camera center according to the known coordinate values ​​of the coded mark corners on the expected landing point relative to the landing center; obtain the calculated pixel coordinates of the coded mark in the image through the rotation matrix and translation matrix and the coordinate value of the center of the coded mark in the landing point center coordinate system; subtract the calculated pixel coordinates from the identified pixel coordinates to obtain the pixel difference; calculate the relative position of the drone and the landing point according to the pixel difference and the drone body coordinate system and the geodetic coordinate system of the landing point, and the calculation formula is: P xyz =C[W[T x T y T z ]+(1-W)[I x I y I z ]]+P insxyz , Among them, P xyz is the position of the landing point relative to the earth coordinate system of the UAV, C is the rotation matrix from the UAV body coordinate system to the earth coordinate system, W is the weight for adjusting the visual relative positioning data and the inertial navigation positioning data, [T x T y T z ] is the relative visual position, [I x I y I z ] is the inertial navigation positioning data, P insxyz It is the error bias data of the camera lens relative to the drone body coordinate system.

4. A method for accurate landing of an unmanned aerial vehicle under visual guidance according to claim 1 or 2, characterized in that: The predicted running trajectory is calculated by the visual mapping to obtain the relative position data of the UAV and the distance to be flown to construct an objective function. The objective function comprehensively considers the control amount increment and target detection and tracking error of the UAV running smoothly in a limited time domain. The calculation formula is: Among them, y ref (k+i|k) is the output reference value, y(k+i|k) is the predicted output, Δu(k+j|k) is the control increment, J(k) is the objective function, i and j are the time counts, k is the current control time, N p is the prediction time domain, N c For the control time domain, Q is the output constraint matrix, and R is the constraint matrix of the control increment; According to the objective function, the error generated by the prediction tracking is minimized to the maximum extent, and the longitudinal displacement information of the predicted running trajectory is obtained based on the relative position data and the distance to be flown. The calculation formula is: Where pn(i) is the longitudinal displacement data in the time domain, pn is the longitudinal displacement data at the current moment, i is the time count in the prediction time domain, V is the flight speed in horizontal steady-state flight, ΔT is the system sampling time, and o i is the predicted value of the track angle in the prediction time domain, e i+1 、e i It is the tracking error of the UAV at adjacent moments when executing the planned trajectory; The tracking error calculation formula is: e i =h i -p i -iVΔT sin o, Among them, e i is the tracking error of the UAV at time i when executing the planned trajectory, h i is the flight altitude, i is the time count in the prediction time domain, V is the flight speed in horizontal steady-state flight, ΔT is the system sampling time, and o is the predicted value of the track angle in the prediction time domain under the inertial system; The target image collected by the drone in real time is filtered and binarized, the edge of the block is detected according to the gradient change of the block pixels, and the adjacent pixels are clustered according to the gradient direction of the block edge to achieve semantic segmentation of the block; the desired landing point graphic is screened out according to the characteristics of the block corner points, and the center point and corner points of the image are determined. The mapping transformation based on the image position of the corner points and the actual corner points of the desired landing point is obtained through the visual mapping relationship, and the relative coordinates and relative attitude angle of the center of the desired landing point graphic in the camera coordinate system are obtained according to the mapping transformation; The predicted running trajectory is set according to the longitudinal displacement information and the relative coordinates and relative attitude angle of the center of the expected landing point graphic.

5. The method for accurate landing of an unmanned aerial vehicle under visual guidance according to claim 1, characterized in that: The real-time adjustment method is: automatically adjusting the UAV flight parameters by comparing the relative posture information with the difference between the UAV position information corresponding to the predicted running trajectory at the same moment.

6. The method for accurate landing of an unmanned aerial vehicle under visual guidance according to claim 1, characterized in that: The automatic capture method is as follows: a flight control observer is installed on a ground controller, a visual target positioning module is installed on the flight control observer, the visual target positioning module is used to receive a flight control model sent by a model prediction control module to control the ground observer for real-time tracking, and the model prediction control module is used to set a static flight trajectory of the flight control model.

Images