Unmanned aerial vehicle fixed-point landing method based on vision

Abstract

The invention belongs to the technical field of four-rotor unmanned aerial vehicles, and particularly relates to a vision-based unmanned aerial vehicle fixed-point landing method, which comprises the following steps that S1, an unmanned aerial vehicle captures preliminary data of flight height through an ultrasonic sensor, a laser scanner and a depth sensing camera of the unmanned aerial vehicle, and the preliminary data is used for detecting the height of a vehicle body; performing filtering processing on the data by using a data integration algorithm to generate accurate height information which serves as a basis for selection of a visual servo method and also serves as a basis for calculation based on an image Jacobian matrix in image visual servo; s2, describing image features and image coordinates of the unmanned aerial vehicle; s3, according to the height of the unmanned aerial vehicle, a visual servo method is selected, and the height of the unmanned aerial vehicle is obtained through a height detection sensor. According to the invention, position-based visual servo can be used when the height is high, and image-based visual servo can be used for positioning when the height is low, so that the design of a controller for fixed-point landing of an unmanned aerial vehicle based on different visual servo modes is completed.

Description

Technical Field

[0001] The invention belongs to the technical field of quad-rotor unmanned aerial vehicles, and in particular relates to a vision-based fixed-point landing method for unmanned aerial vehicles. Background Art

[0002] Quadcopter drones are widely used in target detection and tracking when performing complex tasks in the scientific research field due to their excellent reconnaissance performance, stable flight characteristics, simple structural design, vertical take-off and landing capabilities, low production cost, and portability. Despite the continuous development of technology, existing drones still face some challenges and problems in target detection and tracking.

[0003] Existing target tracking technologies are mainly divided into two categories: image-based visual servoing and position-based visual servoing. Although position-based visual servoing technologies dominate the industry due to their maturity and stability, this approach also has obvious limitations. They usually rely on precise position sensors to achieve control, and the performance of these sensors may be affected by noise interference, measurement drift, or insufficient accuracy. In addition, position-based visual servoing technologies often fail to capture the appearance characteristics of the target, which may become their shortcoming when dealing with diverse targets, complex environments, or scenarios that require advanced perception capabilities. They are also relatively weak in adaptability to dynamic environments and target occlusion, because these methods mainly focus on the spatial position of the target and ignore the motion characteristics of the target and changes in visual features.

[0004] Therefore, although the position-based visual servo technology still has its value in specific application scenarios, its limitations also highlight the necessity of image-based visual servo methods. When a more comprehensive, flexible and intelligent control strategy is required, the image-based visual servo method shows its unique advantages, especially in the face of complex tracking tasks that require high adaptability and intelligent judgment.

[0005] Many existing methods of fixed-point landing of drones through visual servoing often take into account that the camera may lose the target during the landing process, or that the drone may be disturbed by wind. The first is the disturbance problem of visual servoing. Visual fixed-point landing depends heavily on whether the detected target is stable and accurate. However, due to light interference and the complexity of the landing environment, visual fixed-point landing is often affected, thus affecting accurate landing. In addition, the insufficient performance of the altitude sensor during the landing process affects the estimation of the most critical image Jacobian matrix in visual servo fixed-point.

[0006] In addition, through patent search, it was found that the publication number is: CN116149193A, a vision-based anti-disturbance control method and system for rotary-wing UAV. The control system did not consider the image detection stability problem of the image-based positioning method, nor did it adjust the visual servo method according to the altitude, which put a lot of pressure on the controller. Summary of the invention

[0007] The purpose of the present invention is to provide a vision-based UAV fixed-point landing method, which can use position-based visual servoing at a high altitude and image-based visual servoing for positioning at a low altitude, and complete the design of a controller for fixed-point landing of a UAV based on different visual servoing modes.

[0008] The technical solution adopted by the present invention is as follows:

[0009] A method for fixed-point landing of an unmanned aerial vehicle based on vision, the method comprising the following steps:

[0010] S1: The drone captures preliminary data of flight altitude through the ultrasonic sensor, laser scanner and depth perception camera equipped to detect the height of the aircraft;

[0011] In S1, a data integration algorithm is used to perform filtering processing on the data to generate accurate height information, which serves as a basis for selecting a visual servoing method and also as a basis for calculating the image Jacobian matrix in image-based visual servoing.

[0012] S2: Describe the features and image coordinates of the drone image;

[0013] In S2, a camera coordinate system C aligned with the drone coordinate system is first defined, and the image captured by the camera on the drone is coupled in the camera coordinate system C, and a virtual plane is assumed to decouple it;

[0014] The origin of the virtual camera is the same as that of the real camera. The virtual camera plane is always parallel to the ground. The roll angle and pitch angle of the virtual camera plane are always zero. The yaw angle is the same as the yaw angle in the actual camera plane. The point features on the camera image plane are projected onto the virtual image plane to establish the feature dynamics on the virtual image plane. The real-time image obtained by the drone through the camera is coupled in the real plane of the camera. The roll angle and pitch angle of the quadrotor are obtained through the IMU, and then reprojected to the virtual plane set at the beginning. Before the reprojection conversion, the roll angle and pitch angle will affect the image points, but will not affect the entire system. The image point features need to be converted into image moment features, and the error of the image moment is used for visual servo control.

[0015] The drone uses the downward-looking camera to capture a point in space. E P,E The coordinates of point P in the camera coordinate system C P=[ c x c y c z] T , the coordinates in the virtual camera coordinate system V P=[ V x V y V z] T ;

[0016] E The coordinates of point P in the camera coordinate system and the coordinates under the virtual camera can be expressed in combination with the earth coordinate system:

[0017]

[0018] In the formula, O c (t),O v (t) are the coordinates of the camera origin and the virtual camera origin in the earth coordinate system, R T (t), is the corresponding rotation matrix; according to the perspective projection model, the two-dimensional coordinates on the camera plane are (u,n) = (λ c x / c z,λ c y / c z); Similarly, the two-dimensional position of point P on the virtual camera plane is expressed as ( V u, V n)=(λ V x / V z,λ V y / V z), λ is the focal length of the camera; the coordinates of the moving point on the virtual image plane are obtained by rotating the coordinates on the actual image plane;

[0019] V Taking the derivative of P with respect to time, we get:

[0020]

[0021] In the formula, v = [v x v y v z ] T is the linear velocity of the body in the virtual camera coordinate system; ( V u, V n) Take the derivative with respect to time and then substitute it into equation (3) to obtain:

[0022]

[0023] It can be seen from the formula that the motion change rate of the pixel points in the virtual plane has nothing to do with the change rate of the roll angle and pitch angle of the drone, and the linear speed of the drone is related to the yaw angle.

[0024] S3: Select the visual servo method according to the height of the drone, and the height of the drone is obtained by using a height detection sensor;

[0025] In S3, when the altitude of the drone is relatively high, a position-based visual servo calculation method with poor accuracy is used to reduce the output pressure of the controller;

[0026] When the drone is at a relatively low altitude, an image-based visual servoing calculation method is used to ensure accurate landing.

[0027] When using position-based visual servoing, the desired Cartesian position of the input controller is a relative position, defined as E pos = * P pos -P pos , from the first step, we can know that * P pos The current Cartesian coordinate system coordinates of the drone measured by the camera, P pos The Cartesian coordinates of the target landing point of the UAV are measured by the camera, and the error in the above formula is input into the controller as the desired Cartesian coordinates of the position-based visual servoing;

[0028] When using image-based visual servoing, the input to the controller is the desired image position, which is also a relative position, E img = * P img -P img , where * P img is the pixel coordinate of the drone under the camera plane, P img The pixel coordinates of the target landing point of the UAV are measured by the camera, and the error in the above formula is input into the controller as the desired Cartesian coordinates of the image-based visual servoing.

[0029] S4: Computation of the visual servo output structure using the control ball.

[0030] In S4, the visual servo output result is put into the controller for anti-disturbance control, and the system model is established:

[0031]

[0032] Design a differential tracker based on the system model:

[0033]

[0034] Where: r is the upper limit of tracking acceleration, satisfying and

[0035]

[0036] The output signal x1(t) of the differential tracker tracks the input signal of the system * P(t), the input target position has a transition process, and the input target signal does not have high tracking requirements for the controller like a pulse signal;

[0037] Then, the extended state observer is designed according to the system to estimate the uncertainty of the model and the comprehensive effect of external interference, namely:

[0038]

[0039] Among them, B1 and B2 are optional parameters for estimating performance; Z1(t) gives an estimate of P(t), and Z2(t) gives an estimate of A(t), which is the combined effect of the system's unmodeled dynamics and external disturbances.

[0040] Design nonlinear state error feedback control and define the system tracking error as E(t) = P * The control input of the Z(t)-Z1(t) system can be obtained by the nonlinear state error feedback of the following formula:

[0041]

[0042] The controller obtains the Cartesian speed and image speed required by the visual servo to reach the target position. When using position-based visual servoing, this output can act on the drone to control the linear speed. When using image-based visual servoing, an additional step is required to convert the speed in the image coordinate system to the linear speed of the body in the Cartesian coordinate system.

[0043] make Then the linear speed required for the UAV to reach the target position can be expressed as

[0044]

[0045] The technical effects achieved by the present invention are:

[0046] A vision-based UAV fixed-point landing method of the present invention improves the change of target input quantity by controlling the UAV under different visual servo controls using a differential tracker, improves control accuracy using nonlinear state feedback, and estimates state quantities using a nonlinear extended state observer, thereby improving system robustness.

[0047] Compared with the existing drone landing method, the vision-based fixed-point landing method of the present invention is more targeted at complex ground conditions, ensuring that the drone can land more safely, and can adjust the landing strategy as needed, dynamically adjust the landing strategy according to different ground conditions, and change the PID parameters at the same time. It is more flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a schematic diagram of the distribution structure of the virtual plane according to an embodiment of the present invention;

[0049] Figure 2 is a flowchart of obtaining the altitude of a drone according to an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of the coordinate system relationship when visual servo control is used in an embodiment of the present invention;

[0051] Figure 4 It is a flowchart of a controller according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is specifically described below in conjunction with embodiments. It should be understood that the following text is only used to describe one or several specific embodiments of the present invention, and does not strictly limit the scope of protection of the specific claims of the present invention.

[0053] like Figure 1 As shown, a vision-based fixed-point landing method for a UAV comprises the following steps:

[0054] S1: The drone captures preliminary data of flight altitude through the ultrasonic sensor, laser scanner and depth perception camera equipped to detect the height of the aircraft;

[0055] In S1, a data integration algorithm is used to perform filtering processing on the data to generate accurate height information, which serves as a basis for selecting a visual servoing method and also as a basis for calculating the image Jacobian matrix in image-based visual servoing.

[0056] S2: Describe the features and image coordinates of the drone image;

[0057] In S2, a camera coordinate system C aligned with the drone coordinate system is first defined, and the image captured by the camera on the drone is coupled in the camera coordinate system C, and a virtual plane is assumed to decouple it;

[0058] The origin of the virtual camera is the same as that of the real camera. The virtual camera plane is always parallel to the ground. The roll angle and pitch angle of the virtual camera plane are always zero. The yaw angle is the same as the yaw angle in the actual camera plane. The point features on the camera image plane are projected onto the virtual image plane to establish the feature dynamics on the virtual image plane. The real-time image obtained by the drone through the camera is coupled in the real plane of the camera. The roll angle and pitch angle of the quadrotor are obtained through the IMU, and then reprojected to the virtual plane set at the beginning. Before the reprojection conversion, the roll angle and pitch angle will affect the image points, but will not affect the entire system. The image point features need to be converted into image moment features, and the error of the image moment is used for visual servo control.

[0059] The drone uses the downward-looking camera to capture a point in space. E P, E The coordinates of point P in the camera coordinate system C P=[ c x c y c z] T , the coordinates in the virtual camera coordinate system V P=[ V x V y V z] T ;

[0060] E The coordinates of point P in the camera coordinate system and the coordinates under the virtual camera can be expressed in combination with the earth coordinate system:

[0061]

[0062] In the formula, O c (t),O v (t) are the coordinates of the camera origin and the virtual camera origin in the earth coordinate system, R T (t), is the corresponding rotation matrix; according to the perspective projection model, the two-dimensional coordinates on the camera plane are (u,n) = (λ c x / c z,λ c y / c z); Similarly, the two-dimensional position of point P on the virtual camera plane is expressed as ( V u, V n)=(λ V x / V z,λ V y / V z), λ is the focal length of the camera; the coordinates of the moving point on the virtual image plane are obtained by rotating the coordinates on the actual image plane;

[0063] V Taking the derivative of P with respect to time, we get:

[0064]

[0065] In the formula, v = [v x v y v z ] T is the linear velocity of the body in the virtual camera coordinate system; ( V u, V n) Take the derivative with respect to time and then substitute it into equation (3) to obtain:

[0066]

[0067] It can be seen from the formula that the motion change rate of the pixel points in the virtual plane has nothing to do with the change rate of the roll angle and pitch angle of the drone, and the linear speed of the drone is related to the yaw angle.

[0068] S3: Select the visual servo method according to the height of the drone, and the height of the drone is obtained by using a height detection sensor;

[0069] In S3, when the altitude of the drone is relatively high, a position-based visual servo calculation method with poor accuracy is used to reduce the output pressure of the controller;

[0070] When the drone is at a relatively low altitude, an image-based visual servoing calculation method is used to ensure accurate landing.

[0071] When using position-based visual servoing, the desired Cartesian position of the input controller is a relative position, defined as E pos = * P pos -P pos , from the first step, we can know that * P pos The current Cartesian coordinate system coordinates of the drone measured by the camera, P pos The Cartesian coordinates of the target landing point of the UAV are measured by the camera, and the error in the above formula is input into the controller as the desired Cartesian coordinates of the position-based visual servoing;

[0072] When using image-based visual servoing, the input to the controller is the desired image position, which is also a relative position, E img = * P img -P img , where * P img is the pixel coordinate of the drone under the camera plane, P imgThe pixel coordinates of the target landing point of the UAV are measured by the camera, and the error in the above formula is input into the controller as the desired Cartesian coordinates of the image-based visual servoing.

[0073] S4: Computation of the visual servo output structure using the control ball.

[0074] In S4, the visual servo output result is put into the controller for anti-disturbance control, and the system model is established:

[0075]

[0076] Design a differential tracker based on the system model:

[0077]

[0078] Where: r is the upper limit of tracking acceleration, satisfying and

[0079]

[0080] The output signal x1(t) of the differential tracker tracks the input signal of the system * P(t), the input target position has a transition process, and the input target signal does not have high tracking requirements for the controller like a pulse signal;

[0081] Then, the extended state observer is designed according to the system to estimate the uncertainty of the model and the comprehensive effect of external interference, namely:

[0082]

[0083] Among them, B1 and B2 are optional parameters for estimating performance; Z1(t) gives an estimate of P(t), and Z2(t) gives an estimate of A(t), which is the combined effect of the system's unmodeled dynamics and external disturbances.

[0084] Design nonlinear state error feedback control and define the system tracking error as E(t) = P * The control input of the Z(t)-Z1(t) system can be obtained by the nonlinear state error feedback of the following formula:

[0085]

[0086] The controller obtains the Cartesian speed and image speed required for the visual servo to reach the target position. When using position-based visual servoing, the output can act on the drone to control the linear speed. When using image-based visual servoing, an additional step is required to convert the speed in the image coordinate system to the linear speed of the body in the Cartesian coordinate system; Formula 3 is used for calculation:

[0087] make Then the linear speed required for the UAV to reach the target position can be expressed as

[0088]

[0089] This scheme uses position-based visual servoing at high altitudes and image-based visual servoing for positioning at low altitudes. It uses differential trackers, nonlinear state sensors, and nonlinear state error feedback control to complete the design of a controller for fixed-point landing of UAVs based on different visual servoing methods.

[0090] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A vision-based UAV fixed-point landing method, characterized by: The method comprises the following steps: S1: The drone captures preliminary data of flight altitude through the ultrasonic sensor, laser scanner and depth perception camera equipped to detect the height of the aircraft; S2: Describe the UAV image features and image coordinates; S3: Select the visual servo method according to the height of the drone, and the height of the drone is obtained by using a height detection sensor; S4: Computation of the visual servo output structure using the control ball.

2. The method for fixed-point landing of a UAV based on vision according to claim 1, characterized in that: In S1, a data integration algorithm is used to perform filtering processing on the data to generate accurate height information, which serves as a basis for selecting a visual servoing method and also as a basis for calculating the image Jacobian matrix in image-based visual servoing.

3. The vision-based fixed-point landing method for unmanned aerial vehicles according to claim 1, characterized in that: In S2, a camera coordinate system C aligned with the drone coordinate system is first defined, and the image captured by the camera on the drone is coupled in the camera coordinate system C, and a virtual plane is assumed to decouple it; The origin of the virtual camera is the same as that of the real camera. The virtual camera plane is always parallel to the ground. The roll angle and pitch angle of the virtual camera plane are always zero. The yaw angle is the same as the yaw angle in the actual camera plane. The point features on the camera image plane are projected onto the virtual image plane to establish the feature dynamics on the virtual image plane. The real-time image obtained by the drone through the camera is coupled in the real plane of the camera. The roll angle and pitch angle of the quadrotor are obtained through the IMU, and then reprojected to the virtual plane set at the beginning. Before the reprojection conversion, the roll angle and pitch angle will affect the image points, but will not affect the entire system. The image point features need to be converted into image moment features, and the error of the image moment is used for visual servo control.

4. The vision-based fixed-point landing method for unmanned aerial vehicles according to claim 3, characterized in that: The drone uses the downward-looking camera on board to capture a point in space. E P, E The coordinates of point P in the camera coordinate system C P=[ c x c y c z] T , the coordinates in the virtual camera coordinate system V P=[ V x V y V z] T ; E The coordinates of point P in the camera coordinate system and the coordinates under the virtual camera can be expressed in combination with the earth coordinate system: In the formula, O c (t),O v (t) are the coordinates of the camera origin and the virtual camera origin in the earth coordinate system, R T (t), is the corresponding rotation matrix; according to the perspective projection model, the two-dimensional coordinates on the camera plane are (u,n) = (λ c x / c z,λ c y / c z); Similarly, the two-dimensional position of point P on the virtual camera plane is expressed as ( V u, V n)=(λ V x / V z,λ V y / V z), λ is the focal length of the camera; the coordinates of the moving point on the virtual image plane are obtained by rotating the coordinates on the actual image plane; V Taking the derivative of P with respect to time, we get: In the formula, v = [v x v y v z ] T is the linear velocity of the body in the virtual camera coordinate system; ( V u, V n) Taking the derivative with respect to time, we get: It can be seen from the formula that the motion change rate of the pixel points in the virtual plane has nothing to do with the change rate of the roll angle and pitch angle of the drone, and the linear speed of the drone is related to the yaw angle.

5. The vision-based fixed-point landing method for unmanned aerial vehicles according to claim 1, characterized in that: In S3, when the altitude of the drone is relatively high, a position-based visual servo calculation method with poor accuracy is used to reduce the output pressure of the controller; When the drone is at a relatively low altitude, an image-based visual servoing calculation method is used to ensure accurate landing.

6. The vision-based fixed-point landing method for unmanned aerial vehicles according to claim 5, characterized in that: When using position-based visual servoing, the desired Cartesian position of the input controller is a relative position, defined as E pos = * P pos -P pos , from the first step, we can know that * P pos The current Cartesian coordinate system coordinates of the drone measured by the camera, P pos The Cartesian coordinates of the target landing point of the UAV are measured by the camera, and the error in the above formula is input into the controller as the desired Cartesian coordinates of the position-based visual servoing; When using image-based visual servoing, the input to the controller is the desired image position, which is also a relative position, E img = * P img -P img , where * P img is the pixel coordinate of the drone under the camera plane, P img The pixel coordinates of the target landing point of the UAV are measured by the camera, and the error in the above formula is input into the controller as the desired Cartesian coordinates of the image-based visual servoing.

7. The vision-based fixed-point landing method for unmanned aerial vehicles according to claim 1, characterized in that: In S4, the visual servo output result is put into the controller for anti-disturbance control, and the system model is established: Design a differential tracker based on the system model: Where: r is the upper limit of tracking acceleration, satisfying and The output signal x1(t) of the differential tracker tracks the input signal of the system * P(t), the input target position has a transition process, and the input target signal does not have high tracking requirements for the controller like a pulse signal; Then, the extended state observer is designed according to the system to estimate the uncertainty of the model and the comprehensive effect of external interference, namely: Among them, B1 and B2 are optional parameters for estimating performance; Z1(t) gives an estimate of P(t), and Z2(t) gives an estimate of A(t), which is the combined effect of the system's unmodeled dynamics and external disturbances.

8. The vision-based fixed-point landing method for unmanned aerial vehicles according to claim 7, characterized in that: Design nonlinear state error feedback control and define the system tracking error as E(t) = P * The control input of the Z(t)-Z1(t) system can be obtained by the nonlinear state error feedback of the following formula: The controller obtains the Cartesian speed and image speed required by the visual servo to reach the target position. When using position-based visual servoing, this output can act on the drone to control the linear speed. When using image-based visual servoing, an additional step is required to convert the speed in the image coordinate system to the linear speed of the body in the Cartesian coordinate system. make Then the linear speed required for the UAV to reach the target position can be expressed as

Images