Unmanned aerial vehicle visual servo invasion target tracking and intercepting method based on time delay compensation

Abstract

The invention provides an unmanned aerial vehicle visual servo intrusion target tracking and intercepting method based on time delay compensation. The method comprises the following steps: acquiring a latest image of a tracking target through a camera on an unmanned aerial vehicle; based on the latest image, determining the position of the tracking target in an image coordinate system of the latest image as a first position; determining a position error of the first position by adopting a prediction algorithm based on improved Kalman filtering based on the delay time of collecting the latest image; based on the position error, determining a control parameter used for visual servo tracking of the corresponding tracking target; and based on the first position and the control parameter, controlling the unmanned aerial vehicle to fly to track and intercept the corresponding tracking target. Therefore, the position error compensation is performed through the delay time of the latest image, and the unmanned aerial vehicle is controlled to track the corresponding target based on the position error, so that the tracking accuracy is improved, the tracking interception error is reduced, and the interception success rate is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) control, and in particular to a UAV visual servo intrusion target tracking and interception method based on time delay compensation. Background Art

[0002] The widespread use of drones has also brought with it a series of serious security issues. Malicious drone intrusions are frequent, posing a serious threat to public safety and the security of important sites. For example, at large events or around critical infrastructure, unauthorized drones may approach with dangerous objects, conduct illegal reconnaissance, disrupt normal activities, and cause unforeseen losses. During the interception process, intruding drones must be accurately tracked. Currently, drone visual imaging systems generally suffer from imaging delays, which significantly negatively impact control performance in high-speed drone interception scenarios, increasing tracking and interception errors and affecting the success rate of interception. Summary of the Invention

[0003] In view of this, the purpose of the embodiments of the present application is to provide a method for tracking and intercepting intruding targets of unmanned aerial vehicles (UAVs) based on visual servoing and delay compensation, which can improve the problem of increased tracking and interception errors and affected interception success rate.

[0004] To achieve the above technical objectives, the technical solutions adopted in this application are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for tracking and intercepting an intrusion target using a UAV visual servo system based on delay compensation, the method comprising:

[0006] Obtain the latest images of the tracking target through the camera on the drone;

[0007] determining, based on the latest image, a position of the tracking target in an image coordinate system of the latest image as a first position;

[0008] Determining a position error of the first position using a prediction algorithm based on an improved Kalman filter based on a delay time for acquiring the latest image;

[0009] determining control parameters for visual servo tracking of the tracking target based on the position error;

[0010] Based on the first position and the control parameters, the UAV is controlled to fly so as to track and intercept the tracking target.

[0011] In some optional implementations, determining the position error of the first position based on a delay time for acquiring the latest image and using a prediction algorithm based on an improved Kalman filter includes:

[0012] At time kT, the first position z(k) of the tracking target in the latest image obtained is expressed as:

[0013] z(k)=[u(k),n(k)] T

[0014] Where u(k) represents the coordinate value of the u-axis in the pixel coordinate system, n(k) represents the coordinate value of the n-axis in the pixel coordinate system; the observed image at time kT is the measured image obtained at time (kD)T, and DT represents the delay time for acquiring the latest image;

[0015] The observation vector is replaced by the optimal estimate of the system state vector, wherein the observation vector includes the first position of the tracked target in the latest image, and the observation vector satisfies:

[0016]

[0017] Where Z(i) represents the observation vector at time iT, T is the control period; X(i) represents the system state vector at time iT; H represents the measurement matrix;

[0018] recursively predicting the system state vector based on a Kalman filter algorithm, selecting a fitting function using historical data during the recursive prediction process, and obtaining a position error of the first position based on the fitting function;

[0019] The fitting function f() is:

[0020]

[0021] w j represents the polynomial coefficients; represents the predicted position of the tracking target;

[0022] The position error e i for:

[0023]

[0024] The historical data is the latest image of the tracked target obtained within a sliding window with a window size of N;

[0025] The polynomial coefficients are calculated by the least squares method to minimize the sum of squares of the fitting errors, which can be expressed as:

[0026] In some optional embodiments, the position error includes a lateral error and a longitudinal error of the tracking target in the image coordinate system;

[0027] Determining control parameters for visual servo tracking of the tracking target based on the position error includes:

[0028] Determining the yaw angular velocity and roll angular velocity of the UAV in a body coordinate system based on the lateral error;

[0029] According to the longitudinal error, the expected thrust and the expected pitch angular velocity of the UAV are determined; the control parameters include the yaw angular velocity, the roll angular velocity, the expected thrust and the expected pitch angular velocity.

[0030] In some optional embodiments, determining the yaw angular velocity and the roll angular velocity of the UAV in a body coordinate system based on the lateral error includes:

[0031] The yaw angular velocity and the roll angular velocity are determined according to the lateral error using a first preset formula and a second preset formula, wherein the first preset formula is:

[0032]

[0033] Where, represents the yaw angular velocity of the UAV in the body coordinate system; k1 and k2 represent the first control parameter and the second control parameter respectively, e u represents the lateral error;

[0034] The second preset formula is:

[0035]

[0036] Where, represents the roll angular velocity of the UAV in the body coordinate system; k3 and k4 represent the third control parameter and the fourth control parameter respectively; φ represents the roll angle of the UAV.

[0037] In some optional embodiments, determining the expected thrust and expected pitch rate of the UAV based on the longitudinal error includes:

[0038] According to the longitudinal error, the expected thrust and the expected pitch angular velocity are determined by a third preset formula and a fourth preset formula. The third preset formula is:

[0039]

[0040] Where, f d represents the desired thrust; m represents the mass of the UAV; θ represents the pitch angle of the UAV; k5 and k6 represent the fifth control parameter and the sixth control parameter respectively; represents the desired longitudinal velocity of the UAV; en represents the longitudinal error; g represents the acceleration due to gravity;

[0041] The fourth preset formula is:

[0042]

[0043] Where, represents the desired pitch angular velocity; k7 and k8 represent the seventh and eighth control parameters; θ d represents the desired pitch angle;

[0044]

[0045] Where θ forward represents the preset pitch angle for tracking the target forward; y Indicates the focal length coefficient of the camera on the y-axis of the camera coordinate system.

[0046] In some optional embodiments, controlling the flight of the drone to track and intercept the target based on the first position and the control parameter includes:

[0047] Based on a pre-established image Jacobian matrix, the control parameters are converted into control quantities for visual servo control;

[0048] Based on the first position and the control amount, the UAV is controlled to operate so that the tracking target is at the center of the image captured by the camera, and the tracking target is tracked and intercepted.

[0049] In some optional implementations, determining, based on the latest image, a position of the tracking target in an image coordinate system of the latest image as the first position includes:

[0050] The position coordinates of the center point of the image region of the tracking target in the latest image are used as the position of the tracking target in the image coordinate system of the latest image to obtain the first position.

[0051] In some optional embodiments, the method further comprises:

[0052] Based on a pre-established transformation matrix between a world coordinate system and an image coordinate system, converting the first position of the tracking target into a second position in the world coordinate system;

[0053] When the second position of the tracking target is within the preset interception range of the UAV, the interception execution device of the UAV is controlled to launch an interference object toward the tracking target.

[0054] In some optional embodiments, the tracking target includes drones other than the drone itself, and the interference object includes a net.

[0055] The invention adopting the above technical solution has the following advantages:

[0056] In the technical solution provided by this application, the position error of a first position is determined using the delay time of the latest image and a prediction algorithm based on an improved Kalman filter. Then, based on the position error, control parameters for visual servo tracking of the corresponding tracking target are determined. Based on the first position and the control parameters, the drone's flight is controlled to track the corresponding tracking target. In this way, position error compensation is performed using the delay time of the latest image, and the drone is controlled to track the corresponding target based on the position error, thereby improving tracking accuracy, reducing tracking and interception errors, and ultimately increasing interception success rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present application may be further illustrated by the non-limiting embodiments provided in the accompanying drawings. It should be understood that the following drawings illustrate only certain embodiments of the present application and are therefore not to be construed as limiting the scope of the present application. It is understood that a person skilled in the art can derive other relevant drawings from these drawings without inventive effort.

[0058] Figure 1 A flowchart of a method for tracking and intercepting an intruding target using UAV visual servoing based on delay compensation is provided in an embodiment of the present application.

[0059] Figure 2 A schematic diagram of the scene of a drone in the camera coordinate system and the world coordinate system is provided for the embodiment of the present application.

[0060] Figure 3 A schematic diagram of image delay provided in an embodiment of the present application.

[0061] Figure 4A Schematic diagram of pixel information obtained during validity verification based on a conventional Kalman filter.

[0062] Figure 4B Schematic diagram of pixel information obtained during validity verification based on the prediction algorithm in an embodiment of the present application.

[0063] Figure 4C Schematic diagram of the pixel trajectory obtained during validity verification based on the conventional Kalman filter.

[0064] Figure 4D Schematic diagram of the pixel trajectory obtained during the validity verification of the prediction algorithm in the embodiment of the present application.

[0065] Figure 5AThis is a schematic diagram of pixel information obtained during dynamic verification based on the prediction algorithm in an embodiment of the present application.

[0066] Figure 5B Schematic diagram of the pixel trajectory obtained during dynamic verification based on the prediction algorithm in the embodiment of the present application.

[0067] Figure 6A Schematic diagram of the interception trajectory based on the conventional IBVS algorithm.

[0068] Figure 6B Schematic diagram of the interception trajectory of the prediction algorithm in an embodiment of the present application.

[0069] Figure 6C Schematic diagram of the speed of the drone in the embodiment of the present application.

[0070] Figure 6D Schematic diagram of a pixel trajectory map of a drone in an embodiment of the present application.

[0071] Figure 7A This is a schematic diagram of the interception trajectory of the drone using a serpentine maneuver interception in an embodiment of the present application.

[0072] Figure 7B This is a schematic diagram of the speed of the drone using a serpentine maneuver to intercept in an embodiment of the present application.

[0073] Figure 8A Schematic diagram of the interception trajectory of a drone intercepting an arbitrary maneuvering target in an embodiment of the present application.

[0074] Figure 8B This is a schematic diagram of the speed of a drone intercepting an arbitrary maneuvering target in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that similar or identical parts in the drawings or descriptions are numbered the same. Implementations not shown or described in the drawings are known to those of ordinary skill in the art. In the description of this application, the terms "first," "second," etc. are used solely to distinguish descriptions and are not to be construed as indicating or implying relative importance.

[0076] Please refer to Figure 1 As shown in Figure 8, this application provides a method for tracking and intercepting intrusion targets using UAV visual servoing based on time delay compensation, referred to as the tracking and interception method. This method can be applied to UAVs to track corresponding targets, and each step of the method can be executed or implemented by the UAV. The target to be tracked (i.e., the tracking target) can be, but is not limited to, UAV-type aircraft, birds, etc., and can be flexibly determined based on actual conditions.

[0077] The drone includes a frame, a flight controller, a gimbal, an industrial computer, a switch, a wireless image and data transmission device, and an interception execution device.

[0078] In this embodiment, the frame of the drone is made of high-strength lightweight materials (such as carbon fiber and aluminum alloy), which ensures the lightness of the fuselage and the stability of the structure. The drone is equipped with a powerful power system, focusing on load capacity and long endurance performance, while providing good scalability; it uses a high-performance brushless motor and is equipped with a matching propeller to ensure that the drone can remain stable during flight and can quickly respond to control commands. The drone blades are equipped with custom baffles. During the take-off, landing or transportation of the drone, the blade-loading baffles can effectively prevent the blades from being hit or scratched, thereby protecting the blades from damage. The baffle can also prevent external objects from directly contacting the drone's fuselage structure, avoiding unnecessary damage to the fuselage.

[0079] The drone's flight controller (FC) can be a Pixhawk 6C, responsible for receiving and processing sensor data and executing flight commands. The flight controller boasts high reliability and stability, capable of completing position and velocity guidance flight missions and independently developing and expanding upon this foundation. By integrating sensors such as inertial navigation, the flight controller can sense the drone's position and attitude in real time, ensuring precise control during flight. During interception missions, the flight controller must precisely control the drone's flight attitude, speed, and altitude to ensure accurate approach and capture of the target. The stability and reliability of the flight controller impact the drone's safe flight and successful target capture.

[0080] A drone's gimbal is a device used to stabilize a camera or sensor, ensuring a stable viewing angle and image quality throughout flight. During interception missions, the gimbal helps drones more accurately identify and track targets.

[0081] Equipping a drone with an industrial computer integrates a powerful computing platform, enabling it to perform more complex tasks and process larger amounts of data. It can also run high-precision positioning and navigation algorithms, improving flight accuracy and stability. For tasks that require real-time processing and analysis of large amounts of data, such as urban environmental monitoring and terrain simulation, industrial computers provide powerful computing power.

[0082] Drones equipped with switches enable data communication and sharing between multiple devices. This can be used for applications such as drone swarm flight and multi-sensor data fusion. Switches transmit data generated by different devices on a drone to other devices or ground stations, enabling data sharing and remote monitoring.

[0083] Drones are equipped with wireless image and data transmission equipment, which transmits images and data collected by cameras and other sensors onboard drones to ground stations or remote terminals in real time. When real-time data from drones is required, such as data related to intercepted objects, wireless image and data transmission ensures timely data transmission.

[0084] The interception execution device can be used to launch small objects, such as net bags or other jammers, to capture / intercept the corresponding target (such as an invading drone). During the execution of the interception mission, the interception execution device can quickly and accurately launch the jammer to ensure the successful interception / capture of the target.

[0085] Please refer to Figure 1 In this embodiment, the tracking and interception method may include the following steps:

[0086] Step 110, obtaining the latest image of the tracking target through the camera on the drone;

[0087] Step 120 , determining a position of the tracking target in an image coordinate system of the latest image based on the latest image as a first position;

[0088] Step 130 , determining a position error of the first position using a prediction algorithm based on an improved Kalman filter based on a delay time for acquiring the latest image;

[0089] Step 140: determining control parameters for visual servo tracking of the tracking target based on the position error;

[0090] Step 150: Based on the first position and the control parameters, control the UAV to fly to track and intercept the tracking target.

[0091] In the above-described embodiment, the delay time of the latest image is used to determine the position error of the first position using a prediction algorithm based on an improved Kalman filter. Then, based on the position error, control parameters for visual servo tracking of the corresponding tracking target are determined. Based on the first position and the control parameters, the drone's flight is controlled to track the corresponding tracking target. In this way, position error compensation is performed using the delay time of the latest image, and the drone is controlled to track the corresponding target based on the position error. This improves tracking accuracy, reduces tracking and interception errors, and ultimately increases interception success rates.

[0092] The following is a detailed description of the various steps in the tracking and interception method:

[0093] In step 110, the camera on the drone can periodically capture the target (e.g., an intruding drone) to obtain images at different time points. The image acquisition period can be flexibly set according to actual conditions and is not specifically limited here. In addition, the camera can be a monocular camera or other camera (e.g., a binocular camera, a depth camera, etc.).

[0094] Generally speaking, the control cycle of a drone is shorter than the cycle of a camera collecting images. The latest image can be understood as the image collected by the camera closest to the current moment.

[0095] In other embodiments, if the drone's control cycle is longer than the camera's image acquisition cycle, the camera may capture multiple images during a single control cycle. In this case, rather than processing each image captured by the camera, the system selects a single image from the captured images based on the control cycle as the target image. The target image within the current control cycle can be considered the most recent image. The control cycle can be flexibly set based on actual circumstances and is not specifically limited here.

[0096] In this embodiment, the user can pre-build a pinhole camera model based on the camera on the drone to obtain the transformation matrix from the world coordinate system to the image coordinate system. The construction methods of the world coordinate system, camera coordinate system, image coordinate system, pixel coordinate system, and body coordinate system are all conventional.

[0097] Please refer to Figure 2 , the world coordinate system and the camera coordinate system can be as follows Figure 2 As an example, assume that the coordinates of the intruder (i.e., the tracking target) in the world coordinate system are p w =(x w ,y w , z w ), the subscript w represents the world coordinate system. After translation t and rotation Re c Then, in the camera coordinate system F c The coordinates in are p c =(x c ,y c , z c ), subscript c represents the camera coordinate system. The coordinate transformation relationship between the world coordinate system and the camera coordinate system is expressed in the form of a homogeneous matrix:

[0098]

[0099] in, Represents the zero vector, t represents a 3×1 translation vector, Re c Represents a 3×3 rotation matrix.

[0100] Let the image coordinate system be F i=[x i ,y i ], according to the pinhole imaging principle and the similar triangle principle, we can get:

[0101]

[0102] Where λ represents the effective focal length of the camera, that is, the distance from the optical center of the camera to the image plane. In the actual detection algorithm, according to the coordinates of any point in the image coordinate system p = (x i ,y i ), we can get the pixel coordinates (u, n) in the pixel coordinate system, and the relationship is:

[0103] u=λ x x i +u0 (4)

[0104] n=λ y y i +n0 (5)

[0105] Where λ x represents the horizontal axis conversion coefficient in the pixel coordinate system; u0 represents the deviation coordinate value of the horizontal axis of the two coordinate systems (pixel coordinate system and image coordinate system); λ y Represents the vertical axis conversion coefficient in the pixel coordinate system; n0 represents the deviation coordinate value of the vertical axes of the two coordinate systems.

[0106] By sorting out the above formulas (4) and (5), we can get the transformation matrix from the world coordinate system to the pixel coordinate system:

[0107]

[0108] Based on this transformation matrix, the coordinates of the pixel points in the latest image can be converted to the corresponding coordinates in the world coordinate system.

[0109] In step 120, based on the latest image, determining the position of the tracking target in the image coordinate system of the latest image as the first position may include:

[0110] The position coordinates of the center point of the image region of the tracking target in the latest image are used as the position of the tracking target in the image coordinate system of the latest image to obtain the first position.

[0111] Understandably, to simplify calculations, the area of ​​the smallest circumscribed rectangular frame of the tracking target in the latest image may be used as the tracking target area, and the position coordinates of the center point of the rectangular frame are the first position.

[0112] Please refer to Figure 3In this embodiment, since camera imaging and image processing introduce a certain delay, the image information obtained at any given time is not instant. Usually, the control frequency of the drone is much higher than the image acquisition frequency of the camera. Figure 3 As shown in the figure, let the control period of the drone be T, and the image delay be DT periods relative to the control period. The solid line represents the image data (image frame) collected by the camera, and the dotted line represents no data.

[0113] At time kT, the image measurement value P is obtained kT satisfy:

[0114]

[0115] Among them, η img Is noise with zero mean and obeys Gaussian distribution. Image measurement value P kT Indicates the coordinate information value of the measured target point (such as a tracking target). The mod function is a modulo operator used to calculate the remainder after dividing two numbers.

[0116] In the pixel coordinate system, the motion equation of the tracking target (intercepted target) can be expressed as:

[0117]

[0118] where u k , n k is the coordinate of the tracked target in the image captured at the previous time step, and v u , v n is the speed of the target along the u-axis and n-axis in the image plane. Therefore, the following formula (17) can be written in the form of a state equation:

[0119] X(k+1)=FX(k)+W(k) (9)

[0120] Where F is the state transfer matrix; W(k) is the comprehensive noise matrix, which contains the system state noise and unknown motion acceleration noise, and is assumed to be a Gaussian random variable with zero mean; X(k) is the state variable at time kT; X(k+1) is the state variable at time (k+1)T;

[0121]

[0122] The state transfer matrix F is:

[0123]

[0124] In step 130, based on the delay time of acquiring the latest image, a prediction algorithm based on an improved Kalman filter is used to determine the position error of the first position, including:

[0125] At time kT, the first position z(k) of the tracking target in the latest image obtained is expressed as:

[0126] z(k)=[u(k),n(k)] T (12)

[0127] In formula (12), T represents transposition, u(k) represents the coordinate value of the u-axis (i.e., the horizontal axis) in the pixel coordinate system, and n(k) represents the coordinate value of the n-axis (i.e., the vertical axis) in the pixel coordinate system. The observed image at time kT is the measured image obtained at time (kD)T, and DT represents the delay time for acquiring the latest image.

[0128] The observation vector is replaced by the optimal estimate of the system state vector, wherein the observation vector includes the first position of the tracked target in the latest image, and the observation vector satisfies:

[0129]

[0130] Z(i) represents the observation vector at time iT, where T is the control period; X(i) represents the system state vector at time iT; H represents the measurement matrix; k-(D-1)T~k represents the time period from kT-(D-1)T to kT; in this embodiment, when "k" appears independently, it is equivalent to "kT", which is equivalent to the abbreviation of time "kT";

[0131] recursively predicting the system state vector based on a Kalman filter algorithm, selecting a fitting function using historical data during the recursive prediction process, and obtaining a position error of the first position based on the fitting function;

[0132] The fitting function f() is:

[0133]

[0134] w j represents the polynomial coefficients; represents the predicted position of the tracking target;

[0135] The position error e i for:

[0136]

[0137] The historical data is the latest image of the tracked target obtained within a sliding window with a window size of N;

[0138] The polynomial coefficients are calculated by the least squares method to minimize the sum of squares of the fitting errors, which can be expressed as:

[0139]

[0140] min represents the minimum function, which is used to return the minimum value among the given parameters.

[0141] Understandably, assuming that the current time is kT, the acquired target image coordinates are z(k)=[u(k),n(k)] T , which is actually the measurement value obtained at time (kD)T. The current target image coordinates cannot be directly obtained, so the measurement delay needs to be compensated. By using the motion equations of the tracking target established above, that is, formulas (8)-(10), a Kalman filter is used to perform D-step prediction. In the Kalman filter algorithm, when the image coordinates from (k-D+1)T to kT cannot be directly obtained, the optimal estimate of the system state vector X(k) is used to replace the observation vector Z(k).

[0142] The calculation process based on the Kalman filter algorithm can be:

[0143]

[0144] P(k) - =FP(k-1)F T +Q(k) (18)

[0145] K(k)=P(k) - H T (H(k)P(k) - +R(k)) -1 (19)

[0146]

[0147] P(k)=(IK(k)H)P(k) - (twenty one)

[0148] In the above calculation formula, formula (17) is the state prediction equation, which is used to perform a priori state estimation using the system model; F is the state transfer matrix.

[0149] Formula (18) is the error covariance prediction equation, which is used to quantify the uncertainty of state prediction and reflect the influence of model error and process noise. P is the state covariance matrix, and Q is the process noise covariance matrix (during UAV flight, factors such as wind interference and motor vibration will cause the actual motion of the UAV to deviate from the ideal motion model. These deviations can be represented by process noise, and the Q matrix is ​​the covariance description of this noise).

[0150] Formula (19) is the Kalman gain calculation equation, which determines the degree of correction of the state estimation by the observation data. R is the measurement noise covariance matrix (when using GPS to measure the position of the UAV, the error of GPS itself will cause uncertainty in the measurement value. This uncertainty can be described by the measurement noise covariance matrix R).

[0151] Formula (20) is the state update equation, which reduces the prediction error and improves the estimation accuracy by utilizing the observation data. K is the Kalman gain matrix and H is the measurement matrix.

[0152] Formula (21) is the error covariance update equation, which is used to correct the covariance in the prediction stage to ensure the convergence and stability of the filtering results. I is the unit matrix.

[0153] To further optimize the performance of the Kalman filter algorithm, considering the limitations of the original Kalman filter algorithm in directly using the system's optimal estimate instead of the observed value for recursion when no target pixel measurement value is available, this application obtains more reasonable observed values ​​through fitting estimation, uses historical data, and selects the above-mentioned fitting function (i.e., Formula (14)) for prediction.

[0154] The optimal polynomial coefficients are obtained by the least squares method to minimize the sum of squares of the fitting errors:

[0155]

[0156] As the amount of image data increases, using all historical data for curve fitting may lead to slower fitting speed or even fitting failure. Therefore, a sliding window approach is adopted to control the historical data used for fitting, that is, the above formula (16) is used to minimize the sum of squares of fitting errors.

[0157] The image data within the latest period of time is maintained in a sliding window of fixed size N, and the image data in the window is used for fitting and prediction of the next step, thereby forming a delay compensation and sampling interval estimation algorithm based on the improved Kalman filter algorithm, which is the prediction algorithm based on the improved Kalman filter of this application. In this way, the accuracy reduced by imaging delay and sampling frequency problems is effectively improved.

[0158] In step 140, the position error includes the lateral error and the longitudinal error of the tracking target in the image coordinate system. The lateral error and the longitudinal error can be calculated by the above formula (15);

[0159] Determining control parameters for visual servo tracking of the tracking target based on the position error includes:

[0160] Step 141: determining the yaw angular velocity and roll angular velocity of the UAV in the body coordinate system based on the lateral error;

[0161] Step 142: Determine the desired thrust and desired pitch rate of the UAV based on the longitudinal error; the control parameters include the yaw rate, the roll rate, the desired thrust, and the desired pitch rate.

[0162] Step 141 may include:

[0163] The yaw angular velocity and the roll angular velocity are determined according to the lateral error using a first preset formula and a second preset formula, wherein the first preset formula is:

[0164]

[0165] Where, represents the yaw angular velocity of the UAV in the body coordinate system; k1 and k2 represent the first control parameter and the second control parameter respectively, e u represents the lateral error;

[0166] The second preset formula is:

[0167]

[0168] Where, represents the roll angular velocity of the UAV in the body coordinate system; k3 and k4 represent the third control parameter and the fourth control parameter respectively; φ represents the roll angle of the UAV.

[0169] In this embodiment, the various control parameters (the first through eighth control parameters described below) are acquired using conventional methods and will not be further elaborated here. For lateral errors of targets in an image, the yaw motion of the drone can be used to lock the target at the center of the image. In the longitudinal channel, the altitude of the drone significantly affects the vertical error of the target in the image.

[0170] Step 142 may include:

[0171] According to the longitudinal error, the expected thrust and the expected pitch angular velocity are determined by a third preset formula and a fourth preset formula. The third preset formula is:

[0172]

[0173] Where, f d represents the desired thrust; m represents the mass of the UAV; θ represents the pitch angle of the UAV; k5 and k6 represent the fifth control parameter and the sixth control parameter respectively; represents the desired longitudinal speed of the UAV, which can be flexibly set by the user; n represents the longitudinal error; g represents the acceleration due to gravity;

[0174] The fourth preset formula is:

[0175]

[0176] Where, represents the desired pitch angular velocity; k7 and k8 represent the seventh and eighth control parameters; θ d represents the desired pitch angle;

[0177]

[0178] Where θ forward Indicates the preset pitch angle for tracking the target to fly forward, that is, the pitch angle to ensure that the drone flies forward; y Indicates the focal length coefficient of the camera on the y-axis of the camera coordinate system.

[0179] In this embodiment, step 150, controlling the flight of the UAV based on the first position and the control parameter to track and intercept the target, may include:

[0180] Based on a pre-established image Jacobian matrix, the control parameters are converted into control quantities for visual servo control;

[0181] Based on the first position and the control amount, the UAV is controlled to operate so that the tracking target is at the center of the image captured by the camera, and the tracking target is tracked and intercepted.

[0182] In image-based visual servoing (IBVS) control, the image Jacobian matrix can establish a mapping relationship between the rate of change of image features and the spatial velocity (linear velocity and angular velocity) of the camera, providing the basis for mapping position error to control variables. Therefore, based on this image Jacobian matrix, the control parameters obtained based on the position error can be converted into control variables for visual servo control.

[0183] The image Jacobian matrix can be established as follows:

[0184] A visual servo model is constructed based on the pinhole camera model. Since the rate of change of the target point in three-dimensional space (such as the center point of the tracking target) can be regarded as the transformation of rotation and translation per unit time, assuming the angular velocity is Ω and the linear velocity is V, the rate of change of the target point is:

[0185]

[0186] Expanding formula (28) yields:

[0187]

[0188] Further deduction yields the equation for the rate of change of pixel coordinates:

[0189]

[0190] Formula (30) is written in matrix form as:

[0191]

[0192] in:

[0193]

[0194]

[0195] J n is the image Jacobian matrix, f refers to the focal length of the camera; u and n represent the horizontal and vertical pixel coordinates in the pixel coordinate system, respectively. x, y, and z represent the coordinate values ​​of the target point in three-dimensional space (such as the center point of the tracking target) in the camera coordinate system. x 、ω y 、ω z , respectively represent the angular velocity of the target point relative to the camera around the x-axis, y-axis, and z-axis of the camera coordinate system, describing the rotational motion of the target point in space. x 、V y 、V z , respectively represent the linear velocity of the target point relative to the camera in the x-axis, y-axis, and z-axis directions of the camera coordinate system, describing the translational motion of the target point in space.

[0196] In some optional embodiments, the method further comprises:

[0197] Based on a pre-established transformation matrix between a world coordinate system and an image coordinate system, converting the first position of the tracking target into a second position in the world coordinate system;

[0198] When the second position of the tracking target is within the preset interception range of the UAV, the interception execution device of the UAV is controlled to launch an interference object toward the tracking target.

[0199] In this embodiment, the transformation matrix can be found in the above formula (6). It can be understood that the above formulas (4) to (6) can be used to transform the first position in the image coordinate system into the second position in the world coordinate system. The preset interception range can be flexibly calibrated according to actual conditions and is not specifically limited here.

[0200] In this embodiment, the tracking target may be a drone or other target (such as a flying bird) other than the drone itself, and the interference object may include a net or other object used to hinder the flight of the tracking target. The tracking target and the interference object are not specifically limited here.

[0201] Based on the above design, in the tracking and interception method provided in this embodiment, real-time motion information and visual feedback are combined to achieve stable tracking and high-precision interception of invading drone targets, overcome the limitations of existing IBVS controllers, and effectively deal with the impact of the complex characteristics of drones on interception control. In addition, the embodiment of the present application proposes an improved prediction algorithm, namely a delay compensation and sampling interval prediction (DCSIP) algorithm based on an improved Kalman filter algorithm, which can accurately estimate the imaging delay and improve the control frequency, solve the problem of reduced interception accuracy caused by imaging delay and sampling frequency problems, and ensure accurate interception in high-speed flight scenarios. Utilize the preset pitch angle θ for tracking the forward flight of the tracking target forward , which can solve the problem of interception failure caused by the backward flight of drones in similar studies, and enable the provided tracking and interception method to successfully intercept any maneuvering target in three-dimensional space, significantly improving the versatility and reliability of drone interception technology. Combining real-time motion and visual feedback, high-precision and stable target tracking and interception are achieved. The DCSIP algorithm optimizes the selection of observation values ​​through fitting estimation, uses a sliding window to control data, accurately estimates imaging delay, improves system frequency and interception accuracy; builds a low-cost physical interception solution based on a monocular camera, which is lower in cost and easier to deploy, and can solve complex problems such as image acquisition delay, low frame rate and target motion, ensuring accurate interception.

[0202] In this embodiment, the drone can adjust its attitude and flight speed in real time based on the target's positional deviation in the image, effectively tracking and intercepting the target. This solves the problem of the drone flying backward when the target is above or below the drone during target tracking, improving the accuracy and reliability of interception. When the tracked target appears within the preset interception range, a jammer is launched at the target to intercept or capture it, which helps to improve the success rate of interception.

[0203] 4 to 8 , the inventors conducted simulation tests and verifications on the tracking and interception method provided in the embodiments of the present application.

[0204] Validation: The experiment sets the initial coordinate of the drone equipped with a fixed monocular camera in three-dimensional space as p m= [0,0,0] meters, the interception target is an object moving in three-dimensional space, and its initial position is p t =[10,0,2] meters, the equation of motion is set to p r (t) = [10, 2sin(t), 2cos(t)] meters. At the same time, to be closer to the actual imaging environment, random Gaussian noise is added to the imaging of the intercepted target.

[0205] In this experiment, the performance of the algorithm is verified when the UAV is stationary and the target imaging is performed. By comparing the estimation results of delay compensation and sampling interval prediction based on Kalman filter and improved Kalman filter, the results are as follows: Figure 4A 、 Figure 4C Based on Kalman filtering and Figure 4B 、 Figure 4D Based on the improved Kalman filter, Real, Delay, and Predict represent the actual value, delayed value, and predicted value, respectively. The results show that both algorithms converge to the true value of the intercept target after approximately 1 second, effectively compensating for the measurement delay. Furthermore, sampling interval predictions are performed between the actual measured values, and the overall prediction trend is very close to the future update value. However, since the initial estimate is set to [0, 0] T , resulting in large initial estimation errors. However, the improved Kalman filter-based algorithm utilizes local information for fitting, approximating the position, velocity, and acceleration of nearby targets at the current moment, resulting in faster estimation convergence. This demonstrates that the improved Kalman filter algorithm has significant advantages in handling measurement delays and predicting target motion. Even when the camera itself is moving, the algorithm can effectively compensate for delays and predict the sampling interval of target image motion.

[0206] Dynamic verification: Let the drone follow the motion equation p m (t) = [cos(0.5t), cos(0.5t), 2cos(0.5t)] meters to move, and the interception target is based on the motion equation p t (t)=[10,2sin(t),2cos(t)]m. The experimental results are as follows Figure 5A and Figure 5B As shown in the figure, the results once again verify the effectiveness of the delay compensation and sampling interval prediction algorithm based on the improved Kalman filter algorithm in dynamic scenarios. Even if the UAV and the target are in motion, the algorithm can still accurately estimate the position and speed of the target, providing reliable data support for subsequent interception control.

[0207] Interception verification based on IBVS: To verify the effectiveness of the new prediction algorithm based on IBVS, experiments were also conducted in the Matlab simulation environment. The initial position of the intercepting UAV is set to p m =[0,0,0]T meters, the initial position of the target drone is p t =[15,-3,-6] meters, that is, to the upper left of the intercepting drone.

[0208] Verification of the effectiveness of the prediction algorithm interception: the target drone is intercepted with v s = [3; 0; -3] m / s flying diagonally upwards, intercepting the drone from p m =[0,0,0] T The experimental results are as follows: 6A to 6D As shown in the figure, under these conditions, the interceptor drone successfully intercepted the target at a speed of approximately 5 meters per second. Intruder and interceptor refer to the intruder (i.e., the target being tracked) and interceptor (the drone used to intercept the target), respectively. Compared to previously proposed IBVS-based interception control algorithms, the algorithm proposed in this application solves the problem of the drone flying backward when the target is initially positioned above the drone, fully demonstrating the algorithm's effective interception capabilities under different target positions and motion states.

[0209] Use snake-like maneuver to intercept the target: the target UAV moves in the xy plane at a speed of v t =[3;3sin(t);0]m / s, flying forward while serpentine maneuvering left and right, intercepting the drone from p m =[0,0,0] T Meters away, start maneuvering to intercept the target. Figure 7A and Figure 7B As shown in the figure, in this complex motion situation, the intercepting drone still successfully intercepted the target at a speed of about 5 meters per second, which proves the effective tracking and interception capability of the algorithm for targets with complex maneuvers.

[0210] Intercept any maneuvering target: The target UAV moves in three-dimensional space at a speed v t = [3; 3sin(t); 3cos(t)] m / s, performing a forward spiral maneuver to intercept the drone from p m =[0,0,0] T The experimental results are as follows: Figure 8A and Figure 8B As shown in the figure, under these complex three-dimensional maneuvers, the interceptor drone successfully intercepted the target at a speed of approximately 5 meters per second. This shows that the prediction algorithm based on IBVS in this application has wide applicability and can effectively handle various complex target maneuvers and achieve high-precision interception.

[0211] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented through hardware or by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0212] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of code, and a part of the module, program segment or code includes one or more executable instructions for implementing the specified logical function. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0213] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for tracking and intercepting intrusion targets of unmanned aerial vehicles (UAVs) based on visual servoing based on time delay compensation, characterized in that: The method comprises: Obtain the latest images of the tracking target through the camera on the drone; determining, based on the latest image, a position of the tracking target in an image coordinate system of the latest image as a first position; Determining a position error of the first position using a prediction algorithm based on an improved Kalman filter based on a delay time for acquiring the latest image; determining control parameters for visual servo tracking of the tracking target based on the position error; Based on the first position and the control parameters, the UAV is controlled to fly so as to track and intercept the tracking target.

2. The method according to claim 1, characterized in that Determining a position error of the first position based on a delay time for acquiring the latest image and using a prediction algorithm based on an improved Kalman filter includes: At time kT, the first position z(k) of the tracking target in the latest image obtained is expressed as: z(k)=[u(k),n(k)] T Where u(k) represents the coordinate value of the u-axis in the pixel coordinate system, n(k) represents the coordinate value of the n-axis in the pixel coordinate system; the observed image at time kT is the measured image obtained at time (kD)T, and DT represents the delay time for acquiring the latest image; The observation vector is replaced by the optimal estimate of the system state vector, wherein the observation vector includes the first position of the tracked target in the latest image, and the observation vector satisfies: Where Z(i) represents the observation vector at time iT, T is the control period; X(i) represents the system state vector at time iT; H represents the measurement matrix; recursively predicting the system state vector based on a Kalman filter algorithm, selecting a fitting function using historical data during the recursive prediction process, and obtaining a position error of the first position based on the fitting function; The fitting function f() is: w j represents the polynomial coefficients; represents the predicted position of the tracking target; The position error e i for: The historical data is the latest image of the tracked target obtained within a sliding window with a window size of N; The polynomial coefficients are calculated by the least squares method to minimize the sum of squares of the fitting errors, which can be expressed as:

3. The method according to claim 1, characterized in that The position error includes a lateral error and a longitudinal error of the tracking target in the image coordinate system; Determining control parameters for visual servo tracking of the tracking target based on the position error includes: Determining the yaw angular velocity and roll angular velocity of the UAV in a body coordinate system based on the lateral error; According to the longitudinal error, the expected thrust and the expected pitch angular velocity of the UAV are determined; the control parameters include the yaw angular velocity, the roll angular velocity, the expected thrust and the expected pitch angular velocity.

4. The method according to claim 3, characterized in that Determining the yaw angular velocity and the roll angular velocity of the UAV in a body coordinate system based on the lateral error includes: The yaw angular velocity and the roll angular velocity are determined according to the lateral error using a first preset formula and a second preset formula, wherein the first preset formula is: Where, represents the yaw angular velocity of the UAV in the body coordinate system; k1 and k2 represent the first control parameter and the second control parameter respectively, e u represents the lateral error; The second preset formula is: Where, represents the roll angular velocity of the UAV in the body coordinate system; k3 and k4 represent the third control parameter and the fourth control parameter respectively; φ represents the roll angle of the UAV.

5. The method according to claim 3, characterized in that Determining the desired thrust and desired pitch velocity of the UAV based on the longitudinal error includes: According to the longitudinal error, the expected thrust and the expected pitch angular velocity are determined by a third preset formula and a fourth preset formula. The third preset formula is: Where, f d represents the desired thrust; m represents the mass of the UAV; θ represents the pitch angle of the UAV; k5 and k6 represent the fifth control parameter and the sixth control parameter respectively; represents the desired longitudinal velocity of the UAV; e n represents the longitudinal error; g represents the acceleration due to gravity; The fourth preset formula is: Where, represents the desired pitch angular velocity; k7 and k8 represent the seventh and eighth control parameters; θ d represents the desired pitch angle; Where θ forward represents the preset pitch angle for tracking the target forward; y Indicates the focal length coefficient of the camera on the y-axis of the camera coordinate system.

6. The method according to claim 1, characterized in that Based on the first position and the control parameter, controlling the UAV to fly to track and intercept the tracking target includes: Based on a pre-established image Jacobian matrix, the control parameters are converted into control quantities for visual servo control; Based on the first position and the control amount, the UAV is controlled to operate so that the tracking target is at the center of the image captured by the camera, and the tracking target is tracked and intercepted.

7. The method according to claim 1, characterized in that Determining, based on the latest image, a position of the tracking target in an image coordinate system of the latest image as a first position, includes: The position coordinates of the center point of the image region of the tracking target in the latest image are used as the position of the tracking target in the image coordinate system of the latest image to obtain the first position.

8. The method according to claim 1, characterized in that The method further comprises: Based on a pre-established transformation matrix between a world coordinate system and an image coordinate system, converting the first position of the tracking target into a second position in the world coordinate system; When the second position of the tracking target is within the preset interception range of the UAV, the interception execution device of the UAV is controlled to launch an interference object toward the tracking target.

9. The method according to claim 8, characterized in that The tracking target includes drones other than the drone itself, and the interference object includes a net.

Images