Vehicle-mounted unmanned aerial vehicle vehicle tracking system, method and computer program product based on visual servo control

Abstract

This invention provides a vehicle tracking system, method, and computer program product for vehicle-mounted unmanned aerial vehicles (UAVs) based on visual servo control. It includes: a coordinate transformation module that acquires the three-dimensional world coordinates of the UAV and the target vehicle, and through a preset multi-layer coordinate system transformation, obtains the first pixel coordinates of the target vehicle in the original image from the camera; a visual tracking module that uses a visual algorithm to detect the second pixel coordinates of the target vehicle in the original image; a coordinate fusion module that fuses the two coordinates to generate estimated pixel coordinates of the target vehicle; and a command module that generates an outer loop command based on the positional relationship between the estimated coordinates and the center of the original image, sending it to the flight controller / gimbal to control attitude / rotation, ensuring the target is centered in the original image; simultaneously, an inner loop command is generated based on the estimated coordinates and the window center, sent to the image processing unit to adjust window cropping, ensuring the target coincides with the window center. This invention improves tracking robustness through multi-source fusion, and the inner and outer loop control balances tracking and mapping, reducing energy consumption and suppressing image jitter.

Description

Technical Field

[0001] This invention relates to the field of vehicle-mounted unmanned aerial vehicle (UAV) technology, specifically to a vehicle-mounted UAV tracking system, method, and computer program product based on visual servo control. Background Technology

[0002] Vehicle-mounted drones, as an aerial extension of mobile platforms, are widely used in vehicle platooning, security patrols, and film and television tracking. One of their core functions is to automatically track the main vehicle or a specific target vehicle on the ground and transmit stable, centered real-time footage back to the vehicle-mounted control terminal.

[0003] Existing technical solutions mainly fall into two categories: The first is absolute position tracking based on the Global Navigation Satellite System (GNSS). The UAV controls its position solely based on the difference in GPS coordinates between itself and the target vehicle. This solution is simple, but its accuracy drops drastically in environments with poor GNSS signals, such as urban canyons (errors often reaching the meter level), and it cannot provide fine visual imaging, making it easy for the target to shift or even move out of the field of view. The second is target tracking based on pure vision. The UAV uses an onboard camera and algorithms such as deep learning or correlation filtering to lock onto the target. This solution is relatively accurate, but it relies entirely on visual information. When the target is briefly obscured, lighting changes drastically, or rapid maneuvers cause image blurring, tracking loss is highly likely, and re-acquiring the target is difficult.

[0004] Furthermore, due to safety and field-of-view considerations, drone cameras typically employ wide-angle lenses, capturing panoramic images far larger than the display screen size. Existing technology lacks an effective solution for intelligently selecting a sub-window from the vast camera view for display, while ensuring the moving target remains centered within that sub-window. Simple image cropping leads to target loss, while frequent adjustments to the drone's attitude cause image jitter and energy waste. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a vehicle tracking system for in-vehicle drones based on visual servo control, comprising: The coordinate transformation module is used to obtain the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system. Through the preset multi-level coordinate system transformation relationship, the first pixel coordinate of the target vehicle in the original image captured by the UAV camera is calculated based on the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system. The visual tracking module is used to detect the second pixel coordinates of the target vehicle in the original image using a visual tracking algorithm; The coordinate fusion module is used to fuse the first pixel coordinates and the second pixel coordinates to obtain the estimated pixel coordinates of the target vehicle in the original image captured by the drone camera. The instruction generation module is used to generate an outer loop control instruction based on the positional relationship between the estimated pixel coordinates and the preset center area of ​​the original image. This outer loop control instruction is sent to the UAV flight control unit to control the UAV's flight attitude, or to the camera gimbal servo unit to control the rotation angle of the camera gimbal, so that the target vehicle remains within the preset center area of ​​the original image. The module then adjusts the estimated pixel coordinates according to the outer loop control instruction to obtain updated coordinates. Based on the positional relationship between the coordinates of the center of the image software's display window in the original image and the updated estimated pixel coordinates, an inner loop control instruction is generated. This inner loop control instruction is sent to the image processing unit to adjust the cropping position of the image software's display window in the original image, so that the coordinates of the center of the image software's display window in the original image coincide with the updated estimated pixel coordinates.

[0006] Furthermore, in the coordinate transformation module, the specific method for calculating the pixel coordinates of the target vehicle in the original image captured by the UAV camera based on the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system through a preset multi-layer coordinate system transformation relationship is as follows: Based on the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system, the three-dimensional position coordinates of the target vehicle in the UAV volume coordinate system are calculated. The conversion formula is as follows: in, The three-dimensional position coordinates of the target vehicle in the UAV body coordinate system. The three-dimensional position coordinates of the target vehicle in the world coordinate system. The three-dimensional position coordinates of the UAV in the world coordinate system. Let be the rotation matrix from the world coordinate system to the UAV body coordinate system. , and These are the UAV yaw angle, UAV pitch angle, and UAV roll angle, respectively, relative to the world coordinate system of the UAV body coordinate system. The 3D position coordinates of the target vehicle in the UAV volume coordinate system are converted to 3D position coordinates in the camera coordinate system using the following formula: in, Let be the three-dimensional position coordinates of the target vehicle in the camera coordinate system. Let be the rotation matrix from the UAV body coordinate system to the camera coordinate system. and These are the camera gimbal pitch angle and camera gimbal yaw angle, respectively, relative to the UAV body coordinate system. This represents the three-dimensional translational offset of the camera's optical center relative to the origin of the UAV's body coordinate system. The 3D position coordinates of the target vehicle in the camera coordinate system are converted to pixel coordinates in the original image coordinate system captured by the UAV camera. The conversion formula is as follows: , in, For the camera intrinsic parameter matrix, The horizontal pixel coordinates of the target vehicle in the original image coordinate system. The vertical pixel coordinates of the target vehicle in the original image coordinate system. This represents the normalized horizontal coordinates of the target vehicle in the camera coordinate system after perspective projection and correction for radial and tangential distortion. This represents the normalized vertical coordinates of the target vehicle in the camera coordinate system after perspective projection and correction for radial and tangential distortion. The equivalent focal length of the camera in the horizontal direction. The equivalent focal length in the vertical direction of the camera. The horizontal pixel coordinates of the camera's principal point. The vertical pixel coordinates of the camera's principal point; The first pixel coordinates of the target vehicle in the raw image captured by the drone camera .

[0007] Furthermore, the world coordinate system is a globally fixed three-dimensional rectangular coordinate system, which serves as a reference for the absolute positions of the UAV and the target vehicle, and is used to characterize the spatial positions of the UAV and the target vehicle in real physical space. The UAV body coordinate system is a three-dimensional Cartesian coordinate system fixed to the UAV body. The origin changes synchronously with the flight position and attitude of the UAV and is used to describe the relative spatial position of the target vehicle relative to the UAV body. The camera coordinate system is a three-dimensional Cartesian coordinate system fixed to the UAV camera, with the camera optical center as the origin, used to connect the three-dimensional physical space coordinates of the body coordinate system and the two-dimensional planar coordinates of the image pixel coordinate system. The original image coordinate system is a two-dimensional discrete plane coordinate system based on the original image captured by the drone camera.

[0008] Furthermore, in the visual tracking module, the visual tracking algorithm is a visual tracker based on correlation filtering or a visual tracker based on Siamese networks.

[0009] Furthermore, in the coordinate fusion module, the specific method for fusing the first pixel coordinates and the second pixel coordinates is as follows: The first pixel coordinates and the second pixel coordinates are fused using an extended Kalman filter.

[0010] Furthermore, the specific method for fusing the first pixel coordinates and the second pixel coordinates using an extended Kalman filter is as follows: Combining the first pixel coordinates and the uniform motion model in the extended Kalman filter, the prior value of the target vehicle's state vector at the current moment is predicted using the following formula: Target vehicle number State vector at time step , For the target vehicle The horizontal position in the image pixel coordinate system at any given time. For the target vehicle The vertical position of the image pixel coordinate system at any given time. For the target vehicle The horizontal velocity in the image pixel coordinate system at any given moment. For the target vehicle The vertical velocity in the image pixel coordinate system at any given moment; in, For the first The posterior estimate at time 1 for the 2nd time The prior prediction of the state vector at time step [time]. It is the first The state vector at time t becomes the first The state transition matrix used to predict the state vector at time step [time]. The target vehicle is number The state vector at time t, It is the noise in the target vehicle motion prediction process. It is the target vehicle geometric position prediction guidance gain, used to adjust the correction weight of the target vehicle geometric position prediction on the predicted state of the uniform motion model. The coordinates of the first pixel In the Coordinates of time, To map the state vector into an observation matrix of pixel observations, which is then used for matching with the second pixel coordinates and calculating residuals; Calculate the first Observe residuals at all times ,in, It is the coordinate of the second pixel. In the The coordinates of time; Calculate the first Time Kalman Gain ,in, Based on the The posterior covariance at time t on the first The prior prediction covariance matrix of the state at time step. It is the transpose symbol. It is the first The noise covariance matrix of visual observations at any given time; Based on the Time Kalman Gain With the Observe residuals at all times Correct the prior prediction value , to obtain the posterior estimate + ; Update the posterior covariance matrix Finally, the position component is extracted from the posterior estimate to obtain the estimated pixel coordinates. , It is the estimated horizontal pixel coordinate of the target vehicle in the original image coordinate system. It is the estimated vertical pixel coordinate of the target vehicle in the original image coordinate system.

[0011] Furthermore, in the instruction generation module, the specific method for generating the outer loop control instruction based on the positional relationship between the estimated pixel coordinates and the preset center region of the original image is as follows: Calculate the pixel deviations between the estimated pixel coordinates and the center of the original image in the horizontal and vertical directions, respectively. When the magnitude of the pixel deviation is greater than a first deviation threshold, generate yaw rate and pitch rate control commands for the UAV or camera gimbal using the PD control method based on the pixel deviations. The yaw rate control command is used to correct the horizontal pixel deviation of the target vehicle in the original image, and the pitch rate control command is used to correct the vertical pixel deviation of the target vehicle in the original image.

[0012] Furthermore, in the instruction generation module, the specific method for generating the inner loop control instruction based on the positional relationship between the estimated pixel coordinates and the center coordinates of the display window of the image software on the current screen in the original image is as follows: The estimated pixel coordinates and the coordinates of the center of the display window of the image software on the current screen in the original image are calculated in the horizontal and vertical directions, respectively. The goal is to make the estimated pixel coordinates of the target vehicle coincide with the coordinates of the center of the display window of the image software on the current screen in the original image. Based on the pixel deviation, the new coordinates of the target cropping position after translation of the display window of the image software on the current screen in the original image are calculated at the upper left corner. It is determined whether the new coordinates exceed the effective cropping boundary of the original image. If they do not exceed the boundary, the display window of the image software on the screen is translated to the cropping position defined by the new coordinates. If they exceed the boundary, the new coordinates are clamped to the effective cropping boundary of the original image, and the outer loop control command is regenerated and sent to adjust the flight attitude of the UAV or the rotation angle of the camera gimbal so that the target vehicle is kept in the preset center area of ​​the original image.

[0013] A vehicle tracking method for vehicular unmanned aerial vehicles based on visual servo control, comprising: The three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system are obtained. Through the preset multi-level coordinate system transformation relationship, the first pixel coordinates of the target vehicle in the original image captured by the UAV camera are calculated based on the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system. The second pixel coordinates of the target vehicle in the original image are detected using a visual tracking algorithm; The first pixel coordinates and the second pixel coordinates are fused to obtain the estimated pixel coordinates of the target vehicle in the original image captured by the drone camera; Based on the positional relationship between the estimated pixel coordinates and the preset center area of ​​the original image, an outer loop control command is generated. The outer loop control command is sent to the UAV flight control unit to control the flight attitude of the UAV, or sent to the camera gimbal servo unit to control the rotation angle of the camera gimbal, so that the target vehicle remains in the preset center area of ​​the original image. The updated estimated pixel coordinates are obtained after adjustment according to the outer loop control command. An inner loop control command is generated based on the positional relationship between the center coordinates of the display window of the image software on the current screen in the original image and the updated estimated pixel coordinates. The inner loop control command is sent to the image processing unit to adjust the cropping position of the display window of the image software on the current screen in the original image, so that the center coordinates of the display window of the image software on the current screen in the original image coincide with the updated estimated pixel coordinates.

[0014] A computer program product includes a computer program / instructions that, when executed by a processor, implement the above-described visual servo control-based vehicle tracking method for vehicular unmanned aerial vehicles.

[0015] The beneficial effects of this invention are as follows: 1. This invention achieves precise mapping from the 3D positions of the UAV and target vehicle in the world coordinate system to the 2D pixels of the original camera image through multi-layer coordinate system transformation, laying a reliable geometric benchmark for dual-source coordinate fusion; it also creatively introduces a geometric position prediction guidance correction term into the EKF prediction step. A dual-drive fusion architecture combining motion model prediction and absolute position guidance is constructed. This architecture not only compensates for the cumulative drift defect of pure motion models, but also achieves scene adaptation through adjustable guidance gain: when vision is normal, the weight of the geometric position prediction guidance correction term is weakened and noise interference is suppressed to ensure tracking accuracy; when vision fails, the weight of the geometric position prediction guidance correction term is strengthened and the target's true position is anchored to avoid tracking loss. This fundamentally solves the industry pain point of single data source failure, and significantly improves the continuity and stability of target tracking in complex road scenarios. The estimated pixel coordinates after fusion are smooth and jitter-free, providing accurate input for subsequent closed-loop control.

[0016] 2. This invention creatively proposes a cascaded architecture combining outer-loop hardware control and inner-loop software control. Based on the deviation between the estimated pixel coordinates and the center of the original image, outer-loop control commands generate UAV attitude or gimbal rotation control instructions, ensuring the target remains at the center of the original image and completely avoiding tracking loss due to the target moving out of the field of view. Inner-loop control commands generate window cropping control based on the deviation between the estimated pixel coordinates and the center of the display window. Pixel-level fine-grained composition is achieved through software window translation, eliminating the need to adjust hardware attitude. This solves the problems of compositional offset and poor image quality in traditional solutions, significantly reduces unnecessary UAV / gimbal movements, effectively lowers system power consumption, extends battery life, and suppresses image jitter. It achieves a synergistic unity of stable tracking with a large field of view and fine-grained composition with a small window, significantly improving the user experience. Attached Figure Description

[0017] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0018] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0019] Example 1 This embodiment describes the practical implementation of a vehicle tracking system for vehicle-mounted drones based on visual servo control. The system in this embodiment relies on the technical concept of deep fusion of position information and visual image information, and combines a cascaded control architecture of inner-ring window translation soft control and outer-ring drone / gimbal attitude hard control. Through the collaborative work of coordinate transformation module, visual tracking module, coordinate fusion module and command generation module, it solves the technical problems of low accuracy when the signal is poor, easy loss of pure visual tracking, easy screen offset and frequent drone / gimbal movements in traditional vehicle-mounted drone tracking. This embodiment clarifies the specific algorithm selection, parameter settings, and execution flow of each functional module. The visual tracking uses a kernel correlation filter (KCF) tracker with excellent engineering adaptability. The coordinate fusion uses an extended Kalman filter to achieve smooth fusion of multi-source pixel coordinates. The control command generation uses the PD control method (proportional-derivative control method) and the window translation algorithm to achieve outer ring physical attitude adjustment and inner ring image cropping optimization, respectively. Finally, it achieves stable tracking of the target vehicle in the original image and center locking of the display window of the image software on the screen. It takes into account the high accuracy and high robustness of tracking and the low power consumption and low jitter of execution, and can be directly adapted to the engineering application requirements of existing vehicle-mounted UAV hardware platforms.

[0020] The hardware foundation of this solution is as follows: (1) The rotatable gimbal needs to integrate a gimbal servo control unit and be installed in a dedicated gimbal mounting position on the underside of the UAV fuselage. The gimbal needs to support precise rotation with dual degrees of freedom of yaw and pitch, and have mechanical limit protection. The gimbal servo control unit has a built-in communication interface, which can communicate bidirectionally with the UAV flight controller and the vehicle-mounted main control terminal to receive outer loop control commands and precisely adjust the gimbal rotation angle. (2) The UAV-borne positioning module (including GPS / BeiDou) is installed in the area on the top of the UAV fuselage without metal obstruction (to avoid satellite signal shielding). It can obtain the three-dimensional position coordinates (longitude, latitude, altitude) of the UAV in the world coordinate system in real time, as well as the UAV's yaw angle, pitch angle, and roll angle, to provide basic data for the coordinate transformation module. The positioning accuracy needs to meet the sub-meter level to ensure the accuracy of coordinate transformation. The vehicle-mounted positioning module (including GPS / Beidou) is installed on the unobstructed area of ​​the roof of the target vehicle (or the location of the center console without metal shielding). It is a positioning system with the UAV-end positioning module (ensuring that the coordinate system is the same). It obtains the three-dimensional position coordinates of the target vehicle in the world coordinate system in real time and synchronizes them to the UAV end through the vehicle-mounted communication module, providing the coordinate transformation module with the basic data of the target vehicle position. (3) The industrial-grade vision camera is installed in the main control board mounting position inside the UAV body. As the core control module of the UAV, it establishes communication connections with the gimbal servo control unit, positioning module, camera and airborne communication module. It is used to receive outer loop control commands. When the gimbal reaches the mechanical limit and cannot correct the deviation, it adjusts the yaw and pitch flight attitude of the UAV, and at the same time transmits the flight status data of the UAV back in real time. (4) The UAV flight control unit is installed in the main control board mounting position inside the UAV fuselage. As the core control module of the UAV, it establishes communication connections with the gimbal servo control unit, positioning module, camera and airborne communication module. It is used to receive outer loop control commands and adjust the yaw and pitch flight attitude of the UAV when the gimbal reaches the mechanical limit and cannot correct the deviation. At the same time, it transmits the flight status data of the UAV back in real time.

[0021] In this invention, the horizontal direction is the x-axis and the vertical direction is the y-axis.

[0022] refer to Figure 1 A vehicle tracking system for in-vehicle drones based on visual servo control, comprising: The coordinate transformation module is used to obtain the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system. Through the preset multi-level coordinate system transformation relationship, the first pixel coordinate of the target vehicle in the original image captured by the UAV camera is calculated based on the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system. The visual tracking module is used to detect the second pixel coordinates of the target vehicle in the original image using a visual tracking algorithm; The coordinate fusion module is used to fuse the first pixel coordinates and the second pixel coordinates to obtain the estimated pixel coordinates of the target vehicle in the original image captured by the drone camera. The instruction generation module is used to generate an outer loop control instruction based on the positional relationship between the estimated pixel coordinates and the preset center area of ​​the original image. The outer loop control instruction is sent to the UAV flight control unit to control the flight attitude of the UAV, or to the camera gimbal servo unit to control the rotation angle of the camera gimbal, so that the target vehicle remains in the preset center area of ​​the original image (after the outer loop control instruction is issued, the UAV flight attitude / camera gimbal angle is adjusted, the camera's shooting angle changes accordingly, and the pixel position of the target vehicle in the original image will inevitably change accordingly, and the corresponding estimated pixel coordinates also need to be updated synchronously). The updated estimated pixel coordinates are obtained after adjustment according to the outer loop control command. An inner loop control command is generated based on the positional relationship between the center coordinates of the display window of the image software on the current screen in the original image and the updated estimated pixel coordinates. The inner loop control command is sent to the image processing unit to adjust the cropping position of the display window of the image software on the current screen in the original image, so that the center coordinates of the display window of the image software on the current screen in the original image coincide with the updated estimated pixel coordinates.

[0023] As a specific implementation method, the coordinate transformation module calculates the pixel coordinates of the target vehicle in the original image captured by the UAV camera based on the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system through a preset multi-layer coordinate system transformation relationship. Based on the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system, the three-dimensional position coordinates of the target vehicle in the UAV volume coordinate system are calculated. The conversion formula is as follows: in, The three-dimensional position coordinates of the target vehicle in the UAV body coordinate system. The three-dimensional position coordinates of the target vehicle in the world coordinate system. The three-dimensional position coordinates of the UAV in the world coordinate system. Let be the rotation matrix from the world coordinate system to the UAV body coordinate system. , and These are the UAV yaw angle, UAV pitch angle, and UAV roll angle, respectively, relative to the world coordinate system of the UAV body coordinate system. The 3D position coordinates of the target vehicle in the UAV volume coordinate system are converted to 3D position coordinates in the camera coordinate system using the following formula: in, Let be the three-dimensional position coordinates of the target vehicle in the camera coordinate system. Let be the rotation matrix from the UAV body coordinate system to the camera coordinate system. and These are the camera gimbal pitch angle and camera gimbal yaw angle, respectively, relative to the UAV body coordinate system. This represents the three-dimensional translational offset of the camera's optical center relative to the origin of the UAV's body coordinate system. The 3D position coordinates of the target vehicle in the camera coordinate system are converted to pixel coordinates in the original image coordinate system captured by the UAV camera. The conversion formula is as follows: , in, For the camera intrinsic parameter matrix, The horizontal pixel coordinates of the target vehicle in the original image coordinate system. The vertical pixel coordinates of the target vehicle in the original image coordinate system. This represents the normalized horizontal coordinates of the target vehicle in the camera coordinate system after perspective projection and correction for radial and tangential distortion. This represents the normalized vertical coordinates of the target vehicle in the camera coordinate system after perspective projection and correction for radial and tangential distortion. The equivalent focal length of the camera in the horizontal direction. The equivalent focal length in the vertical direction of the camera. The horizontal pixel coordinates of the camera's principal point. The vertical pixel coordinates of the camera's principal point; The first pixel coordinates of the target vehicle in the raw image captured by the drone camera .

[0024] The world coordinate system is a globally fixed three-dimensional rectangular coordinate system, which serves as a reference for the absolute positions of the UAV and the target vehicle, and is used to characterize the spatial positions of the UAV and the target vehicle in real physical space. The UAV body coordinate system is a three-dimensional Cartesian coordinate system fixed to the UAV body. The origin changes synchronously with the flight position and attitude of the UAV and is used to describe the relative spatial position of the target vehicle relative to the UAV body. The camera coordinate system is a three-dimensional Cartesian coordinate system fixed to the UAV camera, with the camera optical center as the origin, used to connect the three-dimensional physical space coordinates of the body coordinate system and the two-dimensional planar coordinates of the image pixel coordinate system. The original image coordinate system is a two-dimensional discrete plane coordinate system based on the original image captured by the drone camera.

[0025] Coordinate transformation enables a precise mapping from the 3D physical space coordinates of the UAV and target vehicle to the 2D image pixel coordinates of the UAV camera. This transforms the positional information of the UAV and target vehicle into the first pixel coordinates in the image domain, providing a precise geometric position prediction foundation for subsequent multi-source coordinate fusion. The transformation process comprehensively considers practical engineering factors such as UAV attitude (yaw / pitch / roll angles), camera gimbal angle, camera installation translation offset, and camera radial / tangential distortion, significantly reducing systematic errors in coordinate transformation and improving the calculation accuracy of the first pixel coordinates.

[0026] In one specific implementation, the visual tracking module uses a visual tracking algorithm that is either a visual tracker based on correlation filtering or a visual tracker based on a Siamese network.

[0027] This embodiment employs a visual tracker based on correlation filtering, specifically a pre-trained KCF (kerneled correlation filter) tracker. The training process is as follows: The directional gradient histogram features are extracted from the region of interest image patch of the target vehicle in the original image of the initial frame. After cyclic shifting to generate a dense sample set, it is used as the training data of the KCF tracker. The Gaussian label of the target response (the center response value is 1 and the edge response value decreases linearly) is labeled as the true value of the tracking response of the KCF tracker. The training data of the KCF tracker is divided into training set and validation set. Using the training set as input, the hyperparameters of the KCF tracker are optimized by grid search combined with cross-validation. Based on the validation set, the optimization objective is to minimize the tracking response loss. The kernel function bandwidth and regularization coefficient are optimized to obtain the optimal combination of hyperparameters (the kernel ridge regression regularization coefficient is set to 0.0001 to prevent model overfitting; the Gaussian kernel function bandwidth is set to 0.5). During model training, the KCF tracker takes the first training set as input and outputs the correlation filter response map of the target search region. It uses Fast Fourier Transform combined with Kernel Ridge Regression closed-form solution to solve for the KCF tracker filter template parameters. The Kernel Ridge Regression loss function (the sum of least squares fitting loss and L2 regularization term) is used as the loss function. The first loss function is used to measure the fitting difference between the correlation filter response value output by the KCF tracker based on the training set and the true value of the tracking response during the current training process. Visual tracking training is carried out based on the training set with the goal of minimizing the tracking response loss. The training terminates when the closed-form solution of the filter template is completed.

[0028] Specifically, the KCF tracker is a single-target tracking algorithm. Its training is completed only in the first frame, and all training data comes from the target vehicle region in the initial frame, requiring no additional dataset or a large number of manually labeled samples. In the first frame of the original image captured by the drone, the region of interest of the target vehicle is manually / automatically selected. This rectangular image patch is the KCF tracker's only original training sample. In this embodiment, the body region of the target vehicle in the image is selected, with a width × height of 64 × 64 pixels. Since a single original sample is insufficient for training, the KCF tracker automatically generates a massive number of dense training samples through cyclic shift operations. For all samples generated by cyclic shifts, the oriented gradient histogram (HGP) features are extracted manually. These fixed-dimensional feature vectors constitute the formal training data for the KCF model.

[0029] A visual tracker based on kernel correlation filtering (KCF) is used to track target vehicles: First, the target vehicle region is selected in the initial frame and the correlation filter template is trained; in the original image of the subsequent frames, the image features of the target search region are extracted, and the peak position of the target response is obtained by correlation operation between the features and the filter template; the target vehicle is located based on this position, and finally the second pixel coordinate of the target vehicle in the original image is output.

[0030] KCF is a parametric traditional machine learning model that requires only the initial frame target region for training. It eliminates the need for large-scale datasets and offline training, resulting in low computational cost and fast tracking speed, achieving millisecond-level real-time visual tracking. This makes it suitable for the low-computing-power, high-frame-rate image acquisition requirements of drone-borne systems. KCF automatically generates dense training samples through cyclic shifting, combined with handcrafted features, demonstrating good adaptability to changes in the target vehicle's appearance, scale (within a certain range), and illumination, effectively reducing the probability of target loss and improving tracking stability in complex scenarios. KCF does not require the complex deployment and high computational power of deep neural networks; stable tracking can be achieved with only a lightweight algorithm, reducing the hardware cost and deployment difficulty of vehicle-mounted drones and exhibiting excellent engineering adaptability.

[0031] As a specific implementation method, the coordinate fusion module uses the following specific method to fuse the first pixel coordinates and the second pixel coordinates: The first pixel coordinates and the second pixel coordinates are fused using an extended Kalman filter.

[0032] The specific method for fusing the first pixel coordinates and the second pixel coordinates using an extended Kalman filter is as follows: Combining the first pixel coordinates and the uniform motion model in the extended Kalman filter, the prior value of the target vehicle's state vector at the current moment is predicted using the following formula: Target vehicle number State vector at time step , For the target vehicle The horizontal position in the image pixel coordinate system at any given time. For the target vehicle The vertical position of the image pixel coordinate system at any given time. For the target vehicle The horizontal velocity in the image pixel coordinate system at any given moment. For the target vehicle The vertical velocity in the image pixel coordinate system at any given moment; in, For the first The posterior estimate at time 1 for the 2nd time The prior prediction of the state vector at time step [time]. It is the first The state vector at time t becomes the first The state transition matrix used to predict the state vector at time step [time]. The target vehicle is number The state vector at time t, It is the noise in the target vehicle motion prediction process. It is the guidance gain for predicting the geometric position of the target vehicle. The value ranges from 0 to 1, and is used to adjust the correction weight of the target vehicle's geometric position prediction on the predicted state of the uniform motion model. The coordinates of the first pixel In the Coordinates of time, To map the state vector into an observation matrix of pixel observations, which is then used for matching with the second pixel coordinates and calculating residuals; Calculate the first Observe residuals at all times ,in, It is the coordinate of the second pixel. In the The coordinates of time; Calculate the first Time Kalman Gain ,in, Based on the The posterior covariance at time t on the first The prior prediction covariance matrix of the state at time step. It is the transpose symbol. It is the first The noise covariance matrix of visual observations at any given time; Based on the Time Kalman Gain With the Observe residuals at all times Correct the prior prediction value , to obtain the posterior estimate + ; Update the posterior covariance matrix Finally, the position component is extracted from the posterior estimate to obtain the estimated pixel coordinates. , It is the estimated horizontal pixel coordinate of the target vehicle in the original image coordinate system. It is the estimated vertical pixel coordinate of the target vehicle in the original image coordinate system.

[0033] This scheme adds a geometric position prediction-guided correction term to the standard prediction equation. A dual-drive prediction mechanism combining uniform motion model prediction and geometric position guidance was constructed. This solves the problem of prediction error accumulation and drift caused by the standard EKF relying solely on the motion model, due to the target motion deviating from the uniform motion assumption (such as vehicle acceleration, lane change, and sudden braking) and visual failure. The absolute physical position information provided by the geometric position can correct the model error and accumulated deviation of the motion model in real time, ensuring that the predicted state always converges to the true target position, thus fundamentally avoiding tracking loss. The uniform motion model relies on the target motion assumption and has inherent model error, which will continue to accumulate without external calibration; the geometric position provides absolute position information and has no accumulated drift. The complementary advantages of the two significantly improve prediction accuracy, especially in target maneuvering scenarios. Adjustable prediction guidance gain can be set. Dynamic adjustment The weights of the guiding terms are adjusted to achieve optimal adaptation in different scenarios: when visual tracking is normal (effective observation, low noise), the weights are reduced. This approach weakens geometric position guidance, allowing prediction to primarily rely on motion models and visual updates. It suppresses the interference of large geometric position noise on state estimation, ensuring tracking accuracy. When visual tracking fails (e.g., due to target occlusion, image blurring, or loss during rapid maneuvers), the [measurement method] is increased. It enhances geometric position guidance, allowing predictions to converge towards absolute geometric positions, ensuring tracking continuity, and achieving a balance between accuracy and robustness. It is perfectly adapted to complex road scenarios (urban canyons, tunnels, obstructions, backlighting, etc.) for vehicle-mounted drone tracking.

[0034] Coordinate fusion achieves complementary advantages between the first pixel coordinates predicted by geometry and the second pixel coordinates observed visually, overcoming the shortcomings of a single data source: when the signal is poor (such as in an urban canyon), the error of the first pixel coordinate increases, but the second pixel coordinates observed visually can provide accurate correction; when the target is briefly occluded or the image is blurred, the second pixel coordinate becomes ineffective, but the EKF can continuously estimate the target state through geometric prediction of the first pixel coordinate, solving the problem of easy failure of a single data source in traditional tracking and significantly improving the robustness of pixel coordinate estimation. The extended Kalman filter has dynamic capabilities for state prediction and observation updates, which can smooth the pixel position changes of the target vehicle, suppress random noise in visual observation and localization, and obtain more stable estimated pixel coordinates, providing a jitter-free and reliable input for subsequent control command generation.

[0035] As a specific implementation method, the method for generating outer loop control commands based on the positional relationship between the estimated pixel coordinates and the preset center region of the original image in the command generation module is as follows: Calculate the pixel deviations between the estimated pixel coordinates and the center of the original image in the horizontal and vertical directions, respectively. When the magnitude of the pixel deviation is greater than a first deviation threshold, generate yaw rate and pitch rate control commands for the UAV or camera gimbal using the PD control method (proportional-derivative control method) based on the pixel deviations. The yaw rate control command is used to correct the horizontal pixel deviation of the target vehicle in the original image, and the pitch rate control command is used to correct the vertical pixel deviation of the target vehicle in the original image.

[0036] For example, the PD control law for the yaw channel: , The command is a control command for the desired angular velocity around the yaw axis. If controlling the UAV itself, this command is sent to the UAV flight control system as the UAV's yaw angular velocity to adjust the UAV's left and right yaw attitude; if controlling the gimbal, this command is sent to the gimbal servo system as the gimbal's horizontal rotation angular velocity to adjust the gimbal's left and right turning angle. It is an outer loop proportional control gain. It is the horizontal pixel deviation. It is the outer-loop differential control gain. It is horizontal pixel deviation The derivative with respect to time.

[0037] If the target vehicle's pixel deviation from the preset center area of ​​the original image is within the effective rotation angle range of the camera gimbal's yaw and pitch, the outer loop control commands are only sent to the gimbal servo unit. The pixel deviation is corrected by rotating the gimbal, without needing to adjust the UAV's flight attitude, thus balancing tracking accuracy and flight stability while reducing energy consumption and suppressing image jitter. If the target pixel deviation exceeds the maximum effective rotation angle range of the camera gimbal (the camera gimbal reaches its mechanical limit), the gimbal cannot further correct the deviation. At this time, the outer loop control commands are switched to the UAV flight control unit, which adjusts the UAV's yaw and pitch flight attitude to pull the target back to the preset center area of ​​the original image. Once the deviation returns to the gimbal's effective range, the control switches back to camera gimbal control.

[0038] The PD control method combines the fast response of proportional control with the damping and vibration reduction of derivative control. It can quickly correct the horizontal / vertical pixel deviation of the target vehicle in the original image, while avoiding attitude overshoot of the UAV / gimbal, significantly reducing image jitter caused by frequent attitude adjustments and improving tracking stability. A first threshold is set for the pixel deviation modulus; control commands are only generated when the deviation exceeds the threshold, avoiding frequent UAV / gimbal movements caused by small deviations, effectively reducing UAV energy consumption and increasing endurance. Yaw angular velocity commands correspond to horizontal pixel deviation, and pitch angular velocity commands correspond to vertical pixel deviation, achieving independent control in two dimensions. This results in more precise correction of target pixel deviation, ensuring the target vehicle remains in the preset center area of ​​the original image, solving the problem of the target easily moving out of the camera's field of view in traditional tracking. The outer loop control, targeting the overall field of view of the original image from the UAV camera, uses a hard control method of physical attitude / angle adjustment, providing a stable large field of view foundation for the image display window cropping in the inner loop screen, preventing tracking loss due to the target moving out of the camera's field of view.

[0039] As a specific implementation, the method for generating inner-loop control commands in the command generation module based on the positional relationship between the estimated pixel coordinates and the center coordinates of the display window of the image software on the current screen in the original image is as follows: The estimated pixel coordinates are calculated as follows: the horizontal and vertical pixel deviations between the estimated pixel coordinates and the center coordinates of the display window of the image software on the current screen in the original image are calculated. The goal is to make the estimated pixel coordinates of the target vehicle coincide with the coordinates of the center of the display window of the image software on the current screen in the original image. Based on the pixel deviations, the new coordinates of the target cropping position after translation of the display window of the image software on the current screen in the upper left corner of the original image are calculated. It is determined whether the new coordinates exceed the effective cropping boundary of the original image. If they do not exceed the boundary, the display window of the image software on the screen is translated to the cropping position defined by the new coordinates. If they exceed the boundary, the new coordinates are clamped to the effective cropping boundary of the original image, and an outer loop control command is regenerated and sent to adjust the flight attitude of the UAV or the rotation angle of the camera gimbal so that the estimated pixel coordinates of the target vehicle are in the preset center area of ​​the original image.

[0040] The target is defined as the estimated pixel coordinates of the target vehicle coinciding with the coordinates of the center of the display window of the image software in the original image on the current screen. That is, the pixel deviations between the estimated pixel coordinates and the coordinates of the center of the display window of the image software in the original image in the horizontal and vertical directions are equal to 0. The new coordinates of the target cropping position in the original image after the translation of the display window of the image software in the current screen are calculated as follows: , in, and These are the new pixel coordinates in the horizontal direction of the top left corner of the original image and the new pixel coordinates in the horizontal direction of the target cropping position after the current image software display window is translated, respectively. This represents the target position of the window translation. and These are the width (number of pixels in the horizontal direction) and height (number of pixels in the vertical direction) of the display window of the image software on the screen.

[0041] For the calculated and The core function of first-order low-pass filtering is to: when the target vehicle moves quickly, directly translating the window will cause the screen to stutter and flicker. First-order low-pass filtering makes the window position transition smoothly, improving the display effect; it also filters out high-frequency noise in the estimated target coordinates, avoids frequent small-amplitude window jitter, and ensures the stability of tracking.

[0042] When the target vehicle shifts to the right in the current window Increase, calculate the larger value according to the above formula. To move the target vehicle back to the center of the window, control the window to pan to the right; when the target vehicle shifts upwards within the current window, Decrease, and calculate a smaller value according to the above formula. The control window is panned upwards to correct the target's vertical offset. Throughout the process, only the cropped display area of ​​the image is adjusted; the physical attitude of the drone / gimbal remains unchanged, representing software-level visual tracking compensation.

[0043] Calculated according to the above formula and It may exceed the effective cropping range of the original image, therefore clamping (boundary constraint) is required, with the constraint condition: 0. 0 , and These are the total width (total number of pixels in the horizontal direction) and the total height (total number of pixels in the vertical direction) of the original image, respectively.

[0044] If clamping occurs (i.e., the calculated...) and / or If the target vehicle is out of the constraint range and forcibly restricted to the boundary, it means that the target vehicle is about to move out of the display window of the image software on the screen. The inner ring can no longer center the target by translating the window. At this time, a boundary warning signal is sent to the outer ring controller, which triggers the outer ring controller to generate angular velocity control commands for the UAV / gimbal, adjusts the physical attitude / angle, and brings the target back to the center area of ​​the original image to achieve continuous tracking.

[0045] The inner loop employs software-level window translation and cropping control, eliminating the need to adjust the physical attitude of the drone / gimbal. This enables pixel-level precision imaging, ensuring the target vehicle remains centered within the image software's display window. This solves the problems of target misalignment and poor composition caused by traditional simple image cropping, improving the visual quality of image transmission. An effective cropping boundary judgment and coordinate clamping mechanism are introduced, clearly defining the control boundaries between the inner loop's software control and the outer loop's hardware control: the outer loop's physical attitude adjustment is only re-triggered when the window translation exceeds the effective cropping range of the original image, avoiding unnecessary drone / gimbal movements and further reducing energy consumption and image jitter. The target cropping position calculation for window translation is based on targets with zero pixel deviation, and boundary constraints are applied to the new coordinates, ensuring the effectiveness of the cropping operation and preventing invalid cropping where the display window exceeds the original image, thus improving system reliability. The cascaded control architecture of inner and outer loops solves the core pain points of existing technologies, such as tracking loss, poor mapping, or frequent attitude adjustments, while taking into account the high accuracy and robustness of tracking and the low energy consumption and low jitter of execution.

[0046] This invention employs a layered control strategy of "UAV / gimbal + window panning". The UAV / gimbal ensures the target remains within the camera's wide field of view, while window panning performs pixel-level fine-tuning, ultimately locking the target stably in the center of the screen, effectively eliminating image jitter and improving the viewing experience. The window panning control module includes a boundary judgment mechanism. Only when the target is about to move out of the current display window is the UAV / gimbal triggered to make a significant adjustment, avoiding unnecessary frequent movements of the UAV or gimbal, reducing energy consumption, and increasing battery life. This invention provides a complete and detailed algorithm flow and system architecture from coordinate transformation and data fusion to control decision-making, which can be directly applied to existing vehicle-mounted UAV hardware platforms and has high engineering value.

[0047] Example 2 A vehicle tracking method for vehicular unmanned aerial vehicles based on visual servo control, comprising: The three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system are obtained. Through the preset multi-level coordinate system transformation relationship, the first pixel coordinates of the target vehicle in the original image captured by the UAV camera are calculated based on the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system. The second pixel coordinates of the target vehicle in the original image are detected using a visual tracking algorithm; The first pixel coordinates and the second pixel coordinates are fused to obtain the estimated pixel coordinates of the target vehicle in the original image captured by the drone camera; Based on the positional relationship between the estimated pixel coordinates and the preset center area of ​​the original image, an outer loop control command is generated. The outer loop control command is sent to the UAV flight control unit to control the flight attitude of the UAV, or sent to the camera gimbal servo unit to control the rotation angle of the camera gimbal, so that the target vehicle remains in the preset center area of ​​the original image. The updated estimated pixel coordinates are obtained after adjustment according to the outer loop control command. An inner loop control command is generated based on the positional relationship between the center coordinates of the display window of the image software on the current screen in the original image and the updated estimated pixel coordinates. The inner loop control command is sent to the image processing unit to adjust the cropping position of the display window of the image software on the current screen in the original image, so that the center coordinates of the display window of the image software on the current screen in the original image coincide with the updated estimated pixel coordinates.

[0048] Example 3 A computer program product includes a computer program / instructions that, when executed by a processor, implement the vehicle tracking method for an onboard drone based on visual servo control as described in Embodiment 2.

[0049] The contents not described in detail in this specification are prior art known to those skilled in the art. Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0051] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0052] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the invention.

Claims

1. A vehicle tracking system for vehicle-mounted unmanned aerial vehicles based on visual servo control, characterized in that, include: The coordinate transformation module is used to calculate the first pixel coordinate of the target vehicle in the original image captured by the drone camera based on the three-dimensional position coordinates of the drone and the target vehicle in the world coordinate system through a preset multi-layer coordinate system transformation relationship. The visual tracking module is used to detect the second pixel coordinates of the target vehicle in the original image using a visual tracking algorithm; The coordinate fusion module is used to fuse the first pixel coordinates and the second pixel coordinates to obtain the estimated pixel coordinates of the target vehicle in the original image captured by the drone camera. The instruction generation module is used to generate an outer loop control instruction based on the positional relationship between the estimated pixel coordinates and the preset center area of ​​the original image. This outer loop control instruction is sent to the UAV flight control unit to control the UAV's flight attitude, or to the camera gimbal servo unit to control the rotation angle of the camera gimbal, so that the target vehicle remains within the preset center area of ​​the original image. The estimated pixel coordinates are updated after adjustment based on the outer loop control instruction. An inner loop control instruction is generated based on the positional relationship between the coordinates of the center of the image software's display window in the original image and the updated estimated pixel coordinates. This inner loop control instruction is sent to the image processing unit to adjust the cropping position of the image software's display window in the original image, so that the coordinates of the center of the image software's display window in the original image coincide with the updated estimated pixel coordinates.

2. The vehicle tracking system for vehicle-mounted unmanned aerial vehicles based on visual servo control according to claim 1, characterized in that, In the coordinate transformation module, the specific method for calculating the first pixel coordinate of the target vehicle in the original image captured by the UAV camera based on the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system through a preset multi-layer coordinate system transformation relationship is as follows: Based on the three-dimensional position coordinates of the UAV and the target vehicle in the world coordinate system, the three-dimensional position coordinates of the target vehicle in the UAV volume coordinate system are calculated. The conversion formula is as follows: in, The three-dimensional position coordinates of the target vehicle in the UAV body coordinate system. The three-dimensional position coordinates of the target vehicle in the world coordinate system. The three-dimensional position coordinates of the UAV in the world coordinate system. Let be the rotation matrix from the world coordinate system to the UAV body coordinate system. , and These are the UAV yaw angle, UAV pitch angle, and UAV roll angle, respectively, relative to the world coordinate system of the UAV body coordinate system. The 3D position coordinates of the target vehicle in the UAV volume coordinate system are converted to 3D position coordinates in the camera coordinate system using the following formula: in, Let be the three-dimensional position coordinates of the target vehicle in the camera coordinate system. Let be the rotation matrix from the UAV body coordinate system to the camera coordinate system. and These are the camera gimbal pitch angle and camera gimbal yaw angle, respectively, relative to the UAV body coordinate system. This represents the three-dimensional translational offset of the camera's optical center relative to the origin of the UAV's body coordinate system. The 3D position coordinates of the target vehicle in the camera coordinate system are converted to pixel coordinates in the original image coordinate system captured by the UAV camera. The conversion formula is as follows: ， in, For the camera intrinsic parameter matrix, The horizontal pixel coordinates of the target vehicle in the original image coordinate system. The vertical pixel coordinates of the target vehicle in the original image coordinate system. This represents the normalized horizontal coordinates of the target vehicle in the camera coordinate system after perspective projection and correction for radial and tangential distortion. This represents the normalized vertical coordinates of the target vehicle in the camera coordinate system after perspective projection and correction for radial and tangential distortion. The equivalent focal length of the camera in the horizontal direction. The equivalent focal length in the vertical direction of the camera. The horizontal pixel coordinates of the camera's principal point. The vertical pixel coordinates of the camera's principal point; The first pixel coordinates of the target vehicle in the raw image captured by the drone camera .

3. The vehicle tracking system for vehicle-mounted unmanned aerial vehicles based on visual servo control according to claim 2, characterized in that: The world coordinate system is a globally fixed three-dimensional rectangular coordinate system, which serves as a reference for the absolute positions of the UAV and the target vehicle, and is used to characterize the spatial positions of the UAV and the target vehicle in real physical space. The UAV body coordinate system is a three-dimensional Cartesian coordinate system fixed to the UAV body. The origin changes synchronously with the flight position and attitude of the UAV and is used to describe the relative spatial position of the target vehicle relative to the UAV body. The camera coordinate system is a three-dimensional Cartesian coordinate system fixed to the UAV camera, with the camera optical center as the origin, used to connect the three-dimensional physical space coordinates of the body coordinate system and the two-dimensional planar coordinates of the image pixel coordinate system. The original image coordinate system is a two-dimensional discrete plane coordinate system based on the original image captured by the drone camera.

4. The vehicle tracking system for vehicle-mounted unmanned aerial vehicles based on visual servo control according to claim 1, characterized in that: In the visual tracking module, the visual tracking algorithm is either a visual tracker based on correlation filtering or a visual tracker based on Siamese networks.

5. The vehicle tracking system for vehicle-mounted unmanned aerial vehicles based on visual servo control according to claim 1, characterized in that, The specific method for fusing the first pixel coordinates and the second pixel coordinates in the coordinate fusion module is as follows: The first pixel coordinates and the second pixel coordinates are fused using an extended Kalman filter, specifically as follows: Combining the first pixel coordinates and the uniform motion model in the extended Kalman filter, the prior value of the target vehicle's state vector at the current moment is predicted using the following formula: Target vehicle number State vector at time step , For the target vehicle The horizontal position in the image pixel coordinate system at any given time. For the target vehicle The vertical position of the image pixel coordinate system at any given time. For the target vehicle The horizontal velocity in the image pixel coordinate system at any given moment. For the target vehicle The vertical velocity in the image pixel coordinate system at any given moment; in, For the first The posterior estimate at time 1 for the 2nd time The prior prediction of the state vector at time step [time]. It is the first The state vector at time t becomes the first The state transition matrix used to predict the state vector at time step [time]. The target vehicle is number The state vector at time t, It is the noise in the target vehicle motion prediction process. It is the target vehicle geometric position prediction guidance gain, used to adjust the correction weight of the target vehicle geometric position prediction on the predicted state of the uniform motion model. The coordinates of the first pixel In the Coordinates of time, To map the state vector into an observation matrix of pixel observations, which is then used for matching with the second pixel coordinates and calculating residuals; Calculate the first Observe residuals at all times ,in, It is the coordinate of the second pixel. In the The coordinates of time; Calculate the first Time Kalman Gain ,in, Based on the The posterior covariance at time t on the first The prior prediction covariance matrix of the state at time step. It is the transpose symbol. It is the first The noise covariance matrix of visual observations at any given time; Based on the Time Kalman Gain With the Observe residuals at all times Correct the prior prediction value , to obtain the posterior estimate + ; Update the posterior covariance matrix Finally, the position component is extracted from the posterior estimate to obtain the estimated pixel coordinates. , It is the estimated horizontal pixel coordinate of the target vehicle in the original image coordinate system. It is the estimated vertical pixel coordinate of the target vehicle in the original image coordinate system.

6. The vehicle tracking system for vehicle-mounted unmanned aerial vehicles based on visual servo control according to claim 1, characterized in that, In the instruction generation module, the specific method for generating outer loop control instructions based on the positional relationship between the estimated pixel coordinates and the preset center region of the original image is as follows: Calculate the pixel deviations between the estimated pixel coordinates and the center of the original image in the horizontal and vertical directions, respectively. When the magnitude of the pixel deviation is greater than a first deviation threshold, generate yaw rate and pitch rate control commands for the UAV or camera gimbal using the PD control method based on the pixel deviations. The yaw rate control command is used to correct the horizontal pixel deviation of the target vehicle in the original image, and the pitch rate control command is used to correct the vertical pixel deviation of the target vehicle in the original image.

7. The vehicle tracking system for vehicle-mounted unmanned aerial vehicles based on visual servo control according to claim 1, characterized in that, In the instruction generation module, the specific method for generating inner-loop control instructions based on the coordinates of the center of the image software's display window in the original image and the positional relationship between the updated estimated pixel coordinates is as follows: Calculate the coordinates of the center of the display window of the image software in the original image and the pixel deviations in the horizontal and vertical directions of the updated estimated pixel coordinates. Take the alignment of the center of the display window of the image software in the original image with the updated estimated pixel coordinates as the target. Calculate the new coordinates of the top-left corner of the target cropping position of the display window of the image software in the original image after translation based on the updated pixel deviation. If the new coordinates of the top-left corner do not exceed the effective cropping boundary of the original image, then translate the display window of the image software in the screen to the cropping position defined by the new coordinates.

8. The vehicle tracking system for vehicle-mounted unmanned aerial vehicles based on visual servo control according to claim 7, characterized in that: If the new coordinates of the upper left corner exceed the effective cropping boundary of the original image, the new coordinates will be clamped to the effective cropping boundary of the original image, and the outer loop control command will be regenerated and sent to adjust the flight attitude of the UAV or the rotation angle of the camera gimbal so that the target vehicle remains in the preset center area of ​​the original image.

9. A vehicle tracking method for vehicle-mounted unmanned aerial vehicles based on visual servo control, characterized in that, include: By using a preset multi-layer coordinate system transformation relationship, the first pixel coordinate of the target vehicle in the original image captured by the drone camera is calculated based on the three-dimensional position coordinates of the drone and the target vehicle in the world coordinate system. The second pixel coordinates of the target vehicle in the original image are detected using a visual tracking algorithm; The first pixel coordinates and the second pixel coordinates are fused to obtain the estimated pixel coordinates of the target vehicle in the original image captured by the drone camera; Based on the positional relationship between the estimated pixel coordinates and the preset center area of ​​the original image, an outer loop control command is generated. The outer loop control command is sent to the UAV flight control unit to control the flight attitude of the UAV, or sent to the camera gimbal servo unit to control the rotation angle of the camera gimbal, so that the target vehicle remains in the preset center area of ​​the original image. The updated estimated pixel coordinates are obtained after adjustment according to the outer loop control command. An inner loop control command is generated based on the positional relationship between the center coordinates of the display window of the image software on the current screen in the original image and the updated estimated pixel coordinates. The inner loop control command is sent to the image processing unit to adjust the cropping position of the display window of the image software on the current screen in the original image, so that the center coordinates of the display window of the image software on the current screen in the original image coincide with the updated estimated pixel coordinates.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the vehicle tracking method for vehicle-mounted unmanned aerial vehicles based on visual servo control as described in claim 9.

Images