Unmanned aerial vehicle load target tracking system and method based on multi-sensor fusion

Abstract

The invention discloses an unmanned aerial vehicle load target tracking system and method based on multi-sensor fusion, and the method comprises the steps: obtaining visible light, infrared and laser radar data, carrying out the time sequence synchronization and attitude calibration, and obtaining a target fusion estimation value through a dynamic weighting Bayesian fusion algorithm; the improved Siamese neural network is used for feature matching, the LSTM trajectory predictor is combined to output a short-term motion state, and finally the adaptive mechanical control module drives the holder to realize stable tracking. According to the invention, the robustness, precision and continuity of target tracking in a complex environment are effectively improved.

Description

Technical Field

[0001] This invention relates to unmanned aerial vehicle (UAV) control and intelligent decision-making technology, and in particular to a UAV payload target tracking system and method based on multi-sensor fusion. Background Technology

[0002] With the rapid development of UAV technology, its application scope has expanded from early aerial photography and simple reconnaissance to multiple fields such as military confrontation, disaster emergency rescue, urban security patrol, and intelligent traffic monitoring. In these application scenarios, target recognition and continuous tracking capabilities have become key performance indicators for UAV payload systems. Traditionally, single sensors (such as visible light cameras) have been widely used due to their intuitive imaging and high resolution. However, under conditions of night, rain, fog, smoke, or complex background interference, their imaging quality and target visibility significantly decrease, severely limiting the robustness of the system. To compensate for this deficiency, researchers have gradually introduced multi-source sensors such as infrared thermal imaging and lidar to obtain more comprehensive and stable target information. At the same time, with the development of deep learning and temporal prediction methods, target tracking algorithms are gradually evolving from traditional correlation filtering (such as KCF) to intelligent architectures based on convolutional neural networks (CNN), Siamese networks, and long short-term memory networks (LSTM). These algorithms have shown strong potential in handling target scale changes, motion trajectory prediction, and short-term occlusion recovery. Furthermore, high-precision three-axis gimbal and servo control technologies are continuously improving to address the dynamic environmental characteristics of UAV platforms. Some research has attempted to combine predictive control models to shorten response latency, thereby enhancing target retention capabilities. These advancements collectively drive the development of UAV target tracking technology, namely, building more efficient and stable integrated solutions centered on multi-sensor fusion and intelligent control.

[0003] However, despite the positive progress made in multi-source data fusion and intelligent tracking algorithms, several technical bottlenecks remain to be addressed. First, current multi-sensor data fusion methods are mostly limited to simple feature-level stitching or weighted averaging, failing to fully consider the dimensions, temporal characteristics, and environmental adaptability of different sensor data, resulting in unsatisfactory fusion effects. Under conditions of strong noise, dynamic lighting, or rapid target movement, tracking accuracy and continuity still significantly decrease. Second, common target tracking algorithms are prone to drift or target loss when faced with prolonged occlusion, complex background interference, or drastic changes in target appearance, exhibiting limited recovery capabilities. Third, existing gimbal control systems mostly rely on empirically tuned PID algorithms, lacking deep coupling with target prediction results, easily leading to response lag and difficulty in achieving stable control under high dynamic conditions. Furthermore, the overall system structure is complex and power consumption is relatively high, which is detrimental to the long-term endurance and flexible deployment of UAV platforms. Therefore, there is an urgent need to propose a UAV payload target tracking system with high integration, low power consumption, and the ability to combine multi-sensor adaptive fusion, deep learning prediction, and feedforward servo control, so as to achieve all-weather, continuous, and high-precision target tracking in complex environments. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application provides a method for UAV payload target tracking based on multi-sensor fusion, comprising the following steps: Step S1. Acquire observation data from the visible light camera, infrared thermal imager, and lidar, and perform time synchronization and attitude calibration on each observation data. Step S2. Obtain the fusion estimate of the target using a dynamically weighted Bayesian fusion algorithm; Step S3. Input the fused estimate into the target matching module based on the improved Siamese neural network to extract features and calculate similarity; Step S4. Use a long short-term memory network predictor to make short-term trajectory predictions, and use the prediction results as feedforwards to input into the adaptive mechanical control module to adjust the gimbal angle and achieve continuous and stable tracking of the target.

[0005] As a preferred technical solution for UAV payload target tracking based on multi-sensor fusion, the fusion estimate of the dynamically weighted Bayesian fusion algorithm is calculated according to the following formula:

[0006] in, For the first Observations from each sensor; For the first The dynamic weights of each sensor satisfy... ; This is the estimated value of the target after fusion.

[0007] As a preferred technical solution for UAV payload target tracking based on multi-sensor fusion, the dynamic weights Calculate using the following formula:

[0008] in, For the first The correlation coefficient between sensor observations and predicted values; This is an environmental adaptive factor used to reflect the impact of the current environment on sensor performance.

[0009] As a preferred technical solution for UAV payload target tracking based on multi-sensor fusion, the environmental adaptive factor Adjust dynamically according to the following formula:

[0010] in, The forgetting factor is a real number between 0 and 1, used to balance historical images; The correlation coefficient between observed and predicted values; As an indicator function, when the sensor quality factor Less than the threshold Take 1 at the time; This is the weight suppression constant, used to reduce the weight of low-quality sensors.

[0011] As a preferred technical solution for UAV payload target tracking based on multi-sensor fusion, to improve robustness in low-light or high-noise environments, an information entropy correction term is further introduced in the weight calculation, wherein the weight is:

[0012] in, For the first The information entropy of sensor data is used to measure uncertainty. The maximum information entropy value set; item Used to reduce the impact of noisy sensors.

[0013] As a preferred technical solution for UAV payload target tracking based on multi-sensor fusion, the Siamese neural network uses MobileNetV3 as the feature extraction backbone and performs feature alignment through a channel attention module. Its similarity calculation formula is as follows:

[0014] in, Feature mapping for candidate regions; Feature mapping for template targets; For cross-correlation operators;

[0015] in, This is the channel attention weight vector; This is a channel-by-channel product.

[0016] As a preferred technical solution for UAV payload target tracking based on multi-sensor fusion, the matching result output by Siamese is combined with an unscented Kalman filter (UKF) for nonlinear state filtering, and its state update formula is as follows:

[0017] in, For the first One Sigma sampling point; The mean weighting coefficient; Let be the state dimension.

[0018] As a preferred technical solution for UAV payload target tracking based on multi-sensor fusion, the LSTM predictor uses the target's state vector As input, the hidden state update formula is:

[0019] The predicted output is:

[0020] in, These refer to the target position and velocity, respectively. For attitude angle; For the gating activation function of the LSTM unit; It is in a hidden state; To predict the location; To predict attitude angles.

[0021] As a preferred technical solution for a UAV payload target tracking method based on multi-sensor fusion, the control equation of the adaptive mechanical control module is:

[0022] in, The gimbal controls the output torque; For target position error; The rate of change of error; The feedforward position is predicted by LSTM; The desired attitude angle; To predict attitude angles; These are the proportional, derivative, feedforward, and attitude gain coefficients, respectively.

[0023] According to the method described in claim 9, the controller achieves high dynamic stability control through an active damping algorithm and disturbance compensation, wherein the damping equation is:

[0024] in, Here is the damping coefficient matrix; The angular velocity of the gimbal; The desired angular velocity; Here is the moment of inertia matrix of the gimbal; Rate of change of angular velocity of the aircraft; The final control output is , The control system has a pitch range of ±90°, a roll range of ±45°, and a 360° continuous yaw rotation, with a control accuracy better than 0.01°.

[0025] This application effectively overcomes the performance degradation limitations of single sensors in complex environments such as nighttime and rain / fog by using dynamic weighted Bayesian fusion of multiple sensors (visible light camera, infrared thermal imager, and lidar), significantly improving the robustness and environmental adaptability of target tracking. It employs an improved Siamese neural network (based on MobileNetV3 and incorporating a channel attention mechanism) to achieve high-precision, lightweight target matching, effectively addressing changes in target appearance and background interference, and enhancing the recovery capability against occlusion and deformation. Combining an LSTM trajectory prediction module and an adaptive mechanical control module, it implements feedforward-based gimbal servo control. Through active damping and disturbance compensation strategies, it significantly reduces response latency and tracking jitter, ultimately achieving high-precision, continuous, and stable tracking better than 0.01° within the gimbal's range of pitch ±90°, roll ±45°, and heading 360°. This provides an all-weather, highly reliable target tracking solution for UAVs in complex scenarios such as military, emergency response, and inspection. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a UAV payload target tracking method based on multi-sensor fusion according to the present invention. Figure 2 This is a schematic diagram illustrating the principle of a UAV payload target tracking system based on multi-sensor fusion according to the present invention. Detailed Implementation

[0027] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the examples in the specification.

[0028] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0029] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places throughout this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments.

[0030] Example 1 This embodiment provides a UAV payload target tracking method based on multi-sensor fusion. This method achieves continuous and stable target tracking by acquiring multi-source sensor data, dynamically fusing and processing it, performing feature matching, trajectory prediction, and adaptive control. The following is combined with... Figure 1 and Figure 2 Describe in detail the implementation process of this method, where Figure 1 This is a flowchart illustrating a UAV payload target tracking method based on multi-sensor fusion according to the present invention. Figure 2 This is a schematic diagram illustrating the principle of a UAV payload target tracking system based on multi-sensor fusion according to the present invention.

[0031] First, in step S1, observation data from the visible light camera, infrared thermal imager, and lidar are acquired, and the data is synchronized in time and calibrated in attitude. Specifically, the UAV payload system carries a visible light camera to capture high-resolution color images, an infrared thermal imager to detect heat source signals in low-light or nighttime environments, and a lidar to provide accurate 3D distance and point cloud data. These sensors may have different data acquisition frequencies (e.g., 30Hz for the visible light camera, 25Hz for the infrared thermal imager, and 10Hz for the lidar), therefore, time synchronization is achieved through timestamp alignment. Simultaneously, due to changes in the UAV's flight attitude (e.g., pitch, roll, and heading), attitude calibration is performed using IMU (Inertial Measurement Unit) data to transform the observation data from each sensor into a unified coordinate system. The mechanism of this step is that multi-sensor data often exhibits time discrepancies and coordinate inconsistencies; without synchronization and calibration, subsequent fusion errors can be amplified. By using time synchronization (e.g., linear interpolation) and attitude calibration (e.g., rotation matrix transformation), data consistency is ensured, thereby improving the robustness and accuracy of the overall system. The beneficial effects are that it reduces data mismatch in complex dynamic environments, improves the reliability of target detection, and avoids the failure of traditional single-sensor methods under time or attitude shifts.

[0032] Next, in step S2, a dynamically weighted Bayesian fusion algorithm is used to obtain the fusion estimate of the target. The specific fusion estimate is calculated using the following formula:

[0033] in, For the first Observations from each sensor; For the first The dynamic weights of each sensor satisfy... ; This is the estimated value of the fused target. The mechanism of this formula is based on the Bayesian probabilistic framework, which achieves optimized estimation of the target's position, velocity, and other states by weighted summation and fusion of multi-source data.

[0034] Dynamic weights Calculate using the following formula:

[0035] in, For the first The correlation coefficient between sensor observations and predicted values; This is an environmental adaptive factor used to reflect the impact of the current environment on sensor performance.

[0036] It should be noted that the reliability of sensor data is assessed through correlation coefficients, and environmental factors are introduced to dynamically adjust the weights, avoiding the failure of fixed weights in changing environments. For example, in clear weather, the weight of visible light cameras is higher; in foggy weather or at night, the weights of infrared and lidar are increased, improving the adaptability of fusion and reducing noise interference compared to traditional average fusion, thus improving the accuracy of target estimation.

[0037] Furthermore, environmental adaptive factors Adjust dynamically according to the following formula:

[0038] in, The forgetting factor is a real number between 0 and 1, used to balance historical images; The correlation coefficient between observed and predicted values; As an indicator function, when the sensor quality factor Less than the threshold Take 1 at the time; This is the weight suppression constant, used to reduce the weight of low-quality sensors.

[0039] It should be noted that a recursive approach combined with a forgetting factor is used to achieve smooth adjustment, while an indicator function is used to penalize low-quality data (such as when noise is too high). This allows for rapid adaptation to sensor performance during rapid environmental changes (such as from day to night), reducing fusion errors and improving the system's robustness in low-light or high-noise environments.

[0040] To further improve robustness in low-light or high-noise environments, an information entropy correction term is introduced into the weight calculation. The weight is:

[0041] in, For the first The information entropy of sensor data is used to measure uncertainty. The maximum information entropy value set; item Used to reduce the impact of noisy sensors.

[0042] It should be noted that information entropy is used. Quantitative First The randomness and uncertainty of individual sensor data are compared with the maximum information entropy value. ratio Generate correction factor The weights of each sensor in the fusion process are dynamically adjusted. In low-light or high-noise environments (such as at night or in fog), the data uncertainty from high-noise sensors (such as visible light cameras) increases, leading to... Increasing the value of an unreliable sensor reduces its weight, prioritizing the fusion of more reliable data from sources like infrared thermal imagers or lidar. This mechanism adaptively mitigates the influence of unreliable sensors, ensuring the quality of the fused estimate. This ensures the accuracy of target tracking, thereby maintaining robustness and stability in complex environments.

[0043] In step S3, the fused estimate is input into the target matching module based on the improved Siamese neural network to extract features and calculate similarity. Specifically, the Siamese neural network uses MobileNetV3 as the feature extraction backbone (lightweight design reduces computational burden) and performs feature alignment through a channel attention module. Its similarity calculation formula is as follows:

[0044] in, Feature mapping for candidate regions; Feature mapping for template targets; For cross-correlation operators;

[0045] in, This is the channel attention weight vector; This is a channel-by-channel product.

[0046] It should be noted that by inputting the fused estimated values ​​into the target matching module based on the improved Siamese neural network and using MobileNetV3 as the feature extraction backbone network, the number of model parameters and computational complexity are significantly reduced while maintaining recognition accuracy, making it suitable for platforms with limited computing power, such as UAVs. The introduction of a channel attention module adaptively assigns weights to the features of the template target and candidate regions, effectively enhancing the response of key channels and suppressing background interference, thereby improving the accuracy and discriminative power of feature alignment. This structure uses cross-correlation operations to measure the similarity between features, enabling the network to maintain stable matching performance under changes in target appearance, scale variations, and complex backgrounds. Compared with the traditional Siamese structure, this improved scheme significantly improves target loss rate, mismatch rate, and real-time performance, enhancing the robustness of the system in target recognition and tracking scenarios with multi-source fusion.

[0047] Furthermore, the controller achieves high dynamic stability control through an active damping algorithm and disturbance compensation, with the damping equation being:

[0048] in, Here is the damping coefficient matrix; The angular velocity of the gimbal; The desired angular velocity; Here is the moment of inertia matrix of the gimbal; Rate of change of angular velocity of the aircraft; The final control output is , The control system has a pitch range of ±90°, a roll range of ±45°, and a 360° continuous yaw rotation, with a control accuracy better than 0.01°.

[0049] It should be noted that by employing an active damping algorithm and a disturbance compensation mechanism, vibrations and jitters caused by airflow disturbances, sudden attitude changes, and load inertia can be effectively suppressed during UAV flight, achieving stable control in highly dynamic environments. This method dynamically adjusts the damping coefficient matrix to rapidly approximate the desired gimbal angular velocity, and combines this with inertia compensation to offset nonlinear disturbances in the system, thereby significantly improving control response speed and steady-state accuracy. Compared to traditional PID control, this scheme can maintain image stability and target lock during rapid target movement or violent aircraft maneuvers, reducing the accumulation of attitude errors. The system control accuracy is better than 0.01°, with a wide range of pitch, roll, and yaw, and overall features fast response, strong anti-disturbance capabilities, and high-precision control.

[0050] Example 2 Please refer to Figure 2 As shown, the system of this application mainly consists of a multi-source sensing module, a data fusion module, a target matching module, a trajectory prediction module, and an adaptive control module. The multi-source sensing module includes a visible light camera, an infrared thermal imager, and a lidar, used to acquire multi-dimensional environmental and target information. The data fusion module, based on a dynamic weighted Bayesian algorithm, performs time-series synchronization and weighted adaptive fusion of multi-sensor observation data, outputting a high-precision target fusion estimate. The target matching module employs an improved Siamese neural network, using MobileNetV3 as the feature extraction backbone and combining it with a channel attention mechanism to achieve accurate matching and similarity calculation of fused features. The trajectory prediction module utilizes a Long Short-Term Memory (LSTM) network to perform short-term prediction of the target's motion state, providing feedforward information for subsequent servo control. The adaptive control module combines an active damping algorithm and a disturbance compensation strategy to adjust the gimbal attitude in real time based on the prediction results, achieving stable target tracking and dynamic compensation. These modules work collaboratively to form a high-precision UAV payload target tracking system integrating perception, decision-making, and control.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for tracking UAV payload targets based on multi-sensor fusion, characterized in that, Includes the following steps: Step S1. Acquire observation data from the visible light camera, infrared thermal imager, and lidar, and perform time synchronization and attitude calibration on each observation data. Step S2. Use the dynamic weighted Bayesian fusion algorithm to obtain the fusion estimate of the target; Step S3. Input the fused estimate into the target matching module based on the improved Siamese neural network to extract features and calculate similarity; Step S4. Use a long short-term memory network predictor to make short-term trajectory predictions, and use the prediction results as feedforwards to input into the adaptive mechanical control module to adjust the gimbal angle and achieve continuous and stable tracking of the target.

2. The method according to claim 1, characterized in that, The fusion estimate of the dynamically weighted Bayesian fusion algorithm is calculated using the following formula: ； in, For the first Observations from each sensor; For the first The dynamic weights of each sensor satisfy... ; This is the estimated value of the target after fusion.

3. The UAV payload target tracking method based on multi-sensor fusion according to claim 2, characterized in that, The dynamic weight Calculate using the following formula: ； in, For the first The correlation coefficient between sensor observations and predicted values; This is an environmental adaptive factor used to reflect the impact of the current environment on sensor performance.

4. The method according to claim 3, characterized in that, The environmental adaptive factor Adjust dynamically according to the following formula: ； in, The forgetting factor is a real number between 0 and 1, used to balance historical images; The correlation coefficient between observed and predicted values; As an indicator function, when the sensor quality factor Less than the threshold Take 1 at the time; This is the weight suppression constant, used to reduce the weight of low-quality sensors.

5. The method according to any one of claims 2 to 4, characterized in that, To improve robustness in low-light or high-noise environments, an information entropy correction term is further introduced into the weight calculation. The weight is: ； in, For the first The information entropy of sensor data is used to measure uncertainty. The maximum information entropy value set; item Used to mitigate the effects of noisy sensors.

6. The method according to claim 1, characterized in that, The Siamese neural network uses MobileNetV3 as its feature extraction backbone and performs feature alignment through a channel attention module. Its similarity calculation formula is as follows: ； in, Feature mapping for candidate regions; Feature mapping for template target; For cross-correlation operators; ； in, This is the channel attention weight vector; This is a channel-by-channel product.

7. The method according to claim 6, characterized in that, The matching result output by Siamese is combined with an unscented Kalman filter (UKF) for nonlinear state filtering, and its state update formula is as follows: ； in, For the first One Sigma sampling point; The mean weighting coefficient; Let be the state dimension.

8. The method according to claim 1, characterized in that, The LSTM predictor uses the target's state vector As input, the hidden state update formula is: ； The predicted output is: ; in, These refer to the target position and velocity, respectively. For attitude angle; For the gating activation function of the LSTM unit; It is in a hidden state; To predict the location; To predict attitude angles.

9. The method according to claim 1, characterized in that, The control equation of the adaptive mechanical control module is: ； in, The gimbal controls the output torque; For target position error; The rate of change of error; The feedforward position is predicted by LSTM; The desired attitude angle; To predict attitude angles; These are the proportional, derivative, feedforward, and attitude gain coefficients, respectively.

10. The method according to claim 9, characterized in that, The controller achieves high dynamic stability control through active damping algorithm and disturbance compensation. The damping equation is: ； in, Here is the damping coefficient matrix; The angular velocity of the gimbal; The desired angular velocity; The moment of inertia matrix of the gimbal; Rate of change of angular velocity of the aircraft; The final control output is , ; The control system has a pitch range of ±90°, a roll range of ±45°, and a 360° continuous yaw rotation, with a control accuracy better than 0.01°.

Images