An unmanned aerial vehicle dynamic target stable capturing method based on motion pose decoupling

Abstract

This invention relates to the field of UAV visual surveillance and target tracking technology, specifically disclosing a UAV dynamic target stabilization acquisition method based on motion pose decoupling. This method addresses the image disturbance problem caused by the synchronous motion of the UAV platform and the target. It constructs an accurate optical flow field model caused by the self-motion of the UAV platform by synchronously acquiring airborne images and UAV pose change information; then, it subtracts the self-motion component from the observed optical flow to obtain a compensated residual optical flow field that only represents the true motion of the target; based on this, it extracts the target's motion vector in the image plane and back-projects it to a unified world coordinate system to obtain the target's three-dimensional motion state; finally, it achieves stable tracking through a state estimator. This invention achieves motion decoupling at the algorithmic level, eliminating the need for a mechanical gimbal, and effectively improving the tracking stability and accuracy of UAVs for small moving targets on the ground in highly dynamic flight environments.

Description

Technical Field

[0001] This invention belongs to the field of UAV visual monitoring and target tracking technology, specifically relating to a method for stably capturing and tracking small moving targets on the ground under highly dynamic flight conditions. Background Technology

[0002] When UAVs perform visual tracking of dynamic ground targets, their own flight motion (including translation, rotation, high-frequency vibration, and wind disturbance) causes strong disturbances in the onboard camera images, resulting in unexpected and complex apparent motion of the target within the image plane. This disturbance mixes with the target's true motion, severely affecting the extraction of the target's true motion vector and stable tracking.

[0003] Existing technologies mainly have the following paths and limitations:

[0004] The first type of pure visual tracking algorithm directly detects and tracks targets on jittery image sequences without modeling and compensating for the platform's own motion. In scenarios involving drone maneuvers or wind disturbances, the apparent motion of the background and the target becomes mixed, leading to tracking drift or loss, especially for targets that occupy a small portion of the image.

[0005] The second type of mechanical gimbal stabilization solution provides a stable field of view by physically isolating attitude changes, but it has problems such as response delay, increased weight and power consumption, mechanical wear, and failure under high-frequency vibration.

[0006] The third type of electronic image stabilization or global motion estimation method calculates global transformations between images to stabilize the background, but fails to effectively separate independent moving targets, which can easily lead to the distortion of the target's true motion information through "correction".

[0007] The fourth category of visual methods that partially fuse pose information, while introducing pose data, mostly employs coupled processing or only coarsely compensates for low-frequency poses, failing to establish the six-degree-of-freedom motion of the platform and the target motion at the pixel level.

[0008] Explicit, precise decoupling model.

[0009] The common drawback of existing technologies is that they fail to achieve effective pixel-level decoupling between the platform's self-motion and the target's actual motion at the image observation level. This results in the inability to accurately obtain the target's three-dimensional motion state relative to the ground in high-speed, highly maneuverable dynamic scenarios, leading to low tracking accuracy and poor robustness. Summary of the Invention

[0010] The purpose of this invention is to provide a method for stable acquisition of dynamic targets by unmanned aerial vehicles based on motion pose decoupling.

[0011] The algorithm achieves pixel-level decoupling between platform self-motion and target motion, eliminating the need for a mechanical gimbal and significantly improving the tracking stability and accuracy of moving targets on the ground in highly dynamic environments.

[0012] To achieve the above objectives, the technical solution of the present invention is as follows:

[0013] A method for stabilizing and capturing dynamic targets from a UAV based on motion pose decoupling includes:

[0014] Acquire continuous frame images captured by the UAV's onboard camera, and simultaneously acquire the UAV's pose change information;

[0015] Based on the pose change information and camera parameters, the optical flow field of the image plane caused by the self-motion of the UAV platform is calculated; specifically, a projection model is used to calculate the theoretical displacement of each pixel p:

[0016] ;

[0017] in For the camera intrinsic parameter matrix, and For the pose change of the drone, For pixel depth, For projection functions;

[0018] Subtracting the image plane optical flow field caused by the self-motion from the observed optical flow yields the residual optical flow field characterizing the target's motion relative to the ground;

[0019] Based on the residual optical flow field, the motion information of the target in the image plane is extracted;

[0020] The motion information within the image plane is combined with the target depth information and back-projected onto the world coordinate system to obtain the target's three-dimensional motion state, which is then continuously tracked using a state estimator.

[0021] Furthermore, the calculation of the image plane optical flow field caused by the self-motion of the UAV platform adopts a projection model.

[0022] Furthermore, the target depth information is obtained through depth estimation or depth measurement.

[0023] Furthermore, the pose change information is obtained through a pose measurement unit or its fusion system.

[0024] Furthermore, the extraction of the target's motion information within the image plane includes target region localization and motion statistical processing of the residual optical flow field.

[0025] Furthermore, the state estimator employs a filtering algorithm.

[0026] Furthermore, the method does not rely on a mechanical gimbal device, but uses algorithmic compensation to suppress platform motion interference.

[0027] The advantages of this invention over the prior art are as follows:

[0028] 1. By using an explicit pose decoupling model, the optical flow of the platform's self-movement and the optical flow of the target's motion are separated in real time at the pixel level, eliminating high-frequency micro-motion interference from the source and improving the signal-to-noise ratio of the extracted target motion vector.

[0029] 2. Perform 3D motion tracking in the world coordinate system to eliminate the effects of image planar perspective distortion, significantly improving the tracking success rate and positioning accuracy of small targets (image proportion <0.5%).

[0030] 3. Fully algorithm-based implementation, eliminating the need for a mechanical gimbal, reducing system weight, power consumption, and cost. With optimization, it can achieve real-time processing (≥30fps) on embedded platforms.

[0031] 4. It is highly adaptable to the violent maneuvers, high-speed turns and external wind disturbances of UAVs, and the tracking loss rate is significantly lower than that of existing methods. Attached Figure Description

[0032] Figure 1 This is an overall flowchart of the UAV dynamic target stabilization and capture method based on motion pose decoupling provided in the embodiments of the present invention;

[0033] Figure 2 This is a schematic diagram illustrating the principle of the camera pose change and target motion decoupling model in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the optical flow field calculation caused by the self-motion of the UAV in an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram comparing the motion fields of the image before and after compensation in an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram of the geometric relationship of back-projecting the motion vector of the target image onto the world coordinate system in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0038] Example 1 This example provides a method for stabilizing and capturing dynamic targets from a UAV based on motion pose decoupling. The overall process can be referred to Figure 1 As shown. This embodiment uses a quadcopter drone platform as an example. This platform is equipped with a global shutter visible light camera, an inertial measurement unit (IMU), a visual odometry (VO) module, and an embedded AI computing platform (such as...). Camera intrinsic parameter matrix Pre-calibrated, IMU and VO data are fused through a tightly coupled filter (such as an extended Kalman filter) to output high-frequency, high-precision pose change information between adjacent frames in real time, including rotation matrix. Translation vector .

[0039] Step 1: The airborne camera acquires a continuous sequence of images at a fixed frame rate (e.g., 30 frames per second or higher). and The image processing unit synchronously reads the pose change that is strictly aligned with the image acquisition time. This ensures the consistency of timestamps.

[0040] Step 2: This embodiment uses a depth-assisted projection model for accurate calculation. First, a lightweight monocular depth estimation network is used to perform calculations on the current frame. Perform forward reasoning to generate a dense depth map. For each pixel in the image Assign a depth value .

[0041] For any point on the image plane (In homogeneous coordinates), the theoretical pixel displacement between adjacent frames is purely caused by the six degrees of freedom motion of the drone itself. It can be calculated using the following projective geometry formula:

[0042] ;

[0043] in, Let be the projection function from the 3D camera coordinate system to the 2D image plane, i.e. By traversing the pixels or a set of sparse feature points in the image, the self-moving optical flow field of the entire field of view or key region can be calculated. The core of this step is to use the known precise pose and scene geometry to predict and quantify the "background flow" generated by the drone's own motion on the image.

[0044] Step 3: Calculate the optical flow from frame using an efficient optical flow estimation algorithm (e.g., Farneback dense optical flow method or deep learning-based RAFT architecture). arrive The actual observed optical flow field Uobs is obtained. This observed field is the result of the interaction between the platform motion and any independent moving target. Subsequently, the core decoupling operation is performed, which involves subtracting the predicted self-motion component from the observations pixel-by-pixel or feature-point-by-feature-point:

[0045] ;

[0046] From this, we obtain This is called the compensated residual optical flow field. Theoretically, after this subtraction operation, the global and consistent image offset caused by the UAV's own motion has been effectively removed. The system primarily retains the true motion information of the moving target relative to the stationary background (or with a motion pattern inconsistent with the background), thus achieving an initial separation between the target motion and the platform motion.

[0047] Step 4: After eliminating platform motion interference Performing target motion analysis on-site will be more accurate and reliable. First, run a lightweight target detector (such as YOLOv5s or Nanodet) to locate all potential moving target candidate bounding boxes in the current frame. For each bounding box defining a target region, analyze all targets within that region... Statistical analysis of the vectors (such as calculating the median or mean vector) is performed to obtain the pure motion vector of the target within the current frame image plane. Meanwhile, in order to maintain the consistency of the target's identity over time, a simple data association algorithm based on motion trajectory prediction (such as Kalman filter prediction) and / or appearance feature matching is adopted to associate the target detected in the current frame with the existing tracking trajectory, thereby completing the continuous tracking of the target.

[0048] Step 5: To obtain the target's motion state in the physical world and support advanced decision-making and control, the two-dimensional motion estimation of the image needs to be upscaled to three-dimensional space. This process requires two key pieces of information: first, the target's distance information, which is here achieved using the average depth of the target region. (From the depth map) (Calculated from the corresponding region in the middle); secondly, the current global pose of the UAV camera relative to the world coordinate system, i.e., the rotation matrix. Translation vector This is provided by the drone's state estimation system.

[0049] Using the above information, the target's image motion vector can be obtained through back-projection geometry. Transforming to the world coordinate system yields a three-dimensional velocity vector. The estimate. This. As observations, they are input into a state estimator (using an Extended Kalman Filter (EKF) maintained independently for each tracked target in this embodiment). The state variables of this EKF typically include the target's three-dimensional position and three-dimensional velocity in the world coordinate system. "Prediction" (based on the target motion model) and "update" (integrating new observations) are performed recursively. In this step, EKF can output a smooth, continuous motion trajectory with short-term predictive capabilities for future moments. This mechanism greatly enhances the robustness of the tracking system, providing stable and reliable target state estimates even when the target is briefly occluded or when there is noise in the image observation. Ultimately, it enables accurate, smooth, and uninterrupted stable capture of dynamic ground targets even under the high-speed maneuvering of the UAV itself.

[0050] Technical Effects: To objectively evaluate the effectiveness of this invention, comparative tests were conducted on simulation and physical flight platforms. Key data are as follows: Tracking Accuracy: Under the intense maneuvering of the UAV with a 90° roll per second, the root mean square error (RMSE) of target localization using the traditional undecoupled optical flow method is ≥45 pixels, while the error is reduced to ≤5 pixels after adopting the method of this invention, improving accuracy by nearly 9 times. Small Target Tracking Success Rate: For ground pedestrian targets occupying less than 0.5% of the image, under simulated 10m / s wind speed disturbance, the tracking success rate of the traditional method is approximately 40%, while the success rate of the method of this invention remains stable at over 92%. System Real-Time Performance: In On the embedded platform, the average processing time for the complete process (including depth estimation, optical flow calculation, motion decoupling, and state tracking) is less than 30 milliseconds, meeting the real-time processing requirement of 30fps. Hardware load: Pure algorithm implementation eliminates the need for a mechanical gimbal. While achieving the same tracking performance, the system weight is reduced by approximately 300 grams, and power consumption is reduced by approximately 15%.

[0051] In summary, this invention achieves effective separation of platform motion and target motion at the pixel level through an explicit pose decoupling model, and performs three-dimensional tracking in the world coordinate system by combining depth information, fundamentally improving the stability, accuracy, and system practicality of UAV tracking small moving targets in dynamic environments.

[0052] Example 2

[0053] The main difference between this embodiment and Embodiment 1 lies in the hardware configuration and data processing method of the depth perception system. For a drone platform equipped with a binocular stereo camera, the depth information required in step S2... This can be directly provided by real-time binocular stereo matching algorithms (such as semi-global matching SGBM). This approach does not rely on the generalization ability of data-driven models and can provide more accurate and reliable depth maps in textured scenes. Experimental results show that in well-lit structured scenes, the relative error of binocular depth estimation is approximately 30% lower than that of monocular depth network estimation, thus directly improving the residual optical flow field. The purity of the material and the accuracy of subsequent three-dimensional motion state estimation.

[0054] Example 3: This example addresses targets with highly nonlinear motion patterns and potential for sudden, violent maneuvers (such as emergency obstacle avoidance vehicles). The state estimation step in step S5 is enhanced by employing an unscented Kalman filter (UKF) instead of an EKF to better handle highly nonlinear motion models. In simulation tests, when the target performs an emergency turn with an acceleration change exceeding 2g, the UKF-based method reduces the position prediction error by an average of approximately 35% compared to the basic EKF version. This demonstrates the flexibility of the invention's architecture: front-end motion decoupling provides "clean" observation information, and the back-end state estimator can be flexibly selected based on the target's dynamic characteristics (such as EKF, UKF, or particle filter), thereby maintaining robust tracking performance in complex scenarios.

[0055] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for stable acquisition of dynamic targets by unmanned aerial vehicles based on motion pose decoupling, characterized in that, include: Acquire continuous frame images captured by the UAV's onboard camera, and simultaneously acquire pose change information output by the pose measurement unit carried by the UAV; Based on the pose change information and camera intrinsic parameters, the image plane optical flow field caused by the six-degree-of-freedom self-motion of the UAV platform is calculated by the projection model. Calculate the observed optical flow field between consecutive frames of images, and subtract the image plane optical flow field caused by the self-motion of the UAV platform from the observed optical flow field to obtain the residual optical flow field characterizing the target's motion relative to the ground; Based on the residual optical flow field, the motion vector of the target in the image plane is extracted; The motion vector of the target in the image plane is combined with the target depth information obtained from the airborne depth sensing system and back-projected to the world coordinate system to obtain the three-dimensional motion state estimate of the target. The target is then continuously tracked by a state estimator based on a filtering algorithm.

2. The method according to claim 1, characterized in that, The pose measurement unit includes an inertial measurement unit (IMU) and a visual odometry (VO), and its data is filtered and fused to output high-frequency pose change information.

3. The method according to claim 1, characterized in that, The calculation of the optical flow field in the image plane caused by the self-motion of the UAV platform is specifically performed pixel-by-pixel or feature point-by-feature point using the following projection formula: ， in These are the homogeneous coordinates of the image pixels. For the camera intrinsic parameter matrix, and The rotation matrix and translation vector between adjacent frames. For pixels The depth value, This is the projection function from the 3D camera coordinate system to the 2D image plane.

4. The method according to claim 1, characterized in that, The target depth information is obtained through any of the following methods: a depth estimation neural network based on a monocular camera, or a stereo matching algorithm based on a binocular camera.

5. The method according to claim 1, characterized in that, The extraction of the target's motion vector in the image plane includes: running a target detector on the residual optical flow field to locate the target region, and performing statistical analysis on all optical flow vectors within the region to obtain the target's image motion vector.

6. The method according to claim 1, characterized in that, The state estimator employs an extended Kalman filter (EKF), an unscented Kalman filter (UKF), or a particle filter (PF).

7. The method according to claim 1, characterized in that, The method does not rely on mechanical gimbal devices; it achieves platform motion interference suppression and target motion decoupling entirely through algorithms.

8. A dynamic target acquisition device for unmanned aerial vehicles (UAVs), characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

10. A drone, characterized in that, It includes the airframe, an airborne camera, a pose measurement unit, a depth sensing system, and the UAV dynamic target acquisition device as described in claim 8.

Images