Unmanned aerial vehicle pointing calibration and drop point prediction method and device and medium

Abstract

The invention provides an unmanned aerial vehicle pointing calibration and drop point prediction method and device, and a medium. The method comprises the steps of obtaining a real-time image sequence and obtaining a target pixel coordinate sequence; acquiring and fusing attitude angle data of the holder IMU and the airborne IMU; obtaining a target direction vector in the dynamically normalized camera coordinates based on the target pixel coordinate sequence; converting a target direction vector in the camera coordinate into a target direction vector in an unmanned aerial vehicle body coordinate according to the fused attitude angle data; obtaining a target direction vector in the unmanned aerial vehicle body coordinates after dynamic normalization according to the scene to which the target belongs; and obtaining a target prediction pixel drop point coordinate based on a target direction vector in the unmanned aerial vehicle body coordinate after dynamic normalization. According to the method, the device and the medium, the problems of insufficient pointing control real-time performance, poor attitude stability and large drop point pre-judgment deviation of the existing unmanned aerial vehicle in a maneuvering target real-time indication and drop point prediction scene can be solved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a method, apparatus, and medium for UAV pointing calibration and landing point prediction. Background Technology

[0002] With the widespread application of unmanned aerial vehicle (UAV) platforms in both civilian and military fields, target recognition and localization capabilities based on airborne vision have become key performance indicators. In current UAV missions, common vision-based target processing primarily deals with static or low-dynamic targets. Many control and pointing point prediction algorithms assume the target is located in a known or nearly stationary spatial position, employing simplified geometric model derivation methods. These methods achieve acceptable results under conditions of low-speed targets and stable viewing angles. However, when the target is a high-speed, frequently changing maneuvering target, these assumptions are broken, leading to significantly amplified errors in landing point prediction and real-time pointing control. Furthermore, these methods do not consider the interference of UAV body vibration and gimbal rotation on attitude estimation during maneuvering flight. Relying on attitude data from a single sensor easily generates cumulative errors, resulting in poor attitude stability.

[0003] In summary, existing UAVs suffer from insufficient real-time pointing control, poor attitude stability, and significant deviations in landing point prediction in scenarios involving real-time indication of maneuvering targets and landing point prediction. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art by providing a method, device and medium for pointing calibration and landing point prediction of UAVs, so as to solve the problems of insufficient real-time pointing control, poor attitude stability and large landing point prediction deviation of existing UAVs in real-time target indication and landing point prediction scenarios.

[0005] In a first aspect, the present invention provides a method for pointing calibration and landing point prediction of a UAV, comprising: A real-time image sequence is acquired, and a target pixel coordinate sequence is obtained based on the real-time image sequence using a target tracking algorithm; The attitude angle data of the gimbal inertial measurement unit (IMU) and the airborne IMU are acquired, and the attitude angle data of the gimbal IMU and the airborne IMU are fused to obtain the fused attitude angle data. The target pixel coordinate sequence is inversely transformed using the camera intrinsic parameter matrix to obtain the target direction vector in the camera coordinates. The target direction vector in the camera coordinates is then dynamically normalized to obtain the target direction vector in the camera coordinates after dynamic normalization. The rotation matrix is ​​calculated based on the fused attitude angle data, and the target direction vector in the dynamically normalized camera coordinates is converted into the target direction vector in the UAV body coordinates based on the rotation matrix and the translation vector. Based on the scene to which the target belongs, the target direction vector in the UAV body coordinates is scaled or simplified, and the target direction vector in the UAV body coordinates after scale correction or simplification is dynamically normalized to obtain the target direction vector in the UAV body coordinates after dynamic normalization. The scene is divided into near-range scene and far-range scene. Based on the target direction vector in the dynamically normalized UAV body coordinates, the target predicted pixel landing point coordinates are obtained.

[0006] Furthermore, the step of obtaining the target pixel coordinate sequence based on the real-time image sequence using a target tracking algorithm specifically includes: Locate the initial position of the target from the first frame or the detection trigger frame of the real-time image sequence; Based on the initial position of the target, the target pixel coordinate sequence is determined in consecutive frames of the real-time image sequence.

[0007] Furthermore, the process of fusing the attitude angle data of the gimbal IMU and the airborne IMU to obtain fused attitude angle data specifically includes: The attitude angle data of the gimbal IMU and the airborne IMU are interpolated and timestamped to obtain the attitude angle data of the gimbal IMU and the airborne IMU after interpolation and timestamping. The attitude angle data of the gimbal IMU and the airborne IMU, after interpolation and timestamp alignment processing, are fused using an extended Kalman filter (EKF) to obtain the fused attitude angle data.

[0008] Furthermore, the step of dynamically normalizing the target direction vector in the camera coordinates to obtain the dynamically normalized target direction vector in the camera coordinates specifically includes: The target direction vector in the dynamically normalized camera coordinates can be obtained using the following formula:

[0009] in, Let be the target direction vector in the camera coordinate system. This is the target direction vector in the dynamically normalized camera coordinates. Let be the magnitude of the target direction vector in camera coordinates. It is a tiny constant. This is the scaling adjustment coefficient. The estimated distance to the target.

[0010] Furthermore, the step of scaling or simplifying the target direction vector in the UAV body coordinates according to the scene to which the target belongs specifically includes: In response to the fact that the target belongs to a close-range scene, the target direction vector in the UAV body coordinates is scaled and corrected. In response to the fact that the target belongs to a distant scene, the target direction vector in the UAV body coordinates is simplified.

[0011] Furthermore, obtaining the target prediction pixel landing point coordinates based on the target direction vector in the dynamically normalized UAV body coordinates specifically includes: Calculate the spatial coordinates of the target pointing point based on the target direction vector in the dynamically normalized UAV body coordinates; The spatial coordinates of the target pointing point are mapped back to the image through inverse coordinate transformation to obtain the predicted pixel landing point coordinates of the target.

[0012] Furthermore, after mapping the spatial coordinates of the target pointing point back to the image through inverse coordinate transformation to obtain the predicted pixel landing point coordinates of the target, the method further includes: The coordinates of the predicted target pixel landing point are displayed on the image.

[0013] Secondly, the present invention provides a UAV pointing calibration and landing point prediction device, comprising: The acquisition module is used to acquire real-time image sequences and, based on the real-time image sequences, to obtain target pixel coordinate sequences using a target tracking algorithm; A fusion module is connected to the acquisition module and is used to acquire attitude angle data of the gimbal inertial measurement unit (IMU) and the airborne IMU, and fuse the attitude angle data of the gimbal IMU and the airborne IMU to obtain fused attitude angle data. The inverse transformation normalization module, connected to the acquisition and fusion module, is used to perform an inverse transformation on the target pixel coordinate sequence using the camera intrinsic parameter matrix to obtain the target direction vector in the camera coordinates, and to dynamically normalize the target direction vector in the camera coordinates to obtain the dynamically normalized target direction vector in the camera coordinates. The calculation and conversion module, connected to the inverse transformation and normalization module, is used to calculate the rotation matrix based on the fused attitude angle data, and convert the target direction vector in the dynamically normalized camera coordinates into the target direction vector in the UAV body coordinates based on the rotation matrix and the translation vector. The normalization module, connected to the calculation and conversion module, is used to perform scale correction or simplification processing on the target direction vector in the UAV body coordinates according to the scene to which the target belongs, and to dynamically normalize the target direction vector in the UAV body coordinates after scale correction or simplification processing to obtain the target direction vector in the UAV body coordinates after dynamic normalization. The scene is divided into near-range scene and far-range scene. The module is connected to the processing normalization module and is used to obtain the target predicted pixel landing point coordinates based on the target direction vector in the dynamically normalized UAV body coordinates.

[0014] Thirdly, the present invention provides a UAV pointing calibration and landing point prediction device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the UAV pointing calibration and landing point prediction method described in the first aspect above.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the UAV pointing calibration and landing point prediction method described in the first aspect.

[0016] The present invention provides a method, apparatus, and medium for UAV pointing calibration and landing point prediction. First, a real-time image sequence is acquired, and based on this sequence, a target pixel coordinate sequence is obtained using a target tracking algorithm. Then, attitude angle data from the gimbal inertial measurement unit (IMU) and the airborne IMU are acquired and fused to obtain fused attitude angle data. Next, the target pixel coordinate sequence is inversely transformed using a camera intrinsic parameter matrix to obtain a target direction vector in the camera coordinate system. This target direction vector is then dynamically normalized to obtain a dynamically normalized target direction vector in the camera coordinate system. Finally, the fused attitude angle data is used to determine the target direction vector. A rotation matrix is ​​calculated from the angular data. Based on the rotation matrix and the translation vector, the target direction vector in the dynamically normalized camera coordinates is converted into the target direction vector in the UAV body coordinates. Then, according to the scene to which the target belongs, the target direction vector in the UAV body coordinates is scaled or simplified, and the scaled or simplified target direction vector in the UAV body coordinates is dynamically normalized to obtain the target direction vector in the dynamically normalized UAV body coordinates. The scene is divided into near-range scene and far-range scene. Finally, based on the target direction vector in the dynamically normalized UAV body coordinates, the target predicted pixel landing point coordinates are obtained. This invention effectively suppresses errors from a single IMU by fusing multi-source attitude angle data from a gimbal IMU and an airborne IMU, obtaining high-precision attitude parameters and significantly improving attitude stability and noise resistance. It performs inverse transformation and dynamic normalization on the target pixel coordinate sequence obtained from the real-time image sequence using a target tracking algorithm, yielding a dynamically normalized target direction vector in the camera coordinates. This vector, combined with a translation vector and a rotation matrix calculated based on the fused attitude angle data, is then transformed to obtain the target direction vector in the UAV body coordinates. This target direction vector in the UAV body coordinates undergoes scale correction or simplification based on the target's scene and dynamic normalization. Based on this dynamically normalized target direction vector in the UAV body coordinates, the predicted target pixel landing point coordinates are accurately obtained, significantly reducing landing point prediction deviation. This enables real-time pointing guidance and landing point prediction for ground or air targets, solving the problems of insufficient real-time pointing control, poor attitude stability, and large landing point prediction deviations in existing UAVs for real-time indication and landing point prediction of maneuvering targets. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this drawing or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this drawing. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a UAV pointing calibration and landing point prediction method according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram illustrating the landing point prediction and ground station visualization according to an embodiment of the present invention; Figure 3 This is an overall structural diagram of the UAV pointing calibration and landing point prediction system according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a UAV pointing calibration and landing point prediction device according to Embodiment 2 of the present invention; Figure 5 This is a schematic diagram of the structure of a drone pointing calibration and landing point prediction device according to Embodiment 3 of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0020] It is understood that the specific embodiments and accompanying drawings described herein are merely for explaining the invention and are not intended to limit the invention.

[0021] It is understood that, without conflict, the various embodiments and features in the embodiments of the present invention can be combined with each other.

[0022] It is understood that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, while the parts unrelated to the present invention are not shown in the drawings.

[0023] It is understood that each unit or module involved in the embodiments of the present invention may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0024] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this invention may occur in a different order than that marked in the accompanying drawings.

[0025] It is understood that the flowcharts and block diagrams of this invention illustrate the possible architecture, functions, and operations of systems, apparatuses, devices, and methods according to various embodiments of this invention. Each block in the flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagram and flowchart can be implemented using a hardware-based system to achieve the specified function, or using a combination of hardware and computer instructions.

[0026] It is understood that the units and modules involved in the embodiments of the present invention can be implemented by software or by hardware. For example, the units and modules can be located in a processor.

[0027] Application Overview With the widespread application of unmanned aerial vehicle (UAV) platforms in both civilian and military fields, target recognition and localization capabilities based on airborne vision have become a key performance indicator for systems. In current UAV missions, common vision-based target processing primarily deals with static or low-dynamic targets. Many control and pointing point prediction algorithms assume the target is located in a known or nearly stationary spatial position, employing simplified geometric model derivation methods. These methods achieve acceptable results when the target is low-speed and the viewpoint is stable. However, when the target is a high-speed, frequently changing maneuvering target, these assumptions are broken, leading to significantly amplified prediction bias and real-time performance issues.

[0028] Traditional pointing point simulation and prediction often focus on the dynamic modeling or static geometric projection of the pointing point itself, rarely considering the error sources along the entire coordinate chain (pixel → camera → body → georeference). In real-world systems, camera intrinsic errors, lens distortion, installation offset (translation) between the gimbal and the body, relative translation during gimbal rotation, and temporal deviations between the IMU and the visual frame can all accumulate into significant spatial offsets within a short period. This is especially true in close-range scenes and when the viewing angle is significantly tilted, where the contribution of translation components to the final projection error is often underestimated. Furthermore, common pixel-to-spatial-point normalization processes, if the scale and threshold are not clearly defined (e.g., scale correction strategies for close-range scenes), can lead to numerical instability or drift, thus affecting the reliability of the landing point prediction.

[0029] At the target perception level, traditional popular tracking algorithms (such as KCF and MOSSE) are known for their efficiency and lightweight nature. However, they are usually based on linear features, fixed windows, and simplified model assumptions, making them prone to losing track or drifting under conditions of rapid motion, scale changes, occlusion, background interference, or drastic changes in lighting. Recent advancements in deep learning and correlation filtering-based methods have improved robustness, but often at the cost of higher computational overhead, making it difficult to maintain high frame rates in real-time processing on resource-constrained airborne embedded platforms.

[0030] In summary, existing UAVs suffer from insufficient real-time pointing control, poor attitude stability, and significant landing point prediction errors in real-time target designation and landing point prediction scenarios. Therefore, a tracking scheme is urgently needed for UAVs in these scenarios that can: maintain high robustness at the perception level; and systematically handle error sources such as camera intrinsic parameters, attitude translation, temporal alignment, and scale adaptation at the geometric level, while employing different numerical strategies for near-range and long-range scenarios to ensure stability and accuracy. Only by organically combining modules such as visual tracking, attitude fusion, gimbal translation correction, and dynamic normalization, and considering real-time visualization requirements, can the engineering requirements for high dynamic response and high-precision landing point prediction be met.

[0031] To address the aforementioned technical problems, this application provides a method, apparatus, and medium for UAV pointing calibration and landing point prediction. By fusing multi-source attitude angle data from a gimbal IMU and an airborne IMU, errors from a single IMU are effectively suppressed, high-precision attitude parameters are obtained, and attitude stability and noise resistance are significantly improved. The target pixel coordinate sequence obtained from a real-time image sequence using a target tracking algorithm undergoes inverse transformation and dynamic normalization to obtain the target direction vector in the dynamically normalized camera coordinates. This target direction vector is then combined with a translation vector and a rotation matrix calculated based on the fused attitude angle data to obtain the target direction vector in the UAV body coordinates. This target direction vector in the UAV body coordinates is then subjected to scale correction or simplification based on the target's scene and dynamically normalized. Based on the dynamically normalized target direction vector in the UAV body coordinates, the target prediction pixel landing point coordinates are accurately obtained, significantly reducing landing point prediction deviation. This enables real-time pointing guidance and landing point prediction for ground or air targets, thus at least solving the problems of insufficient real-time pointing control, poor attitude stability, and large landing point prediction deviations in existing UAVs for real-time indication and landing point prediction of maneuvering targets.

[0032] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0033] Example 1: This embodiment provides a method for UAV pointing calibration and landing point prediction, such as Figure 1 As shown, the method includes: Step S101: Obtain a real-time image sequence, and based on the real-time image sequence, use a target tracking algorithm to obtain a target pixel coordinate sequence.

[0034] It should be noted that a real-time image sequence can refer to a collection of single-frame images that are continuously acquired in real time and arranged in chronological order.

[0035] Specifically, image acquisition and target tracking: The optical camera mounted on the gimbal acquires video image sequences in real time, and the system sends the image sequence data to the preset target tracking module through a frame buffering mechanism. This module adopts a lightweight but robust target tracking algorithm, and can flexibly select a lightweight network based on deep features according to different hardware platforms.

[0036] In one optional embodiment, obtaining the target pixel coordinate sequence based on the real-time image sequence using a target tracking algorithm specifically includes: Locate the initial position of the target from the first frame or the detection trigger frame of the real-time image sequence; Based on the initial position of the target, the target pixel coordinate sequence is determined in consecutive frames of the real-time image sequence.

[0037] Specifically, the algorithm first locates the initial position of the target in the first frame or the detection trigger frame, and establishes an initial matching model by extracting multi-scale feature templates of the target. Then, in consecutive frames, a correlation response map search mechanism is used to determine the target pixel center coordinate (u,v) sequence (i.e., the target pixel coordinate sequence) and its scale changes. Simultaneously, a confidence score C_track is output to characterize the tracking reliability. When the confidence score drops below a threshold (e.g., C_track < 0.5), the system automatically initiates a re-detection mechanism to restore target lock, ensuring continuous tracking even in complex scenes.

[0038] Step S102: Acquire the attitude angle data of the gimbal inertial measurement unit (IMU) and the airborne IMU, and fuse the attitude angle data of the gimbal IMU and the airborne IMU to obtain the fused attitude angle data.

[0039] It should be noted that an IMU (Inertial Measurement Unit), also known as a motion sensor, is a device used to measure the three-axis attitude angles and acceleration of an object. Typically, an IMU consists of an accelerometer and a gyroscope; the accelerometer measures the object's acceleration, and the gyroscope measures its angular velocity. In UAV systems, gimbal-mounted IMUs and airborne IMUs are used to acquire real-time attitude angles (pitch, roll, yaw) and angular velocity information of the gimbal and the airframe, providing attitude data support for target pointing and landing point prediction. Attitude angle definition (right-hand coordinate system): Whether for the gimbal or the airframe, attitude angles are based on the following standard definition (unit: radians; the "degrees" output by the IMU must first be converted to radians): Yaw angle (denoted as ( ): Rotate around the Z-axis, clockwise is positive (corresponding to the UAV's heading to the right); Pitch angle (denoted as ( ): Rotation around the Y-axis, with upward being positive (corresponding to camera tilt and body tilt); Roll angle (denoted as ) ): Rotate around the X-axis, tilting to the right is positive (corresponding to the right side of the body being raised).

[0040] In one optional embodiment, fusing the attitude angle data of the gimbal IMU and the airborne IMU to obtain fused attitude angle data specifically includes: The attitude angle data of the gimbal IMU and the airborne IMU are interpolated and timestamped to obtain the attitude angle data of the gimbal IMU and the airborne IMU after interpolation and timestamping. The attitude angle data of the gimbal IMU and the airborne IMU, after interpolation and timestamp alignment processing, are fused using an extended Kalman filter (EKF) to obtain the fused attitude angle data.

[0041] Specifically, attitude data fusion and time alignment: The system collects attitude angle (pitch, roll, yaw) and angular velocity information from the gimbal IMU and the body IMU respectively. Since there are differences in sampling frequency and time delay between the two sets of data, the present invention first performs interpolation and timestamp alignment processing on them.

[0042] Specifically, at the visual frame time point At that time, the system uses two sets of data with adjacent timestamps in the IMU buffer. and The following can be obtained through linear interpolation:

[0043] in, This refers to the time point corresponding to the visual frame (i.e., the target time point to which it needs to be aligned). For visual frame time points The angular velocity value obtained through interpolation (this is the final aligned data required) is located at that point. In the IMU buffer, timestamps earlier than The time points corresponding to adjacent IMU data, In the IMU buffer, the timestamp is later than The time points corresponding to adjacent IMU data , For IMU at a point in time The raw angular velocity data collected, For IMU at a point in time The raw angular velocity data collected.

[0044] This allows us to obtain attitude data that strictly corresponds to the visual frame. If the time difference exceeds a set threshold, the latest IMU data is pushed forward to the frame time using the angular velocity integral method to compensate for the delay.

[0045] It should be noted that Kalman filtering is an algorithm that uses the state equations of a linear system to optimally estimate the system state using system input and output observation data. Kalman filtering establishes the state equations and measurement equations to jointly estimate the attitude angles, angular rates, and covariance. In this embodiment, it is used to fuse IMU data, suppress noise, and improve attitude estimation accuracy.

[0046] Specifically, to further suppress IMU inherent noise (high-frequency random disturbances) and gyroscope drift (long-term accumulated errors), this embodiment uses extended Kalman filtering (EKF) to perform multi-source fusion optimization on the time-aligned gimbal and airborne IMU attitude information (i.e., the attitude angle data of the gimbal IMU and airborne IMU after interpolation and timestamp alignment). The specific implementation process is as follows: (1) EKF state modeling: A joint state vector containing "attitude core quantities" and "error correction quantities" is constructed by specifically incorporating key error sources: x= .in, q cloud, q The attitude quaternions of the gimbal and the body relative to the ground coordinate system are respectively (to avoid the gimbal lock problem of attitude angle and ensure the continuity of rotation description). b cloud, b The zero bias of the gyroscopes of the gimbal and the IMU of the machine are respectively used for real-time estimation as a source of drift error, reducing the cumulative deviation at its source. The formula contains... T This indicates transpose. In state estimation and control theory, state vectors are usually represented as column vectors.

[0047] (2) Two-stage filtering: combining aligned data to achieve accurate estimation: Prediction Phase: Based on the IMU dynamic simulation state, time-aligned IMU angular velocity measurements are used. Subtracting the currently estimated gyroscope bias 'b', we obtain the true angular velocity. ; Then, using the quaternion kinematic equations, ,in, Let q be the derivative of the attitude quaternion with respect to time, representing the rate of change of the quaternion with time, i.e., the rate of change of attitude. For the true angular velocity The resulting antisymmetric matrix (the operation matrix in quaternion multiplication). These are attitude quaternions used to describe rotational attitude. The true angular velocity, i.e. ; Predicted values ​​of gimbal and aircraft attitude at the next moment (k-1) Simultaneously, assuming the gyroscope's zero bias changes slowly in the short term, the prediction... And through the covariance matrix:

[0048] in, The prior error covariance matrix represents the uncertainty of the state estimate at time k based solely on the dynamic model prediction (before incorporating current observations). The subscript... Indicates "based on up to k" "Information at time 1 is used to estimate information at time k". Let be the posterior error covariance matrix, representing the variance of the variance in k. At moment 1, k was incorporated. The uncertainty of the state estimate after the observation information at time 1 reflects the confidence level of the updated state estimate. F is the state transition Jacobian matrix. Q is the transpose of the state transition Jacobian matrix, and Q is the process noise covariance matrix, which represents the uncertainty of the model during state prediction, including IMU measurement noise, model approximation error, unmodeled dynamics, etc. Q increases the uncertainty of the prediction and prevents overconfidence in filtering.

[0049] Update Phase: Multi-source information is fused to correct biases. A time-aligned visual frame pose reference (e.g., visually calculated relative pose) and accelerometer gravity vector observations (to aid in judging pose rationality) are introduced as the measurement value z, and the measurement equation is constructed:

[0050] in, The measurement vector is the gravity vector observed by the accelerometer (used to help determine the rationality of the attitude). For the measurement function, the state vector Mapping to predicted measurements, for example: converting the attitude quaternion q into the projection of the gravity vector in body coordinates. For measuring noise.

[0051] Calculate the Kalman gain:

[0052] in, This is the Kalman gain matrix, used to balance the relative confidence of the predicted state and the measured value. A larger gain indicates greater confidence in the measured value. To measure the Jacobian matrix (balancing the reliability of prediction and measurement). To measure the transpose of the Jacobian matrix, To measure the noise covariance matrix.

[0053] Finally, the predicted state is corrected using the measurement residuals to obtain the optimal estimate. And update the covariance matrix. ,in, This is the posterior state estimate, which is the optimal state estimate that incorporates the current measurements. For prior state prediction, the state prediction values ​​come from the prediction phase. These are the predicted values ​​based on prior state predictions. The residual (news) is used to measure the difference between the actual and predicted measurements. The posterior error covariance matrix represents the uncertainty (confidence level) after the state update. It is usually smaller than the prior covariance because it incorporates measurement information. It is an identity matrix.

[0054] (3) Output and Derivative Calculation: Connecting to subsequent coordinate solution requirements: The EKF outputs fused gimbal and aircraft attitude quaternions, which can be directly converted into attitude matrices. , Simultaneously, the state covariance matrix P is output as the confidence matrix (the diagonal elements correspond to the estimated variances of the attitude and zero-bias components; the smaller the variance, the higher the confidence). Based on the orthogonality property of the rotation matrix (the inverse equals the transpose), the attitude matrix of the aircraft relative to the gimbal is further calculated. =R b - ¹× This provides a precise relative attitude reference for subsequent coordinate transformations.

[0055] The fused attitude matrices are respectively represented as gimbal attitudes. With the body posture and output its confidence matrix. .

[0056] It is worth mentioning that this embodiment, through the dual optimization of "time alignment to eliminate timing deviation + EKF fusion to suppress noise drift", not only ensures the consistency of gimbal and body attitude in the time domain and numerical domain, but also effectively eliminates the impact of attitude jump and time drift on subsequent coordinate calculation, providing a reliable attitude basis for stable system operation.

[0057] Step S103: Perform an inverse transformation on the target pixel coordinate sequence using the camera intrinsic parameter matrix to obtain the target direction vector in the camera coordinates, and dynamically normalize the target direction vector in the camera coordinates to obtain the dynamically normalized target direction vector in the camera coordinates.

[0058] It should be noted that the camera intrinsics matrix is ​​a matrix that describes the relationship between the camera's optical characteristics and imaging geometry, and is usually represented as:

[0059] in, f x , f y The pixel representation of the camera's focal length in the x and y directions. c x , c y The primary point coordinates are used. The camera intrinsic parameter matrix is ​​used for the transformation between pixel coordinates and camera coordinates. In this embodiment, the pixel coordinates are converted into direction vectors in the camera coordinate system through the inverse transformation of the intrinsic parameter matrix.

[0060] Specifically, the inverse transformation from pixel to camera coordinates: for each frame's target pixel coordinates (u,v), using the camera intrinsic parameter matrix... Perform inverse mapping: This yields the direction vector in the camera coordinate system, where... This is the direction vector (homogeneous coordinates) in the camera coordinate system. Internal parameter matrix The inverse matrix, It is the homogeneous form of pixel coordinates.

[0061] Specifically, to avoid numerical instability, this embodiment proposes a dynamic normalization algorithm, where dynamic normalization is an algorithm that adaptively adjusts the scale of the direction vector according to the target distance. Dynamic normalization prevents the direction vector from being excessively amplified in close-range scenarios and degenerates into a conventional normalization model in long-range scenarios, thus ensuring the numerical stability of coordinate calculation.

[0062] Specifically, the dynamic normalization in the camera coordinate system (core formula) is used to correct the direction vector obtained from the inverse transformation of the camera intrinsic parameters. To avoid near-field magnification errors, the target direction vector in the dynamically normalized camera coordinates is obtained using the following formula:

[0063] in, The target direction vector in camera coordinates (i.e., the original direction vector in the camera coordinate system, derived from the camera intrinsic parameter matrix) with pixel coordinates The calculations show that the result is unnormalized. This is the target direction vector in the dynamically normalized camera coordinate system (i.e., the direction vector of the dynamically normalized camera coordinate system, which is ultimately used for the subsequent coordinate transformation of "camera → gimbal → body"). The magnitude of the target direction vector in camera coordinates (i.e., the original vector). The modulus, specifically the Euclidean distance. x / y / z are (three-dimensional components) To prevent division by zero by a small constant (the default value in this embodiment) ,avoid (caused by calculation errors) The scaling factor (an empirical parameter preset by humans to control the "influence of distance on the denominator") ranges from 0.1 to 0.5. In this embodiment, it is specified that α = 0.3 for close-range scenes Z < 100m and α = 0.3 for distant scenes Z ≥ 100m. =0.1), The estimated distance to the target (the core adaptive basis, obtained from the "monocular range estimation formula Z≈f·H / h" or "LiDAR", unit: m; f is the camera focal length, H is the actual height of the target, and h is the target pixel height). This strategy prevents the direction vector from being excessively amplified under close-range conditions, but degenerates into a regular normalized model under long-range scenarios.

[0064] Step S104: Calculate the rotation matrix based on the fused attitude angle data, and convert the target direction vector in the dynamically normalized camera coordinates into the target direction vector in the UAV body coordinates based on the rotation matrix and the translation vector.

[0065] It should be noted that the rotation matrix is ​​a mathematical matrix used to describe rotational transformations between coordinate systems. It is an orthogonal matrix, and its transpose is equal to its inverse. In this embodiment, the rotation matrix is ​​used to transform the target direction vector from the camera coordinate system to the UAV body coordinate system. The rotational transformation between different coordinate systems is achieved by multiplying the gimbal attitude matrix and the UAV attitude matrix.

[0066] Specifically, in the coordinate chain transformation of this embodiment, the rotation matrix... These are not fixed values—they change dynamically with the real-time attitude (pitch, roll, yaw) of the gimbal and the airframe, and are calculated from attitude angle data collected by the gimbal IMU and the airframe IMU (i.e., the onboard IMU). The core is a rotation matrix derived from Euler angles (UAVs commonly use the "ZYX" rotation sequence, i.e., yaw → pitch → roll). The specific calculation logic and examples are as follows: Rotation Sequence and Basic Rotation Matrices: In the UAV field, the default rotation sequence is "ZYX" (yaw first, then pitch, and finally roll), because this sequence better reflects the actual attitude changes of the gimbal rotation and the aircraft's flight. The mathematical expressions for the three basic rotation matrices (rotations around a single axis) are as follows: Rotation matrix (R) about the Z-axis (yaw) Yaw ( )):

[0067] Rotation matrix (R) around the Y-axis (pitch) Pitch ( ))

[0068]

[0069] Rotation matrix (R) around the X-axis (roll) Roll

[0070]

[0071] Specifically, gimbal rotation Specific calculations: Describes the "attitude rotation relationship between the gimbal coordinate system and the geodetic coordinate system", using the gimbal attitude angles acquired by the gimbal IMU. The calculation yields the following formula: = ( ) ( )

[0072] Airplane rotating Specific calculations: Describes the "attitude rotation relationship between the aircraft coordinate system and the geodetic coordinate system," using aircraft attitude angles acquired by the aircraft's IMU. The calculation yields the following formula: = ( ) ( )

[0073] Aircraft rotation matrix relative to the gimbal =R b - ¹× (Note: The matrix multiplication order is "yaw first → pitch then roll". The multiplication order cannot be reversed, otherwise it will lead to attitude calculation errors.)

[0074] Specifically, coordinate chain transformation and translation correction: In order to map the target direction from the camera coordinate system to the UAV body coordinate system, this embodiment establishes a multi-layer coordinate chain model from the camera to the gimbal and then to the UAV body.

[0075] The transformation relationship is expressed as:

[0076] in, The rotation matrix of the aircraft relative to the gimbal; : The translation vector between the camera and the aircraft obtained through calibration; : The target direction vector in the dynamically normalized camera coordinate system (i.e., the direction vector of the camera coordinate system after dynamic normalization). This is the body coordinate vector (i.e., the target direction vector in the UAV body coordinates).

[0077] First half of the formula By rotation matrix ( (Camera → Body), complete the "attitude rotation alignment" to obtain the direction vector considering only the attitude; The second half of the formula : By translating the vector (fixed installation), through Rotate to the body coordinate system, then compensate for the positional offset of "camera → body", and finally obtain the complete body coordinate vector. .

[0078] It is worth mentioning that this coordinate chain transformation and translation correction mechanism ensures the consistency between the camera optical axis direction and the body direction under different attitude states.

[0079] Step S105: Based on the scene to which the target belongs, perform scale correction or simplification on the target direction vector in the UAV body coordinates, and dynamically normalize the target direction vector in the UAV body coordinates after scale correction or simplification to obtain the target direction vector in the UAV body coordinates after dynamic normalization. The scene is divided into near-range scene and far-range scene.

[0080] It should be noted that close-range scenarios refer to situations where the drone is relatively close to the target, while long-range scenarios refer to situations where the drone is relatively far from the target.

[0081] In one optional embodiment, the step of scaling or simplifying the target direction vector in the UAV body coordinates according to the scene to which the target belongs specifically includes: In response to the fact that the target belongs to a close-range scene, the target direction vector in the UAV body coordinates is scaled and corrected. In response to the fact that the target belongs to a distant scene, the target direction vector in the UAV body coordinates is simplified.

[0082] Specifically, this embodiment dynamically determines whether the current scene is in a near-range or far-range state based on the target's pixel scale changes, monocular ranging information, or image feature matching results: When a target with a large pixel size and a small depth estimate is detected in the image (i.e., the target is close), the system automatically activates translation correction and an adaptive scaling factor s. The scaling correction formula is as follows:

[0083] in, This refers to the scale-corrected coordinate vector of the machine. The vector 's' is adjusted in real-time based on target depth information or empirical coefficients to ensure the direction vector maintains consistency with the spatial scale. When the target depth 'd' is obtained, 's' is dynamically adjusted using the formula 's = k / d' (where k is the camera focal length parameter) to ensure the direction vector is consistent with the spatial scale. For example, for a close-range target of 100 meters, 's' increases as the distance decreases to compensate for parallax effects.

[0084] In long-distance scenarios, parallax has a smaller impact, and the system automatically degenerates into a simplified linear projection model to improve computational efficiency. When d > 100 meters, it degenerates to s = 1, and the formula simplifies to... = × × The computational load is reduced by 40%, and the latency is controlled within 20ms.

[0085] It is worth mentioning that this strategy enables adaptive switching between near and far-field scenarios, ensuring a balance between coordinate calculation accuracy and system real-time performance.

[0086] It should be noted that the core formulas of the dynamic normalization algorithm proposed in this embodiment are divided into two categories (corresponding to the camera coordinate system and the body coordinate system, respectively). The dynamic normalization formula for the body coordinate system and the dynamic normalization formula for the camera coordinate system have completely consistent parameter meanings and adjustment logic.

[0087] Step S106: Based on the target direction vector in the dynamically normalized UAV body coordinates, obtain the target predicted pixel landing point coordinates.

[0088] In one optional embodiment, obtaining the target predicted pixel landing point coordinates based on the target direction vector in the dynamically normalized UAV body coordinates specifically includes: Calculate the spatial coordinates of the target pointing point based on the target direction vector in the dynamically normalized UAV body coordinates; The spatial coordinates of the target pointing point are mapped back to the image through inverse coordinate transformation to obtain the predicted pixel landing point coordinates of the target.

[0089] Specifically, after obtaining the target direction vector (i.e., the target direction vector in the dynamically normalized UAV body coordinates) and the body attitude, this invention calculates the pointing point position based on the trajectory model of the projected object. Let the projection starting point be... The initial velocity is , direction is If the acceleration due to gravity is g, then the equation of motion of the landing point is:

[0090] Among them, by adjusting the aircraft's attitude, and dynamic normalization coincide, It is a vertical unit vector.

[0091] Solving for the intersection point with the target plane will yield the spatial coordinates of the target's pointing point. .

[0092] Will The predicted pixel location is obtained by mapping back to the image plane through inverse coordinate transformation. .

[0093] It should be noted that the following are the steps for connecting the supplementary landing point motion equation and pixel coordinate transformation: Camera coordinate transformation:

[0094] Pixel projection:

[0095] Final coordinates:

[0096] in, : The coordinate vector of the target pointing point in the camera coordinate system. : The inverse rotation matrix from the body coordinate system to the camera coordinate system (R is the rotation matrix from camera to body). (its inverse transform) : Spatial coordinates of the target's pointing point (coordinates in the machine / world coordinate system). T Translation vector from the origin of the body coordinate system to the origin of the camera coordinate system. The homogeneous coordinates of the target point projected onto the image plane. K Camera intrinsic parameter matrix (including camera intrinsic parameters such as focal length and principal point coordinates). The pixel coordinates (horizontal and vertical directions) of the predicted target pixel location on the image plane. ): The three components of homogeneous coordinates (w is the homogeneous term, used for normalization to obtain pixel coordinates).

[0097] In an optional embodiment, after mapping the spatial coordinates of the target pointing point back to the image through inverse coordinate transformation to obtain the predicted pixel landing point coordinates of the target, the method further includes: The coordinates of the predicted target pixel landing point are displayed on the image.

[0098] Specifically, the system displays the predicted landing point on the image as marked points or error ellipses, and can also calculate the overall confidence level:

[0099] in, Overall confidence level of the target (assessing the reliability of the predicted landing point), Weighting coefficients (satisfying) (balancing the proportions of the two confidence levels) Target tracking confidence (reflects the stability and accuracy of the target tracking algorithm). Attitude uncertainty matrix (describes the degree of error in the attitude measurement of the UAV body). : Norm of the attitude uncertainty matrix (converts the matrix into a scalar, characterizing the overall magnitude of the attitude error).

[0100] It is worth mentioning that this embodiment realizes real-time visualization of the landing point prediction, which improves the intuitiveness of the direction.

[0101] It should be noted that this invention addresses the problems of insufficient real-time pointing control, poor attitude stability, and large deviations in landing point prediction in existing projection simulation devices mounted on unmanned equipment. The UAV pointing calibration and landing point prediction method provided is based on multi-source attitude fusion and dynamic normalization. The core idea is as follows: high-precision attitude parameters are obtained by fusing multi-source data from the gimbal inertial measurement unit (IMU) and the device's own IMU; combined with the target pixel coordinates output by the visual tracking module, the method achieves real-time pointing guidance and visual prediction of landing points for ground or air targets through coordinate transformation, position correction, and motion trajectory modeling.

[0102] The present invention has the following advantages over the prior art: (1) By using a time synchronization and filtering fusion mechanism, the stability and noise resistance of attitude estimation are significantly improved; (2) The use of scene-specific coordinate transformation and dynamic normalization algorithm ensures the numerical stability of coordinate solution under near and far conditions; (3) Introduce a landing point prediction and error visualization mechanism to enable operators to observe pointing deviation in real time, thereby improving launch accuracy and safety; (4) The engineering design enables the system to operate in real time on a resource-constrained airborne platform.

[0103] In one specific embodiment, this method obtains the target pixel coordinates through a tracking algorithm, and calculates the target direction vector (including dynamic normalization) by combining camera intrinsic parameters, gimbal attitude, and translation vector for near / far-range scenarios, determining the UAV pointing tilt angle; it simulates the pointing point based on the relative attitude angle and visualizes it in real time. This solves the problems of existing technologies not considering translation and unclear normalization, achieving a pointing error of less than 0.5° and a system latency of no more than 40ms. It is suitable for general pointing simulation scenarios of UAVs targeting near and far-range maneuvering targets. Specifically, this UAV pointing calibration and landing point prediction method may include: (1) The gimbal camera acquires real-time image sequences and performs real-time tracking of the target based on a standard tracking algorithm to output a pixel coordinate sequence; Specifically, the tracking module outputs the target confidence score in each frame, and triggers a candidate detection and relocation process when the confidence score is lower than a preset threshold to avoid target drift.

[0104] (2) Acquire attitude angle data from the gimbal IMU and the airborne IMU and perform time alignment and filtering; Specifically, in step (2), the attitude angle data is fused using Kalman filtering, and the filter state includes attitude angle, angular rate and covariance estimation to suppress IMU noise.

[0105] Specifically, it also includes synchronizing the pose and image frames with timestamps, and using interpolation or buffering mechanisms to ensure the best pairing of visual frames and IMU data, so as to reduce the impact of timing errors on pointing results.

[0106] (3) Use the camera intrinsic parameter matrix K to perform an inverse transformation on the pixel coordinates to obtain the normalized camera coordinate direction vector. ; Specifically, the inverse transformation from pixel to camera coordinates is expressed as: .

[0107] (4) Using a rotation matrix With translation vector Transform the direction vector in the camera coordinate system to the UAV body coordinate system; (5) Based on the scene determination strategy, different scale correction and simplification assumptions are applied to the near-distance and far-distance scenes respectively, and the direction vector of the body coordinate system is dynamically normalized to obtain the desired result. ; Specifically, the scene determination is based on a monocular ranging formula. Alternatively, based on a target pixel scale threshold, the scale factor can be prioritized for close-range scenes. Corrections were made for long-distance scenes, while... Simplified processing.

[0108] Specifically, the near-distance correction also includes gimbal translation correction: the translation vector of the camera optical center relative to the body coordinate system is obtained through calibration, and this translation amount is used in coordinate transformation to correct near-distance geometric errors.

[0109] Specifically, the dynamic normalization takes the following form: ,in To prevent division by zero for small positive numbers, For adjustment coefficients, It is used to estimate distances and can adaptively adjust to the estimated distance in close-range scenarios.

[0110] (6) Based on The spatial location of the landing point is calculated using preset projection rules and projected back into the image to obtain the pixel landing point coordinates. The pixel landing point is then visualized and displayed in real time on the ground station image interface.

[0111] Specifically, the calculation of the projected pixel is as follows: , , , .

[0112] Specifically, the method further includes assessing the confidence level of the predicted landing point based on the target tracking confidence level, attitude filtering uncertainty, and scale estimation error, and then visually feeding the confidence level back to the ground station operator.

[0113] Specifically, the ground station visualization includes overlaying predicted landing point markers, historical trajectories, error ellipses, and confidence text information onto the original image.

[0114] Specifically, such as Figure 2 As shown, the spatial location of the pointing point is projected back into the camera image to obtain the predicted pixel landing point, while the error ellipse, historical trajectory and confidence text are superimposed, and the operator interaction (such as locking or manual correction) is supported.

[0115] It should be noted that this method is applied to UAV pointing calibration and landing point prediction systems, such as... Figure 3 As shown, the system includes: a gimbal camera, a gimbal controller, an embedded computing module (containing sub-modules for image acquisition, target tracking, attitude fusion, coordinate transformation, scene determination, and visualization), an airborne IMU, and a ground station display module (connection relationship with the ground station display module). A wireless link is used for data transmission and control.

[0116] Specifically, the embedded computing module includes an image acquisition unit, a target tracking unit, a posture fusion unit, a coordinate transformation unit, a scene determination unit, a normalization unit, and a projection and visualization unit, and exchanges data with the ground station in real time via a wireless link.

[0117] Specifically, the system also includes a monocular depth estimation module or a near-range ranging sensor to improve the estimation accuracy of the near-range scale factor s, and the depth information is integrated into the dynamic normalization and scene determination process.

[0118] Specifically, the system performs the following calibration steps before deployment: calibrating the camera intrinsic parameter K, distortion correction parameters, and the translation vector of the camera optical center relative to the body, and recording the calibration uncertainty for confidence assessment.

[0119] Specifically, the system meets end-to-end real-time performance constraints, with a typical end-to-end latency of less than 40ms (from image acquisition to ground station display) and a target tracking frame rate of no less than 25 frames per second (FPS).

[0120] The UAV pointing calibration and landing point prediction method provided in this embodiment of the invention first acquires a real-time image sequence, and based on the real-time image sequence, uses a target tracking algorithm to obtain a target pixel coordinate sequence; then, it acquires the attitude angle data of the gimbal inertial measurement unit (IMU) and the airborne IMU, and fuses the attitude angle data of the gimbal IMU and the airborne IMU to obtain fused attitude angle data; then, it uses the camera intrinsic parameter matrix to perform an inverse transformation on the target pixel coordinate sequence to obtain the target direction vector in the camera coordinates, and dynamically normalizes the target direction vector in the camera coordinates to obtain the dynamically normalized target direction vector in the camera coordinates; finally, it calculates the target direction vector based on the fused attitude angle data. The rotation matrix is ​​calculated, and the target direction vector in the dynamically normalized camera coordinates is converted into the target direction vector in the UAV body coordinates based on the rotation matrix and the translation vector. Then, according to the scene to which the target belongs, the target direction vector in the UAV body coordinates is scaled or simplified, and the target direction vector in the scaled or simplified UAV body coordinates is dynamically normalized to obtain the target direction vector in the dynamically normalized UAV body coordinates. The scene is divided into near-range scene and far-range scene. Finally, based on the target direction vector in the dynamically normalized UAV body coordinates, the target predicted pixel landing point coordinates are obtained. This invention effectively suppresses errors from a single IMU by fusing multi-source attitude angle data from a gimbal IMU and an airborne IMU, obtaining high-precision attitude parameters and significantly improving attitude stability and noise resistance. It performs inverse transformation and dynamic normalization on the target pixel coordinate sequence obtained from the real-time image sequence using a target tracking algorithm, yielding a dynamically normalized target direction vector in the camera coordinates. This vector, combined with a translation vector and a rotation matrix calculated based on the fused attitude angle data, is then transformed to obtain the target direction vector in the UAV body coordinates. This target direction vector in the UAV body coordinates undergoes scale correction or simplification based on the target's scene and dynamic normalization. Based on this dynamically normalized target direction vector in the UAV body coordinates, the predicted target pixel landing point coordinates are accurately obtained, significantly reducing landing point prediction deviation. This enables real-time pointing guidance and landing point prediction for ground or air targets, solving the problems of insufficient real-time pointing control, poor attitude stability, and large landing point prediction deviations in existing UAVs for real-time indication and landing point prediction of maneuvering targets.

[0121] Example 2: like Figure 4 As shown, this embodiment provides a UAV pointing calibration and landing point prediction device for performing the above-described UAV pointing calibration and landing point prediction method, including: The acquisition module 11 is used to acquire a real-time image sequence and, based on the real-time image sequence, use a target tracking algorithm to obtain a target pixel coordinate sequence; The fusion module 12 is connected to the acquisition module 11 and is used to acquire the attitude angle data of the gimbal inertial measurement unit (IMU) and the airborne IMU, and fuse the attitude angle data of the gimbal IMU and the airborne IMU to obtain the fused attitude angle data. The inverse transformation normalization module 13 is connected to the acquisition and fusion module 12. It is used to perform an inverse transformation on the target pixel coordinate sequence using the camera intrinsic parameter matrix to obtain the target direction vector in the camera coordinates, and to dynamically normalize the target direction vector in the camera coordinates to obtain the dynamically normalized target direction vector in the camera coordinates. The calculation and transformation module 14, connected to the inverse transformation normalization module 13, is used to calculate the rotation matrix based on the fused attitude angle data, and convert the target direction vector in the dynamically normalized camera coordinates into the target direction vector in the UAV body coordinates based on the rotation matrix and the translation vector. The normalization module 15 is connected to the calculation and conversion module 14. It is used to perform scale correction or simplification processing on the target direction vector in the UAV body coordinates according to the scene to which the target belongs, and to dynamically normalize the target direction vector in the UAV body coordinates after scale correction or simplification processing to obtain the target direction vector in the UAV body coordinates after dynamic normalization. The scene is divided into near-range scene and far-range scene. The module 16 is connected to the processing normalization module 15 and is used to obtain the target predicted pixel landing point coordinates based on the target direction vector in the dynamically normalized UAV body coordinates.

[0122] Furthermore, the acquisition module 11 specifically includes: The positioning unit is used to locate the initial position of the target from the first frame or the detection trigger frame of the real-time image sequence; A determining unit is configured to determine the target pixel coordinate sequence in consecutive frames of the real-time image sequence based on the target's initial position.

[0123] Furthermore, the acquisition fusion module 12 specifically includes: The processing unit is used to perform interpolation and timestamp alignment processing on the attitude angle data of the gimbal IMU and the airborne IMU to obtain the attitude angle data of the gimbal IMU and the airborne IMU after interpolation and timestamp alignment processing. The fusion unit is used to fuse the attitude angle data of the gimbal IMU and the airborne IMU after interpolation and timestamp alignment using an extended Kalman filter (EKF) to obtain the fused attitude angle data.

[0124] Furthermore, the inverse transform normalization module 13 is specifically used for: The target direction vector in the dynamically normalized camera coordinates can be obtained using the following formula:

[0125] in, Let be the target direction vector in the camera coordinate system. This is the target direction vector in the dynamically normalized camera coordinates. Let be the magnitude of the target direction vector in camera coordinates. It is a tiny constant. This is the scaling adjustment coefficient. The estimated distance to the target.

[0126] Furthermore, the processing normalization module 15 specifically includes: The scaling correction unit is used to perform scaling correction on the target direction vector in the UAV body coordinates in response to the target's location being a near-field scene. The simplification processing unit is used to simplify the target direction vector in the UAV body coordinates in response to the target belonging to a distant scene.

[0127] Furthermore, the obtaining module 16 specifically includes: The calculation unit is used to calculate the spatial coordinates of the target pointing point based on the target direction vector in the dynamically normalized UAV body coordinates; The mapping unit is used to map the spatial coordinates of the target pointing point back to the image through inverse coordinate transformation to obtain the predicted pixel landing point coordinates of the target.

[0128] Furthermore, the obtaining module 16 also includes: The display unit is used to display the target predicted pixel landing point coordinates on the image.

[0129] Example 3: refer to Figure 5 This embodiment provides a UAV pointing calibration and landing point prediction device, including a memory 21 and a processor 22. The memory 21 stores a computer program, and the processor 22 is configured to run the computer program to execute the UAV pointing calibration and landing point prediction method in Embodiment 1.

[0130] The memory 21 is connected to the processor 22. The memory 21 can be a flash memory, a read-only memory or other memory, and the processor 22 can be a central processing unit or a microcontroller.

[0131] Example 4: This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the UAV pointing calibration and landing point prediction method in Embodiment 1 above.

[0132] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, computer program modules or other data. Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0133] In summary, the UAV pointing calibration and landing point prediction method, apparatus, and medium provided in this embodiment of the invention first acquire a real-time image sequence, and based on the real-time image sequence, use a target tracking algorithm to obtain a target pixel coordinate sequence; then, acquire the attitude angle data of the gimbal inertial measurement unit (IMU) and the airborne IMU, and fuse the attitude angle data of the gimbal IMU and the airborne IMU to obtain fused attitude angle data; then, use the camera intrinsic parameter matrix to perform an inverse transformation on the target pixel coordinate sequence to obtain the target direction vector in the camera coordinates, and dynamically normalize the target direction vector in the camera coordinates to obtain the dynamically normalized target direction vector in the camera coordinates; then, based on the fused... The attitude angle data is used to calculate a rotation matrix. Based on the rotation matrix and translation vector, the target direction vector in the dynamically normalized camera coordinates is converted into the target direction vector in the UAV body coordinates. Then, according to the scene to which the target belongs, the target direction vector in the UAV body coordinates is scaled or simplified, and the scaled or simplified target direction vector in the UAV body coordinates is dynamically normalized to obtain the target direction vector in the dynamically normalized UAV body coordinates. The scene is divided into near-range scene and far-range scene. Finally, based on the target direction vector in the dynamically normalized UAV body coordinates, the target predicted pixel landing point coordinates are obtained. This invention effectively suppresses errors from a single IMU by fusing multi-source attitude angle data from a gimbal IMU and an airborne IMU, obtaining high-precision attitude parameters and significantly improving attitude stability and noise resistance. It performs inverse transformation and dynamic normalization on the target pixel coordinate sequence obtained from the real-time image sequence using a target tracking algorithm, yielding a dynamically normalized target direction vector in the camera coordinates. This vector, combined with a translation vector and a rotation matrix calculated based on the fused attitude angle data, is then transformed to obtain the target direction vector in the UAV body coordinates. This target direction vector in the UAV body coordinates undergoes scale correction or simplification based on the target's scene and dynamic normalization. Based on this dynamically normalized target direction vector in the UAV body coordinates, the predicted target pixel landing point coordinates are accurately obtained, significantly reducing landing point prediction deviation. This enables real-time pointing guidance and landing point prediction for ground or air targets, solving the problems of insufficient real-time pointing control, poor attitude stability, and large landing point prediction deviations in existing UAVs for real-time indication and landing point prediction of maneuvering targets.

[0134] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for pointing calibration and landing point prediction of a UAV, characterized in that, The method includes: A real-time image sequence is acquired, and a target pixel coordinate sequence is obtained based on the real-time image sequence using a target tracking algorithm; The attitude angle data of the gimbal inertial measurement unit (IMU) and the airborne IMU are acquired, and the attitude angle data of the gimbal IMU and the airborne IMU are fused to obtain the fused attitude angle data. The target pixel coordinate sequence is inversely transformed using the camera intrinsic parameter matrix to obtain the target direction vector in the camera coordinates. The target direction vector in the camera coordinates is then dynamically normalized to obtain the target direction vector in the camera coordinates after dynamic normalization. The rotation matrix is ​​calculated based on the fused attitude angle data, and the target direction vector in the dynamically normalized camera coordinates is converted into the target direction vector in the UAV body coordinates based on the rotation matrix and the translation vector. Based on the scene to which the target belongs, the target direction vector in the UAV body coordinates is scaled or simplified, and the target direction vector in the UAV body coordinates after scale correction or simplification is dynamically normalized to obtain the target direction vector in the UAV body coordinates after dynamic normalization. The scene is divided into near-range scene and far-range scene. Based on the target direction vector in the dynamically normalized UAV body coordinates, the target predicted pixel landing point coordinates are obtained.

2. The method according to claim 1, characterized in that, The step of obtaining the target pixel coordinate sequence based on the real-time image sequence using a target tracking algorithm specifically includes: Locate the initial position of the target from the first frame or the detection trigger frame of the real-time image sequence; Based on the initial position of the target, the target pixel coordinate sequence is determined in consecutive frames of the real-time image sequence.

3. The method according to claim 1, characterized in that, The process of fusing the attitude angle data of the gimbal IMU and the airborne IMU to obtain the fused attitude angle data specifically includes: The attitude angle data of the gimbal IMU and the airborne IMU are interpolated and timestamped to obtain the attitude angle data of the gimbal IMU and the airborne IMU after interpolation and timestamping. The attitude angle data of the gimbal IMU and the airborne IMU, after interpolation and timestamp alignment processing, are fused using an extended Kalman filter (EKF) to obtain the fused attitude angle data.

4. The method according to claim 1, characterized in that, The step of dynamically normalizing the target direction vector in the camera coordinates to obtain the dynamically normalized target direction vector in the camera coordinates specifically includes: The target direction vector in the dynamically normalized camera coordinates can be obtained using the following formula: in, Let be the target direction vector in camera coordinates. This is the target direction vector in the dynamically normalized camera coordinates. Let be the magnitude of the target direction vector in camera coordinates. It is a tiny constant. This is the scaling factor. The estimated distance to the target.

5. The method according to claim 1, characterized in that, The step of scaling or simplifying the target direction vector in the UAV body coordinates according to the scene to which the target belongs specifically includes: In response to the fact that the target belongs to a close-range scene, the target direction vector in the UAV body coordinates is scaled and corrected. In response to the fact that the target belongs to a distant scene, the target direction vector in the UAV body coordinates is simplified.

6. The method according to claim 1, characterized in that, The process of obtaining the target prediction pixel landing point coordinates based on the target direction vector in the dynamically normalized UAV body coordinates specifically includes: Calculate the spatial coordinates of the target pointing point based on the target direction vector in the dynamically normalized UAV body coordinates; The spatial coordinates of the target pointing point are mapped back to the image through inverse coordinate transformation to obtain the predicted pixel landing point coordinates of the target.

7. The method according to claim 6, characterized in that, After mapping the spatial coordinates of the target pointing point back to the image through inverse coordinate transformation to obtain the predicted pixel landing point coordinates of the target, the method further includes: The coordinates of the predicted target pixel landing point are displayed on the image.

8. A device for pointing calibration and landing point prediction of unmanned aerial vehicles (UAVs), characterized in that, include: The acquisition module is used to acquire real-time image sequences and, based on the real-time image sequences, to obtain target pixel coordinate sequences using a target tracking algorithm; A fusion module is connected to the acquisition module and is used to acquire attitude angle data of the gimbal inertial measurement unit (IMU) and the airborne IMU, and fuse the attitude angle data of the gimbal IMU and the airborne IMU to obtain fused attitude angle data. The inverse transformation normalization module, connected to the acquisition and fusion module, is used to perform an inverse transformation on the target pixel coordinate sequence using the camera intrinsic parameter matrix to obtain the target direction vector in the camera coordinates, and to dynamically normalize the target direction vector in the camera coordinates to obtain the dynamically normalized target direction vector in the camera coordinates. The calculation and conversion module, connected to the inverse transformation and normalization module, is used to calculate the rotation matrix based on the fused attitude angle data, and convert the target direction vector in the dynamically normalized camera coordinates into the target direction vector in the UAV body coordinates based on the rotation matrix and the translation vector. The normalization module, connected to the calculation and conversion module, is used to perform scale correction or simplification processing on the target direction vector in the UAV body coordinates according to the scene to which the target belongs, and to dynamically normalize the target direction vector in the UAV body coordinates after scale correction or simplification processing to obtain the target direction vector in the UAV body coordinates after dynamic normalization. The scene is divided into near-range scene and far-range scene. The module is connected to the processing normalization module and is used to obtain the target predicted pixel landing point coordinates based on the target direction vector in the dynamically normalized UAV body coordinates.

9. A device for pointing calibration and landing point prediction of unmanned aerial vehicles (UAVs), characterized in that, It includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to implement the UAV pointing calibration and landing point prediction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the UAV pointing calibration and landing point prediction method as described in any one of claims 1-7.

Images