Aircraft target tracking method and system based on compensation prediction

Abstract

The application discloses an aircraft target tracking method and system based on compensation prediction, which is used for improving the target tracking accuracy in the image transmission delay scene. The method first acquires a target aircraft image frame sequence by using an airborne monocular camera, and extracts target center coordinates by using a small target detection algorithm to construct a position sequence. In view of the transmission delay of the image frame, current high-frequency IMU data is extracted and input into an LSTM-DKF model constructed by fusing an LSTM and a delay Kalman filter to predict and generate a process noise and an observation noise covariance matrix. The delay Kalman filter is initialized by using the above matrix to recursively predict the target position during the delay. When the delayed image frame is received, the filter state is backtracked and updated, and is recursively predicted to the current time again to output the compensated target position. Finally, the pixel deviation is calculated according to the compensated position, aircraft tracking control instructions are generated, and high-precision target tracking is realized.

Description

Technical Field

[0001] This invention relates to the field of aircraft tracking technology, and in particular to an aircraft target tracking method and system based on compensation prediction. Background Technology

[0002] In recent years, aircraft, with their advantages of small size, high maneuverability, and low cost, have become an important direction for improving the autonomous capabilities of aircraft by autonomously tracking enemy or target aircraft in complex dynamic environments using visual means. However, in practical applications, target tracking tasks face a common but critical problem: due to the inherent delay in image frame transmission and processing, there is a time asynchrony between visual information and the control system, directly affecting the tracking accuracy and response speed of the aircraft. Furthermore, after the image enters the onboard processor, computationally intensive tasks such as target detection and classification must be performed, further amplifying the delay problem of visual information.

[0003] Traditional target tracking methods, such as attitude adjustment based on image center position or state estimation based on Kalman filtering, typically assume that image observation data is real-time synchronous input, ignoring the latency characteristics of visual perception in high-speed tasks. This assumption can maintain basic control accuracy under low-speed or static targets, but when facing high-speed, nonlinear, long-range small target aircraft, the image information used by the controller is often lagging behind by one or more control cycles, causing the accumulation of estimation errors, lag in attitude adjustment, and in severe cases, even target loss or control instability.

[0004] Some studies have attempted to compensate for image latency using the Delayed Kalman Filter (DKF) method, but it still relies on a statically set noise covariance matrix, making it difficult to maintain prediction stability and accuracy in complex dynamic scenes. Furthermore, while reinforcement learning methods theoretically possess strong generalization capabilities, they face practical problems such as large training sample requirements, difficulties in actual deployment, and weak real-time control, and they also fail to adequately address the sensor latency issue.

[0005] Therefore, there is an urgent need for a target tracking method that can effectively identify and compensate for image frame transmission delays and dynamically fuse and predict targets by combining IMU high-frequency data and visual low-frequency data. This would enable the aircraft to maintain high-precision and high-responsive tracking and control capabilities even when operating at high speeds and with asynchronous information, and to maintain stable and effective strike performance, especially when facing small, fast-moving targets. Summary of the Invention

[0006] To address at least one of the aforementioned technical problems, this invention proposes a method and system for tracking aircraft targets based on compensation prediction.

[0007] The first aspect of this invention provides a method for tracking aircraft targets based on compensation prediction, comprising:

[0008] Image frame sequence data of the target aircraft is acquired based on an airborne monocular camera. The image frame sequence data is analyzed based on a small target detection algorithm to determine the center position coordinates of the target aircraft and construct a center position coordinate sequence.

[0009] Obtain the image frame transmission delay information of the monocular camera; when there is a transmission delay, extract the high-frequency IMU data at the current moment.

[0010] The high-frequency IMU data is input into the LSTM-DKF compensation prediction model constructed based on the fusion of LSTM and delayed Kalman filter to generate the process noise covariance matrix Q and the observation noise covariance matrix R of the delayed Kalman filter.

[0011] The delayed Kalman filter is initialized based on the matrix Q and matrix R, and the position of the target aircraft image center point during the image frame transmission delay is recursively predicted based on the high-frequency IMU data.

[0012] When a delayed image frame is received, the state variables of the Kalman filter are updated back to the delayed time, and the compensated target aircraft position is recalculated back to the current time.

[0013] The corrected pixel difference is calculated based on the compensated target aircraft position, and a tracking control command is generated based on the pixel difference.

[0014] In this solution, the step of acquiring image frame sequence data of the target aircraft based on an airborne monocular camera, analyzing the image frame sequence data based on a small target detection algorithm to determine the center position coordinates of the target aircraft, and constructing a center position coordinate sequence specifically involves:

[0015] Image frame sequence data of the target aircraft are acquired by the airborne monocular camera of the tracking aircraft, and a small target detection algorithm is constructed based on global detectors and local detectors.

[0016] The image frame sequence data is imported into a global detector for YOLOv5 target detection to determine the region where the target aircraft is located in the image frame sequence data and to determine the recognition accuracy. When the recognition accuracy is less than a preset value, moving target detection is performed on the image frame sequence data based on sparse optical flow motion compensation and inter-frame difference method to determine the moving target in the image frame sequence data. The moving target is then identified by the aircraft classifier, and the region where the target aircraft with high confidence is located in the image frame sequence data is output.

[0017] The region where the target aircraft is located in the image frame sequence data is input into the local detector. Based on the LightTrack target tracker, the target aircraft is identified and selected in the region where the target aircraft is located. The center position coordinates of the target aircraft in the image coordinate system in each image frame are determined, and the center position coordinates are arranged in the time order of the image frames to construct a center position coordinate sequence.

[0018] In this solution, obtaining the image frame transmission delay information from the monocular camera, and extracting the high-frequency IMU data at the current moment when a transmission delay exists, specifically involves:

[0019] The frame generation timestamp data of the image frame sequence data acquired by the monocular camera and the current system timestamp data of the image frame arriving at the onboard processing system of the tracking aircraft are obtained, and the difference between the frame generation timestamp and the current system timestamp is calculated.

[0020] The difference is compared with a preset frame period threshold. When the difference is greater than the frame period threshold, it is determined that there is an image frame transmission delay and the delay duration is recorded to obtain image frame transmission delay information.

[0021] If there is a transmission delay, the high-frequency IMU data of the tracking aircraft at the current moment is obtained. The IMU data includes angular velocity data, acceleration data, and attitude quaternion data.

[0022] In this scheme, the step of inputting the high-frequency IMU data into the LSTM-DKF compensation prediction model constructed based on the fusion of LSTM and delayed Kalman filter to generate the process noise covariance matrix Q and observation noise covariance matrix R of the delayed Kalman filter is as follows:

[0023] An LSTM-DKF compensation prediction model is constructed based on LSTM and a delayed Kalman filter. The time-series high-frequency IMU data is input into the LSTM network of the compensation prediction model, and the hidden state h is calculated through gating mechanisms including forget gates, input gates, and output gates. t The Tanh activation function is used to generate a 128-dimensional hidden layer output.

[0024] From the hidden state h t Separate the two fully connected branches;

[0025] The first branch outputs a 3D vector, which is reconstructed into the Cholesky lower triangular matrix L of the process noise covariance matrix Q. Q Satisfying Q=L Q L Q T and for L Q Apply an exponential activation function to the diagonal elements;

[0026] The second branch outputs a 3D vector, which is reconstructed into the Cholesky lower triangular matrix L of the observation noise covariance matrix R. R Satisfying R=L R L R T and for L R Apply an exponential activation function to the diagonal elements;

[0027] L based on reconstruction Q and L R Calculate the final process noise covariance matrix Q and observation noise covariance matrix R.

[0028] In this scheme, the initialization of the delayed Kalman filter based on the matrix Q and matrix R, and the recursive prediction of the target aircraft image center point position during the image frame transmission delay based on the high-frequency IMU data, specifically involves:

[0029] A state vector is constructed based on the high-frequency IMU data and the center position coordinate sequence. The state vector includes the attitude quaternion of the tracking aircraft, angular velocity, acceleration, relative position and relative velocity between the tracking aircraft and the target aircraft, and the center position coordinates of the target aircraft in the image coordinate system from the perspective of the tracking aircraft.

[0030] Construct a state transition matrix F based on the state vector. k The state transition matrix describes the continuous-time evolution relationship of attitude quaternions, relative position, and relative velocity;

[0031] Construct the control input vector u based on high-frequency IMU data. k The control input vector includes angular velocity, acceleration, and attitude information. A control input matrix B is constructed based on the control input vector. k The control input matrix maps the angular velocity and acceleration in the high-frequency IMU data to the state space;

[0032] An observation matrix H is constructed based on the geometric projection relationship between the image coordinate system and the collective coordinate system of the tracking aircraft. The observation matrix establishes a mapping relationship between the state vector and the coordinates of the center position of the target aircraft image.

[0033] Based on the state transition matrix and control input matrix, the predicted state vector is calculated using the state prediction equation. The specific expression is:

[0034]

[0035] Update the prediction error covariance matrix P based on the matrix Q. k∣k-1 The specific expression is:

[0036] Pk∣k-1 =F k P k-1 F k T +Q,

[0037] Where k represents the current time step, P k-1 Let be the final error covariance matrix of the previous time step, and be the initial preset value. k|k-1 represents time step k-1 to k.

[0038] When there is an image frame transmission delay, the Kalman gain K is calculated based on the observation noise covariance matrix R and the observation matrix H. k The specific expression is:

[0039] K k =P k∣k-1 H T HP k∣k-1 H T +R) -1 ,

[0040] The center position coordinates z of the delayed image frame are fused using Kalman gain. k And update the final error covariance matrix at the current time, with the specific formula as follows:

[0041]

[0042] P k =(IK k H)P k∣k-1 ,

[0043] Where z k That is, the center coordinates of the target aircraft in the latest image frame arrived by the onboard system tracking the aircraft, where I is the coordinate of P. k∣k-1 Identity matrices of the same dimension;

[0044] From the updated state estimation vector Extract the center coordinates of the target aircraft in the image coordinate system to obtain the center point position of the target aircraft image during the image frame transmission delay.

[0045] In this scheme, when a delayed image frame is received, the process of backtracking to the delayed time to update the Kalman filter state variable and recursively outputting the compensated target aircraft position at the current time is as follows:

[0046] When a delayed image frame is received, the calculated final state estimation vector is stored. And error covariance matrix P k ;

[0047] When entering the next time step k+1, the stored As a new state input Store P k As the new error covariance input P (k+1)-1 The target spacecraft image center point position during the image frame transmission delay in the next time step k+1 is recursively calculated to obtain the latest compensated target spacecraft position at the current time.

[0048] In this scheme, the step of calculating the corrected pixel difference based on the compensated target aircraft position and generating tracking control commands based on the pixel difference specifically includes:

[0049] Based on the compensated target aircraft position, the horizontal pixel difference e between the target center point and the image center point is obtained. x The pixel difference between the vertical axis and the vertical axis e y ;

[0050] According to the pixel difference e x and e y Construct the attitude control matrix w b and lift control quantity f b The attitude control matrix w b and lift control quantity f b The calculation formula is:

[0051]

[0052] in To track the roll angle of the aircraft, θ is the pitch angle, θ d The preset pitch angle offset is given by m, where m is the mass of the aircraft, and v is the velocity. z is the vertical velocity, g is the gravitational acceleration, and k1 to k5 are the control gain coefficients;

[0053] According to the attitude control matrix w b Generate angular velocity commands for the roll, pitch, and yaw channels, and simultaneously adjust the lift control value f. b Generate total lift commands, combine them to form the tracking control commands, and output them to the flight control system of the tracking aircraft.

[0054] A second aspect of the present invention also provides an aircraft target tracking system based on compensation prediction. The system includes a memory and a processor. The memory includes a program for an aircraft target tracking method based on compensation prediction. When the processor executes the program for the aircraft target tracking method based on compensation prediction, it performs the following steps:

[0055] Image frame sequence data of the target aircraft is acquired based on an airborne monocular camera. The image frame sequence data is analyzed based on a small target detection algorithm to determine the center position coordinates of the target aircraft and construct a center position coordinate sequence.

[0056] Obtain the image frame transmission delay information of the monocular camera; when there is a transmission delay, extract the high-frequency IMU data at the current moment.

[0057] The high-frequency IMU data is input into the LSTM-DKF compensation prediction model constructed based on the fusion of LSTM and delayed Kalman filter to generate the process noise covariance matrix Q and the observation noise covariance matrix R of the delayed Kalman filter.

[0058] The delayed Kalman filter is initialized based on the matrix Q and matrix R, and the position of the target aircraft image center point during the image frame transmission delay is recursively predicted based on the high-frequency IMU data.

[0059] When a delayed image frame is received, the state variables of the Kalman filter are updated back to the delayed time, and the compensated target aircraft position is recalculated back to the current time.

[0060] The corrected pixel difference is calculated based on the compensated target aircraft position, and a tracking control command is generated based on the pixel difference.

[0061] This invention discloses a method and system for aircraft target tracking based on compensation prediction, used to improve target tracking accuracy in scenarios with image transmission delay. The method first acquires a sequence of target aircraft image frames using an airborne monocular camera, and extracts the target center coordinates using a small target detection algorithm to construct a position sequence. For cases with image frame transmission delay, current high-frequency IMU data is extracted and input into an LSTM-DKF model constructed by fusing LSTM and a delayed Kalman filter, predicting and generating process noise and observation noise covariance matrices. The delayed Kalman filter is initialized using these matrices, recursively predicting the target position during the delay period. When a delayed image frame is received, the filter state is backtracked and updated, and the prediction is repeated to the current time, outputting the compensated target position. Finally, pixel deviations are calculated based on the compensated position to generate aircraft tracking control commands, achieving high-precision target tracking. Attached Figure Description

[0062] Figure 1 A flowchart of an aircraft target tracking method based on compensation prediction according to the present invention is shown;

[0063] Figure 2 The flowchart illustrating the construction of the center position coordinate sequence according to the present invention is shown;

[0064] Figure 3The flowchart illustrating the present invention for obtaining image frame transmission delay information is shown.

[0065] Figure 4 A block diagram of an aircraft target tracking system based on compensation prediction according to the present invention is shown. Detailed Implementation

[0066] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0067] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0068] Figure 1 A flowchart of an aircraft target tracking method based on compensation prediction according to the present invention is shown.

[0069] like Figure 1 As shown, the first aspect of the present invention provides a method for tracking aircraft targets based on compensation prediction, comprising:

[0070] S102, acquire image frame sequence data of the target aircraft based on the airborne monocular camera, analyze the image frame sequence data based on the small target detection algorithm, determine the center position coordinates of the target aircraft, and construct the center position coordinate sequence;

[0071] S104, Obtain the image frame transmission delay information of the monocular camera; when there is a transmission delay, extract the high-frequency IMU data at the current moment.

[0072] S106, The high-frequency IMU data is input into the LSTM-DKF compensation prediction model constructed based on the fusion of LSTM and delayed Kalman filter to generate the process noise covariance matrix Q and the observation noise covariance matrix R of the delayed Kalman filter;

[0073] S108, initialize the delayed Kalman filter based on the matrix Q and matrix R, and recursively predict the position of the target aircraft image center point during the image frame transmission delay based on the high-frequency IMU data;

[0074] S110, when a delayed image frame is received, backtrack to the delayed time to update the Kalman filter state, and recursively output the compensated target aircraft position at the current time;

[0075] S112, calculate the corrected pixel difference based on the compensated target aircraft position, and generate a tracking control command based on the pixel difference.

[0076] It should be noted that by utilizing an airborne monocular camera combined with a small target detection algorithm to extract the image center coordinates of the target aircraft, a continuous position sequence is constructed, enabling preliminary target identification and tracking. Subsequently, the system detects whether there is a transmission delay in the image frames. If so, it immediately extracts the current high-frequency IMU data and uses a fusion model constructed with LSTM and a Delayed Kalman Filter (DKF) to generate a process noise (Q) and observation noise (R) covariance matrix that more closely reflects the current system dynamics, improving the system's ability to model state uncertainties. Based on this, an IMU data-driven state transition prediction mechanism is used to estimate the target's center position in advance during the period before the image arrives, ensuring that the control logic is not interrupted. When the delayed image frame arrives, the system further corrects the estimation error by backtracking historical states and updating the filter state variables, and recursively pushes the state to the current moment, outputting the compensated target position. Finally, precise tracking control commands are generated based on the image center difference, enabling the aircraft to continuously and stably lock onto and approach the target, achieving robust tracking control of high-speed or maneuvering targets, effectively improving the system's real-time performance, robustness, and strike accuracy.

[0077] Figure 2 A flowchart illustrating the construction of the center position coordinate sequence according to the present invention is shown.

[0078] According to an embodiment of the present invention, the step of acquiring image frame sequence data of the target aircraft based on an airborne monocular camera, analyzing the image frame sequence data based on a small target detection algorithm, determining the center position coordinates of the target aircraft, and constructing a center position coordinate sequence specifically involves:

[0079] S202: Based on the image frame sequence data of the target aircraft obtained by the airborne monocular camera of the tracking aircraft, a small target detection algorithm is constructed based on the global detector and the local detector.

[0080] S204, the image frame sequence data is imported into a global detector for YOLOv5 target detection to determine the region where the target aircraft is located in the image frame sequence data and to determine the recognition accuracy. When the recognition accuracy is less than a preset value, moving target detection is performed on the image frame sequence data based on sparse optical flow motion compensation and inter-frame difference method to determine the moving target in the image frame sequence data. The moving target is then identified by the aircraft classifier, and the region where the target aircraft with high confidence is located in the image frame sequence data is output.

[0081] S206, the region where the target aircraft is located in the image frame sequence data is input into the local detector. The target aircraft is identified and selected based on the LightTrack target tracker in the region where the target aircraft is located. The center position coordinates of the target aircraft in the image coordinate system in each image frame are determined, and the center position coordinates are arranged in the time order of the image frames to construct a center position coordinate sequence.

[0082] It should be noted that the global-local detection and tracking architecture enables small target detection. The global detector includes a YOLOv5 target detection module, a sparse optical flow motion compensation and inter-frame difference target detection module, and an aircraft classifier module. The aircraft classifier is constructed using a convolutional neural network. The global detector is mainly responsible for initially searching for potential target regions in the entire image, using the lightweight and real-time deep learning detection algorithm YOLOv5 to identify and locate target aircraft in the image. When the recognition confidence is low, the system introduces sparse optical flow and inter-frame difference to supplement the detection of moving regions in the image sequence. Sparse optical flow is used to track the motion trajectory of feature points in the image, and inter-frame difference can effectively capture inter-frame changing regions. The combination of the two improves the detection rate of low-visibility targets. Subsequently, the local detector calls LightTrack, a target tracking network based on a lightweight Transformer, to further accurately select candidate target regions and associate them with consecutive frames, finally extracting the center coordinates of the target aircraft in each frame. This detection and tracking mechanism, which combines global and local information and integrates multiple sources, not only improves the accuracy and stability of target recognition, but also maintains reliable extraction of the target aircraft's position even when the target is small, frequently occluded, or the image quality fluctuates.

[0083] Figure 3 A flowchart illustrating the present invention for obtaining image frame transmission delay information is shown.

[0084] According to an embodiment of the present invention, the step of obtaining the image frame transmission delay information of the monocular camera, and extracting the high-frequency IMU data at the current moment when a transmission delay exists, specifically involves:

[0085] S302, acquire the frame generation timestamp data of the image frame sequence data acquired by the monocular camera and the current system timestamp data of the image frame arriving at the airborne processing system of the tracking aircraft, and calculate the difference between the frame generation timestamp and the current system timestamp;

[0086] S304, compare the difference with a preset frame period threshold. When the difference is greater than the frame period threshold, determine that there is an image frame transmission delay and record the delay duration to obtain image frame transmission delay information.

[0087] S306, if there is a transmission delay, acquire the high-frequency IMU data of the tracking aircraft at the current moment, the IMU data including angular velocity data, acceleration data, and attitude quaternion data.

[0088] It's important to note that the system determines whether to activate a compensation mechanism to improve the real-time performance and accuracy of target tracking by assessing the presence of transmission delays in image frames. In scenarios involving high-speed aircraft operation or rapid target movement, visual information delays can lead to control commands being based on outdated image data, resulting in control deviations or even target loss. Therefore, the system needs to monitor the time between acquisition and processing of each image frame. Specifically, the difference between the image frame's generation timestamp (representing the original time of image acquisition) and the timestamp when the system receives and processes the frame is compared to a preset frame period threshold to determine if there is a delay exceeding the normal processing cycle. If a delay exists, the image frame information is considered no longer real-time, triggering a prediction mechanism based on IMU data. Technically, IMU data includes angular velocity, acceleration, and attitude quaternions. Angular velocity and acceleration are core data describing the aircraft's dynamic response, while quaternions provide an efficient and stable three-dimensional attitude representation, avoiding gimbal lock issues that may occur in Euler angle representations. By acquiring high-frequency IMU data and aligning it with image data, the system can provide continuous state input even when the image is not reached or is delayed, ensuring that the state prediction chain remains unbroken and achieving an uninterrupted target tracking process. The high frequency is 100Hz.

[0089] According to an embodiment of the present invention, the step of inputting the high-frequency IMU data into an LSTM-DKF compensation prediction model constructed based on the fusion of LSTM and delayed Kalman filter to generate the process noise covariance matrix Q and the observation noise covariance matrix R of the delayed Kalman filter specifically involves:

[0090] An LSTM-DKF compensation prediction model is constructed based on LSTM and a delayed Kalman filter. The time-series high-frequency IMU data is input into the LSTM network of the compensation prediction model, and the hidden state h is calculated through gating mechanisms including forget gates, input gates, and output gates. t The Tanh activation function is used to generate a 128-dimensional hidden layer output.

[0091] From the hidden state h t Separate the two fully connected branches;

[0092] The first branch outputs a 3D vector, which is reconstructed into the Cholesky lower triangular matrix L of the process noise covariance matrix Q. Q Satisfying Q=L Q L Q T and for LQ Apply an exponential activation function to the diagonal elements;

[0093] The second branch outputs a 3D vector, which is reconstructed into the Cholesky lower triangular matrix L of the observation noise covariance matrix R. R Satisfying R=L R L R T and for L R Apply an exponential activation function to the diagonal elements;

[0094] L based on reconstruction Q and L R Calculate the final process noise covariance matrix Q and observation noise covariance matrix R.

[0095] It should be noted that by constructing an LSTM-DKF compensation prediction model that integrates LSTM and Delayed Kalman Filter (DKF), dynamically generating the process noise covariance matrix Q and the observation noise covariance matrix R, its core function is to improve the accuracy and robustness of Kalman filtering in state estimation of nonlinear, highly dynamic systems. The covariance matrix Q is used to describe the uncertainty in the system state transition process, while R is used to measure the reliability of the observation data. Both are key parameters in the Kalman filter for adjusting the prediction and observation confidence weights. In traditional DKF methods, Q and R are mostly empirically set or fixed values, making it difficult to adapt to dynamically changing flight environments or sensor noise characteristics. LSTM, as a recurrent neural network structure with time memory capabilities, can learn and model the distribution trends of system and sensor noise during flight using historical high-frequency IMU time-series data (including angular velocity, acceleration, and attitude quaternions). In this model, the hidden state is calculated through a gating mechanism (forget gate, input gate, and output gate), and then the outputs from two fully connected branches are used to construct the Cholesky lower triangular matrices of Q and R. An exponential activation function is applied to the diagonal elements to ensure positive definiteness, ultimately yielding dynamically updated Q and R matrices. Since LSTM can fully exploit the potential noise evolution characteristics in time-series data, it can provide more timely and accurate covariance estimation for DKF, resulting in higher stability and accuracy of the filter in the state prediction and image delay compensation update stages, significantly enhancing the system's adaptability in dynamic and complex environments.

[0096] According to an embodiment of the present invention, the step of initializing the delayed Kalman filter based on the matrix Q and the matrix R, and recursively predicting the target aircraft image center point position during the image frame transmission delay based on the high-frequency IMU data, specifically involves:

[0097] A state vector is constructed based on the high-frequency IMU data and the center position coordinate sequence. The state vector includes the attitude quaternion of the tracking aircraft, angular velocity, acceleration, relative position and relative velocity between the tracking aircraft and the target aircraft, and the center position coordinates of the target aircraft in the image coordinate system from the perspective of the tracking aircraft.

[0098] Construct a state transition matrix F based on the state vector. k The state transition matrix describes the continuous-time evolution relationship of attitude quaternions, relative position, and relative velocity;

[0099] Construct the control input vector u based on high-frequency IMU data. k The control input vector includes angular velocity, acceleration, and attitude information. A control input matrix B is constructed based on the control input vector. k The control input matrix maps the angular velocity and acceleration in the high-frequency IMU data to the state space;

[0100] An observation matrix H is constructed based on the geometric projection relationship between the image coordinate system and the collective coordinate system of the tracking aircraft. The observation matrix establishes a mapping relationship between the state vector and the coordinates of the center position of the target aircraft image.

[0101] Based on the state transition matrix and control input matrix, the predicted state vector is calculated using the state prediction equation. The specific expression is:

[0102]

[0103] Update the prediction error covariance matrix P based on the matrix Q. k∣k-1 The specific expression is:

[0104] P k∣k-1 =F k P k-1 F k T +Q,

[0105] Where k represents the current time step, P k-1 Let be the final error covariance matrix of the previous time step, and be the initial preset value. k|k-1 represents time step k-1 to k.

[0106] It should be noted that the state space refers to a set of vectors consisting of the attitude quaternions, angular velocity, acceleration, relative position and relative velocity of the tracking aircraft with respect to the target aircraft, and the center position coordinates of the target aircraft in the image coordinate system; the mapping relationship is the mapping relationship between the state vector and the center position coordinates of the target aircraft in the image, which refers to the mathematical model established by the observation matrix H, used to convert the aircraft motion state information contained in the state vector into two-dimensional coordinate positions in the image plane coordinate system; where P k∣k-1 The uncertainty of state prediction is characterized by Q, which quantifies the impact of IMU measurement noise on the uncertainty of state prediction, and R is used to balance the confidence weights of the predicted state and the delayed image observations.

[0107] When there is an image frame transmission delay, the Kalman gain K is calculated based on the observation noise covariance matrix R and the observation matrix H. k The specific expression is:

[0108] K k =P k∣k-1 H T HP k∣k-1 H T +R) -1 ,

[0109] The center position coordinates z of the delayed image frame are fused using Kalman gain. k And update the final error covariance matrix at the current time, the specific formula is:

[0110]

[0111] P k =(IK k H)P k∣k-1 ,

[0112] Where z k That is, the center coordinates of the target aircraft in the latest image frame arrived by the onboard system tracking the aircraft, where I is the coordinate of P. k∣k-1 Identity matrices of the same dimension;

[0113] From the updated state estimation vector Extract the center coordinates of the target aircraft in the image coordinate system to obtain the center point position of the target aircraft image during the image frame transmission delay.

[0114] It should be noted that by combining high-frequency IMU data with the Delayed Kalman Filter (DKF) algorithm, high-precision prediction and compensation of the target aircraft's image center position during image frame transmission delay is achieved. The core of this approach lies in dynamically initializing the filter and performing state estimation and updating based on a state recursion mechanism. The implementation logic first involves constructing a complete state vector. This vector includes not only the tracking aircraft's attitude quaternions, angular velocity, and acceleration, but also its relative position and velocity with the target aircraft, as well as the target's center position in the image coordinate system, forming a multi-dimensional state representation that integrates visual and inertial information. Subsequently, a state transition matrix is ​​constructed based on this state vector. This matrix describes the evolution of the state over continuous time, including the quaternion integral of attitude changes and the influence of velocity on position. Simultaneously, the system constructs a control input vector based on the high-frequency angular velocity and acceleration data measured by the IMU, and maps these control quantities to the state space through the control input matrix, completing the state prediction equation and predicting the target state at the next moment. To measure the reliability of the prediction, the system needs to update the prediction error covariance matrix using the process noise covariance matrix Q. The Q matrix can reflect the current system's uncertainty based on historical sensor data, improving the filter's response to environmental changes. Simultaneously, the system constructs the observation matrix H based on the projection mapping relationship between the image coordinate system and the aircraft's body coordinate system, realizing the mathematical mapping between the state vector and the image center position. When the delayed image frame z... k Upon arrival, the system uses H and the observation noise covariance matrix R to calculate the Kalman gain, and then converts the observed value z... k Compared with the predicted value H k∣k-1 The difference is calculated, and the state estimation vector and covariance matrix are corrected using the Kalman update formula to obtain the current optimal estimated state. Finally, the center position coordinates in the image coordinate system are extracted from the updated state vector to achieve compensated prediction of the target aircraft's true position during image delay. Since Q and R directly determine the filter's trust weights for the state prediction and observation values, the Q and R generated by LSTM can dynamically reflect the sensor noise level and environmental change characteristics, ensuring the adaptability and accuracy of state updates. Furthermore, the continuous state modeling combined with IMU high-frequency data allows for continuous deduction of the state change trajectory through the physical model even when the image does not arrive in time, thus achieving reliable image delay compensation and ensuring uninterrupted and undevised target tracking. This mechanism significantly improves the system's tracking stability and control real-time performance in high-speed dynamic scenarios.

[0115] According to an embodiment of the present invention, when an image frame with a transmission delay is received, the Kalman filter state is updated back to the delay time, and the compensated target aircraft position is recalculated to the current time. Specifically, this involves:

[0116] When a delayed image frame is received, the calculated final state estimation vector is stored. And error covariance matrix P k ;

[0117] When entering the next time step k+1, the stored As a new state input Store P k As the new error covariance input P (k+1)-1 The target spacecraft image center point position during the image frame transmission delay in the next time step k+1 is recursively calculated to obtain the latest compensated target spacecraft position at the current time.

[0118] It should be noted that after completing the state update at the current time step k, the system stores the calculated final state estimation vector and error covariance matrix in a cache; when entering the next time step k+1, the system automatically caches the stored... As a new state input and store P k As the new error covariance input P (k+1)-1 A control input vector u at time step k+1 is constructed based on real-time acquired high-frequency IMU data. k The predicted state is calculated using the state transition equation, and P is used. (k+1)∣k Update the prediction error covariance matrix; if there is an image frame transmission delay at this time step, the system calculates the Kalman gain based on the observation noise covariance matrix R and the observation matrix H, and fuses the observation value z when the delayed image frame is received. k+1 The updated state and covariance form a closed-loop temporal recursion chain, enabling continuous evolution of state estimation and uncertainty propagation. This temporal recursion mechanism directly uses the final state estimate and error covariance at each time step as the prediction benchmark for the next time step, constructing a seamless state propagation path. This ensures the physical continuity of the dynamic model (such as attitude quaternions, relative position / velocity) and avoids control command jumps caused by frame rate differences.

[0119] According to an embodiment of the present invention, the step of calculating the corrected pixel difference based on the compensated target aircraft position and generating a tracking control command based on the pixel difference specifically includes:

[0120] Based on the compensated target aircraft position, the horizontal pixel difference e between the target center point and the image center point is obtained. x The pixel difference between the vertical axis and the vertical axis e y ;

[0121] According to the pixel difference e x and e y Construct the attitude control matrix w band lift control quantity f b The attitude control matrix w b and lift control quantity f b The calculation formula is:

[0122]

[0123] in To track the roll angle of the aircraft, θ is the pitch angle, θ d The preset pitch angle offset is given by m, where m is the mass of the aircraft, and v is the velocity. z is the vertical velocity, g is the gravitational acceleration, and k1 to k5 are the control gain coefficients;

[0124] According to the attitude control matrix w b Generate angular velocity commands for the roll, pitch, and yaw channels, and simultaneously adjust the lift control value f. b Generate total lift commands, combine them to form the tracking control commands, and output them to the flight control system of the tracking aircraft.

[0125] It should be noted that the controllers for the X and Z axes of the tracking aircraft are P controllers, while the controller for the Y axis is two P controllers connected in parallel. The purpose of the X-axis controller is to stabilize the aircraft's Roll angle, preventing drastic changes in the enemy aircraft's position in the image due to variations in the Roll angle. The purpose of the Y-axis controller is to reduce the error between the enemy aircraft and the image center on the Y-axis, while maintaining a downward tilt to ensure that a portion of the lift generated by the aircraft can be used for forward thrust. The purpose of the Z-axis controller is to gradually reduce the error on the X-axis of the image to zero.

[0126] The purpose of the above three-axis controller is to center the image of the enemy aircraft. Additionally, slightly tilting the nose allows a portion of the aircraft's lift to be diverted horizontally, propelling the aircraft forward horizontally. Lift controller f b It consists of two parts: gravity and Z-axis acceleration, where (k5*e y ) represents the desired velocity in the vertical direction, m((k5*e y -v z The sum of θ and g) constitutes the magnitude of the vertical component of the lift force that the entire aircraft should have. Dividing this by cosθ gives the magnitude of the total lift force, where the horizontal component is determined by θ. d Decision, θ d The larger the force, the greater the horizontal component of the force on the aircraft.

[0127] Figure 4 A block diagram of an aircraft target tracking system based on compensation prediction according to the present invention is shown.

[0128] A second aspect of the present invention also provides an aircraft target tracking system 4 based on compensation prediction. The system includes a memory 41 and a processor 42. The memory includes a program for an aircraft target tracking method based on compensation prediction. When the processor executes the program for the aircraft target tracking method based on compensation prediction, it performs the following steps:

[0129] Image frame sequence data of the target aircraft is acquired based on an airborne monocular camera. The image frame sequence data is analyzed based on a small target detection algorithm to determine the center position coordinates of the target aircraft and construct a center position coordinate sequence.

[0130] Obtain the image frame transmission delay information of the monocular camera; when there is a transmission delay, extract the high-frequency IMU data at the current moment.

[0131] The high-frequency IMU data is input into the LSTM-DKF compensation prediction model constructed based on the fusion of LSTM and delayed Kalman filter to generate the process noise covariance matrix Q and the observation noise covariance matrix R of the delayed Kalman filter.

[0132] The delayed Kalman filter is initialized based on the matrix Q and matrix R, and the position of the target aircraft image center point during the image frame transmission delay is recursively predicted based on the high-frequency IMU data.

[0133] When a delayed image frame is received, the state variables of the Kalman filter are updated back to the delayed time, and the compensated target aircraft position is recalculated back to the current time.

[0134] The corrected pixel difference is calculated based on the compensated target aircraft position, and a tracking control command is generated based on the pixel difference.

[0135] This invention discloses a method and system for aircraft target tracking based on compensation prediction, used to improve target tracking accuracy in scenarios with image transmission delay. The method first acquires a sequence of target aircraft image frames using an airborne monocular camera, and extracts the target center coordinates using a small target detection algorithm to construct a position sequence. For cases with image frame transmission delay, current high-frequency IMU data is extracted and input into an LSTM-DKF model constructed by fusing LSTM and a delayed Kalman filter, predicting and generating process noise and observation noise covariance matrices. The delayed Kalman filter is initialized using these matrices, recursively predicting the target position during the delay period. When a delayed image frame is received, the filter state is backtracked and updated, and the prediction is repeated to the current time, outputting the compensated target position. Finally, pixel deviations are calculated based on the compensated position to generate aircraft tracking control commands, achieving high-precision target tracking.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0137] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0138] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0139] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for tracking aircraft targets based on compensation prediction, characterized in that, Includes the following steps: Image frame sequence data of the target aircraft is acquired based on an airborne monocular camera. The image frame sequence data is analyzed based on a small target detection algorithm to determine the center position coordinates of the target aircraft and construct a center position coordinate sequence. Obtain the image frame transmission delay information of the monocular camera; when there is a transmission delay, extract the high-frequency IMU data at the current moment. The high-frequency IMU data is input into the LSTM-DKF compensation prediction model constructed based on the fusion of LSTM and delayed Kalman filter to generate the process noise covariance matrix Q and the observation noise covariance matrix R of the delayed Kalman filter, specifically: An LSTM-DKF compensation prediction model is constructed based on LSTM and a delayed Kalman filter. The time-series high-frequency IMU data is input into the LSTM network of the compensation prediction model, and the hidden state h is calculated through gating mechanisms including forget gates, input gates, and output gates. t The Tanh activation function is used to generate a 128-dimensional hidden layer output. From the hidden state h t Separate the two fully connected branches; The first branch outputs a 3D vector, which is reconstructed into the Cholesky lower triangular matrix L of the process noise covariance matrix Q. Q Satisfying Q=L Q L Q T and for L Q Apply an exponential activation function to the diagonal elements; The second branch outputs a 3D vector, which is reconstructed into the Cholesky lower triangular matrix L of the observation noise covariance matrix R. R Satisfying R=L R L R T and for L R Apply an exponential activation function to the diagonal elements; L based on reconstruction Q and L R Calculate the final process noise covariance matrix Q and observation noise covariance matrix R; The delayed Kalman filter is initialized based on the matrix Q and matrix R, and the position of the target aircraft image center point during the image frame transmission delay is recursively predicted based on the high-frequency IMU data. When a delayed image frame is received, the state variables of the Kalman filter are updated back to the delayed time, and the compensated target aircraft position is recalculated back to the current time. The corrected pixel difference is calculated based on the compensated target aircraft position, and a tracking control command is generated based on the pixel difference.

2. The aircraft target tracking method based on compensation prediction according to claim 1, characterized in that, The process involves acquiring image frame sequence data of the target aircraft using an airborne monocular camera, analyzing the image frame sequence data based on a small target detection algorithm to determine the center position coordinates of the target aircraft, and constructing a center position coordinate sequence. Specifically: Image frame sequence data of the target aircraft are acquired by the airborne monocular camera of the tracking aircraft, and a small target detection algorithm is constructed based on global detectors and local detectors. The image frame sequence data is imported into a global detector for YOLOv5 target detection to determine the region where the target aircraft is located in the image frame sequence data and to determine the recognition accuracy. When the recognition accuracy is less than a preset value, moving target detection is performed on the image frame sequence data based on sparse optical flow motion compensation and inter-frame difference method to determine the moving target in the image frame sequence data. The moving target is then identified by the aircraft classifier, and the region where the target aircraft with high confidence is located in the image frame sequence data is output. The region where the target aircraft is located in the image frame sequence data is input into the local detector. Based on the LightTrack target tracker, the target aircraft is identified and selected in the region where the target aircraft is located. The center position coordinates of the target aircraft in the image coordinate system in each image frame are determined, and the center position coordinates are arranged in the time order of the image frames to construct a center position coordinate sequence.

3. The aircraft target tracking method based on compensation prediction according to claim 1, characterized in that, The step of obtaining the image frame transmission delay information from the monocular camera, and extracting the high-frequency IMU data at the current moment when a transmission delay exists, specifically involves: The frame generation timestamp data of the image frame sequence data acquired by the monocular camera and the current system timestamp data of the image frame arriving at the onboard processing system of the tracking aircraft are obtained, and the difference between the frame generation timestamp and the current system timestamp is calculated. The difference is compared with a preset frame period threshold. When the difference is greater than the frame period threshold, it is determined that there is an image frame transmission delay and the delay duration is recorded to obtain image frame transmission delay information. If there is a transmission delay, the high-frequency IMU data of the tracking aircraft at the current moment is obtained. The IMU data includes angular velocity data, acceleration data, and attitude quaternion data.

4. The aircraft target tracking method based on compensation prediction according to claim 1, characterized in that, The process of initializing the delayed Kalman filter based on the matrix Q and matrix R, and recursively predicting the target aircraft image center point position during the image frame transmission delay based on the high-frequency IMU data, specifically involves: A state vector is constructed based on the high-frequency IMU data and the center position coordinate sequence. The state vector includes the attitude quaternion of the tracking aircraft, angular velocity, acceleration, relative position and relative velocity between the tracking aircraft and the target aircraft, and the center position coordinates of the target aircraft in the image coordinate system from the perspective of the tracking aircraft. Construct a state transition matrix F based on the state vector. k The state transition matrix describes the continuous-time evolution relationship of attitude quaternions, relative position, and relative velocity; Construct the control input vector u based on high-frequency IMU data. k The control input vector includes angular velocity, acceleration, and attitude information. A control input matrix B is constructed based on the control input vector. k The control input matrix maps the angular velocity and acceleration in the high-frequency IMU data to the state space; An observation matrix H is constructed based on the geometric projection relationship between the image coordinate system and the collective coordinate system of the tracking aircraft. The observation matrix establishes a mapping relationship between the state vector and the coordinates of the center position of the target aircraft image. Based on the state transition matrix and control input matrix, the predicted state vector is calculated using the state prediction equation. k∣k−1 The specific expression is: k∣k−1 =F k k−1 +B k u k−1， Update the prediction error covariance matrix P based on the matrix Q. k∣k−1 The specific expression is: P k∣k−1 =F k P k−1 F k T +Q, Where k represents the current time step, P k−1 The final error covariance matrix at the previous time step is denoted by , and is the initial preset value. k∣k−1 This represents time intervals from k-1 to k; When there is an image frame transmission delay, the Kalman gain K is calculated based on the observation noise covariance matrix R and the observation matrix H. k The specific expression is: K k =P k∣k−1 H T (HP k∣k−1 H T +R) −1 , The center position coordinates z of the delayed image frame are fused using Kalman gain. k And update the final error covariance matrix at the current time, with the specific formula as follows: k = k∣k−1 +K k (z k −H k∣k−1 ), P k =(I−K k H)P k∣k−1 , Where z k That is, the center coordinates of the target aircraft in the latest image frame arrived by the onboard system tracking the aircraft, where I is the coordinate of P. k∣k−1 Identity matrices of the same dimension; From the updated state estimation vector k Extract the center coordinates of the target aircraft in the image coordinate system to obtain the center point position of the target aircraft image during the image frame transmission delay.

5. The aircraft target tracking method based on compensation prediction according to claim 4, characterized in that, When a delayed image frame is received, the process involves backtracking to the delayed time to update the Kalman filter state, and then recursively outputting the compensated target aircraft position at the current time. Specifically: When a delayed image frame is received, the calculated final state estimation vector is stored. k And error covariance matrix P k ; When entering the next time step k+1, the stored k As a new state input (k+1)−1 , store P k As the new error covariance input P (k+1)−1 The target spacecraft image center point position during the image frame transmission delay in the next time step k+1 is recursively calculated to obtain the latest compensated target spacecraft position at the current time.

6. The aircraft target tracking method based on compensation prediction according to claim 1, characterized in that, The step of calculating the corrected pixel difference based on the compensated target aircraft position and generating tracking control commands based on the pixel difference specifically includes: Based on the compensated target aircraft position, the horizontal pixel difference e between the target center point and the image center point is obtained. x The pixel difference between the vertical axis and the vertical axis e y ; According to the pixel difference e x and e y Constructing the attitude control matrix and lift control amount The attitude control matrix and lift control amount The calculation formula is: ， ， in To track the roll angle of the aircraft, The pitch angle, The pitch angle offset is the preset value, and m is the mass of the aircraft. The velocity is in the vertical direction. It is the acceleration due to gravity. to To control the gain coefficient; According to the attitude control matrix Generate angular velocity commands for the roll, pitch, and yaw channels, and simultaneously adjust them according to the lift control parameters. Generate total lift commands, combine them to form the tracking control commands, and output them to the flight control system of the tracking aircraft.

7. An aircraft target tracking system based on compensation prediction, characterized in that, The aircraft target tracking system based on compensation prediction includes a storage unit and a processor. The storage unit includes a program for an aircraft target tracking method based on compensation prediction. When the processor executes the program for an aircraft target tracking method based on compensation prediction, it performs the following steps: Image frame sequence data of the target aircraft is acquired based on an airborne monocular camera. The image frame sequence data is analyzed based on a small target detection algorithm to determine the center position coordinates of the target aircraft and construct a center position coordinate sequence. Obtain the image frame transmission delay information of the monocular camera; when there is a transmission delay, extract the high-frequency IMU data at the current moment. The high-frequency IMU data is input into the LSTM-DKF compensation prediction model constructed based on the fusion of LSTM and delayed Kalman filter to generate the process noise covariance matrix Q and the observation noise covariance matrix R of the delayed Kalman filter, specifically: An LSTM-DKF compensation prediction model is constructed based on LSTM and a delayed Kalman filter. The time-series high-frequency IMU data is input into the LSTM network of the compensation prediction model, and the hidden state h is calculated through gating mechanisms including forget gates, input gates, and output gates. t The Tanh activation function is used to generate a 128-dimensional hidden layer output. From the hidden state h t Separate the two fully connected branches; The first branch outputs a 3D vector, which is reconstructed into the Cholesky lower triangular matrix L of the process noise covariance matrix Q. Q Satisfying Q=L Q L Q T and for L Q Apply an exponential activation function to the diagonal elements; The second branch outputs a 3D vector, which is reconstructed into the Cholesky lower triangular matrix L of the observation noise covariance matrix R. R Satisfying R=L R L R T and for L R Apply an exponential activation function to the diagonal elements; L based on reconstruction Q and L R Calculate the final process noise covariance matrix Q and observation noise covariance matrix R; The delayed Kalman filter is initialized based on the matrix Q and matrix R, and the position of the target aircraft image center point during the image frame transmission delay is recursively predicted based on the high-frequency IMU data. When a delayed image frame is received, the state variables of the Kalman filter are updated back to the delayed time, and the compensated target aircraft position is recalculated back to the current time. The corrected pixel difference is calculated based on the compensated target aircraft position, and a tracking control command is generated based on the pixel difference.

Images