An interactive multi-model infrared time-sensitive target tracking method based on image feature fusion

Abstract

The present invention addresses the problem of difficulty in determining the motion state of ground targets due to the rigid motion of airborne cameras affected by air currents and the strong maneuverability of ground targets in infrared target tracking in air-to-ground scenarios. An interactive multi-model infrared time-sensitive target tracking method that integrates image features is invented. Within the framework of Kalman filtering, the present invention uses target detection results as state observers, introduces inter-frame registration to compensate for errors caused by coordinate system transformation due to the shaking of the airborne platform, uses interactive multi-models to characterize the strong maneuverability of ground moving targets, and finally accurately tracks the moving targets through data association. The tracking method proposed by the present invention has low complexity, can well adapt to the complex environment of air-to-ground imaging, and the tracking accuracy is not affected by the accuracy of the target detection results, which has important engineering significance.

Description

Technical Field

[0001] The present invention belongs to the field of image analysis, and in particular relates to an interactive multi-model infrared time-sensitive target tracking method that fuses image features, which is suitable for infrared target tracking tasks in the background of air-to-ground imaging. Background Art

[0002] With the rapid development of drone remote sensing technology, the use of drones carrying various payloads for reconnaissance and strike missions can significantly impact battlefield dynamics. By analyzing and processing drone-collected video, time-sensitive targets can be intelligently detected and tracked, enabling efficient perception, monitoring, and analysis of battlefield conditions. Given the complex and intense nature of modern warfare and the diverse nature of interference, infrared detection technology, with its advantages of concealment and strong anti-interference capabilities, plays a vital role and is widely used in early warning, reconnaissance, tracking, and guidance.

[0003] As a passive detection method, infrared imaging offers the advantages of concealment, anti-interference, and 24 / 7 operation. However, due to the need to detect targets as far away as possible in complex battlefield environments, the target imaging area in infrared detection systems is small and lacks important information such as texture and structure. Furthermore, infrared imaging is susceptible to environmental clutter and noise, making the target difficult to detect due to the cacophony and noise. Numerous methods have been studied for infrared target detection. While existing technologies can obtain the target's position coordinates, they suffer from certain errors.

[0004] When it comes to target tracking in air-to-ground imaging, the airborne platform is subject to irregular motion due to airflow. This motion causes the infrared camera's field of view to rotate, tilting the imaged background (roads, trees, buildings, etc.) and the target on the ground. This shifts the coordinate systems corresponding to the two images, affecting the accurate representation of the target's position. This error caused by the imaging payload's motion further increases the error in target detection, impacting the accurate tracking of time-sensitive air-to-ground targets.

[0005] At the same time, when imaging ground targets from an airborne platform, the motion model of the ground target is difficult to describe well using a single model due to the inherent mobility of the ground target and the relative motion between the UAV platform and the ground target. Using a traditional single model to model the target's motion has significant limitations, significantly affecting target tracking accuracy.

[0006] To address existing issues, this paper proposes an interactive multi-model infrared time-sensitive target tracking method that integrates image features. This method is suitable for scenarios where airborne infrared platforms are detecting and tracking maneuvering ground targets. Within the framework of Kalman filtering, this method uses a target detection algorithm as the observation model and an interactive multi-model as the prediction model. Image features are extracted for inter-frame registration. The affine transformation matrix derived from image registration is used to correct the coordinates of the detected target to compensate for errors caused by coordinate system transformations due to airborne platform motion. Finally, data association is performed using the Hungarian algorithm, enabling accurate tracking of ground infrared targets. Summary of the Invention

[0007] The purpose of the present invention is to solve the many problems that restrict target tracking accuracy in the above-mentioned air-to-ground imaging scenarios. An interactive multi-model infrared time-sensitive target tracking method that integrates image features is proposed. The method can accurately track targets detected on the ground in air-to-ground imaging scenarios with complex backgrounds. It can well overcome the significant impact of the shaking of the airborne platform and the strong maneuverability of the tracked target on the tracking accuracy.

[0008] The present invention provides an interactive multi-model infrared time-sensitive target tracking method integrating image features, which specifically comprises the following steps:

[0009] (1) Create a state observation model: When the tracking camera of the airborne platform captures a moving target on the ground, the target detection algorithm is used to obtain the pixel position of the moving target in the image, and the target detection result is used as the state observation model of the tracking system;

[0010] (2) Creating the state quantity of the target in the image, including the following steps:

[0011] (2.1) Input images: including the k-1th frame image and the kth frame image, where k = 2, 3, ..., n is the sequence number;

[0012] (2.2) Establish the target state: Perform target detection on the k-1th frame image and the kth frame image, and obtain the target frame set S of the k-1th frame image and the kth frame image respectively. k-1 and S k , establish the target state vector as

[0013]

[0014] Among them, x k and y k They refer to the horizontal and vertical position coordinates of the target k at the moment along the image, and are the pixel movement speeds along the horizontal and vertical directions of the image at time k, and are the accelerations of the horizontal and vertical pixel motions along the image at time k, respectively. Since the image sequence between frames is discretized, the velocity is calculated by the difference of the coordinates, and the acceleration is calculated by the difference of the velocity. The specific calculation formula is:

[0015]

[0016] Among them, x k-1 and y k-1 are the horizontal and vertical position coordinates of the target at time k-1 along the image, and are the pixel motion speeds along the horizontal and vertical directions of the image at time k-1, respectively. The construction of the initial state quantity requires the use of the first and second frames of image. Starting from the third frame, the calculation of the state quantity is performed using the current frame and the previous frame.

[0017] (3) constructing an interactive multi-model Kalman filter model and estimating the target state online, and predicting the target state in the next frame of the image based on the target state established in step (2);

[0018] (3.1) Constructing an interactive multi-model Kalman filter model set

[0019] The possible motion modes of the target are mapped into a Kalman filter model set. Each model in the model set represents a different motion mode. Assuming that the interactive multi-model filter contains r different Kalman filter models, the general formula for determining the model is:

[0020] X j (k+1)=F j (k)X j (k)+G j (k)W j (k) (1)

[0021] Z(k)=H j (k)X j (k)+V j (k) (2)

[0022] Wherein, formula (1) represents the state equation of the j-th motion model, formula (2) represents the measurement equation of the j-th motion model, X j (k) is the state vector at time k, F j (k) is the state transfer matrix at time k, G j (k) is the noise driving matrix at time k, W j (k) is the system noise at time k, V j (k) is the observation noise at time k, H j(k) is the observation matrix at time k, Z(k) is the observation quantity at time k; j = 1, 2..., r is the model number, and W j The system noise covariance matrix of (k) is expressed as Q j (k), V j The observation noise covariance matrix of (k) is expressed as R j (k);

[0023] According to the actual motion situation of the ground motion target, the motion mode of the ground motion target is set to target uniform linear motion and target uniform accelerated linear motion, and the state equations under the two motion modes are further determined;

[0024] The state equation of the target uniform linear motion is:

[0025]

[0026] The state equation of the target uniformly accelerated linear motion is:

[0027]

[0028] Where, x(k) and y(k) are the horizontal and vertical coordinates of the target in the field of view at time k, respectively. are the target lateral velocity and longitudinal velocity at time k, are the lateral acceleration and longitudinal acceleration of the target at time k; x(k+1) and y(k+1) are the horizontal and vertical coordinates of the target in the field of view at time k+1, respectively. are the target lateral velocity and longitudinal velocity at time k+1 respectively, are the target lateral acceleration and longitudinal acceleration at time k+1 respectively; T is the sampling time interval, ω j (k) is the model noise, which is modeled as white noise;

[0029] The model is designed with n1 uniform linear motion equations and n2 uniform accelerated linear motion equations, satisfying n1+n2=r. Each motion equation corresponds to a different Q j and R j ;

[0030] (3.2) The uniform linear motion and uniformly accelerated linear motion models of the target are used to form a maneuvering target filter model. There are r Kalman filter models in total. According to the target state quantity constructed in step (2), the motion model parameters Q1, Q2, Q3...Q r 、R1,R2,R3...R r And combine the weight factors corresponding to each model to perform input interaction to obtain the estimated value of each target state quantity;

[0031] (4) Calculating the affine transformation matrix of the inter-frame images through inter-frame registration and correcting the estimated value of the target state quantity in step (3) includes the following steps:

[0032] (4.1) Extract key points from the previous and current frames based on their image features, and place the generated descriptors into two sets T1 and T2. Calculate the Euclidean distance between the descriptors in the two sets to characterize their similarity, perform key point matching, and after key point extraction, calculate the affine transformation matrix between the two images based on their corresponding relationship.

[0033] (4.2) transforming the predicted bounding box from the coordinate system of the previous frame image to the coordinate system of the current frame image according to the affine transformation matrix to compensate for the effect of camera motion on the predicted position of the target;

[0034] (4.3) The estimated value of the state quantity in step (3) is corrected by the affine transformation matrix obtained by inter-frame registration, and the estimated value is transformed into the coordinate system corresponding to the current image as the new state estimate;

[0035] (5) Perform state observation on the current frame image, i.e., perform target detection to obtain a state observation quantity; perform data association on the obtained state observation quantity and the new state estimation quantity obtained in step (4) to make a preliminary match between the state observation quantity and the state estimation quantity; and divide the targets into matched targets and unmatched targets according to the matching results; and divide the unmatched targets into unmatched prediction frames and unmatched detection frames, and then perform the following classification operations on different combinations of targets according to different situations:

[0036] (5.1) For an unmatched prediction box, if it is not at the edge of the image, it is considered to be an unoccluded target and the tracking box is still retained, but its observation noise R j The whole image is magnified 3 times. If it is at the edge of the image, it is considered that the target has moved out of the field of view, and the prediction box is directly deleted and no longer considered in the subsequent tracking process.

[0037] (5.2) For the unmatched detection box, it is considered as the target, a new ID is assigned to it, and a new Kalman filter is created for it to participate in the subsequent tracking process;

[0038] (5.3) For the correctly matched target, continue to retain its ID and continue tracking;

[0039] (6) Using state estimation and state observation to fuse information and update various parameters in the tracking system, specifically including the following steps:

[0040] (6.1) Hybrid model correction, Kalman filtering is performed on the model, and the input quantities are the initial state of the model at time k-1, X 0j(k-1|k-1), the covariance P of the model at time k-1 0j (k-1|k-1), the observed state Z(k) at time k; determine the predicted state at time k and the predicted covariance for

[0041]

[0042] Among them, F j (k|k-1) is the state transition matrix of the j-th motion model from time k-1 to time k, G j (k|k-1) is the noise driving matrix of the j-th motion model from time k-1 to time k, W j (k-1) is the system noise at time k-1, and W j The (k-1) system noise covariance matrix is ​​expressed as Q j (k-1);

[0043] The model transformation matrix is ​​expressed as

[0044]

[0045] Among them, p ij represents the model transition probability from the i-th motion model to the j-th motion model, 1≤i≤r,1≤j≤r;

[0046] The model transition probability μ at time k-1 ij (k-1|k-1) is expressed as

[0047]

[0048] r is the total number of models in the model set, μ i (k-1) is the probability corresponding to the i-th model at time k-1;

[0049] Determine the covariance matrix of measurement error and calculate S j (k) is:

[0050]

[0051] Among them, R j (k) is V in the observation equation j (k) The observation noise covariance matrix;

[0052] Determine the Kalman gain K j (k) is:

[0053]

[0054] Determine the state quantity at time k after filtering and covariance for:

[0055]

[0056]

[0057] (6.2) Update the probability of each model; the update of the model will directly affect the accuracy of the algorithm. The maximum likelihood function method is used to update the model. The model weight is determined by calculating the similarity between the current model and the current motion state; the maximum likelihood function Λ that best matches model j at time k is j (k)

[0058]

[0059] in,

[0060] Then we get the updated probability μ of model j j (k)

[0061]

[0062] Where c is the normalization constant, that is,

[0063]

[0064] (6.3) Output interaction: Output the result of information fusion and update the probability of each model; determine the state quantity after the model output fusion at time k And the covariance P(k|k) after fusion is

[0065]

[0066] The target state after information fusion is used as the target tracking result of the current frame image;

[0067] (7) When the camera captures the next frame of image, the above steps (2) to (6) are repeated to continuously obtain the target's motion position and realize air-ground time-sensitive infrared target tracking.

[0068] Preferably, the target detection algorithm in step (1) uses an infrared target detection algorithm based on block matching to perform target detection on the image to obtain the target position.

[0069] Preferably, the target detection algorithm in step (1) uses an infrared target detection algorithm based on deep learning to detect infrared targets in the image and obtain the target position.

[0070] Preferably, in the step (3.1), a variable acceleration motion model, or a uniform speed turning model, or a Singer model, or a combination of the variable acceleration motion model, the uniform speed turning model, and the Singer model is added to construct an interactive multi-model Kalman filter model set, and used as a state prediction model of the Kalman filter.

[0071] Preferably, the step (4.1) is performed by extracting SIFT features of the image for registration, and the specific operation steps are as follows:

[0072] a. Construct scale space; construct Gaussian difference scale space based on Gaussian kernel function filtering. The Gaussian kernel function G(x, y, σ) used for the image is

[0073]

[0074] Among them, x, y represent the pixel coordinates in the image, σ represents the size of the scale, and a Gaussian kernel function with different σ is selected to perform a convolution operation with the original image to obtain the value L(x, y, σ) of the image at different scales:

[0075] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0076] Among them, "*" is the convolution operation, I(x,y) is the pixel value at (x,y); after obtaining L(x,y,σ) at different scales, the L(x,y,σ) of adjacent scales is differentiated to obtain the information value at the original scale, and then the DOG scale space is constructed. The specific calculation method of the operation is:

[0077] D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)=L(x,y,kσ)-L(x,y,σ)

[0078] Among them, k is an integer, and the information at different scales can be obtained by adjusting the k value;

[0079] b. Extract key points from the image; after constructing the DOG scale space in step a, determine whether each point is a key point; select 8 points in the same scale space and 9 points in adjacent scales for calculation. Since there may be extreme points on the edge, they need to be eliminated. First, calculate the Hessian matrix H at the current pixel:

[0080]

[0081] Among them, D xx , D yy Indicates that the image of a certain scale in the DOG pyramid is differentiated twice in the x-direction and the y-direction, and the derivative is estimated by the neighbor difference of adjacent sampling points; D xyThe image of a certain scale in the DOG pyramid is first differentiated in the x direction and then in the y direction. The derivative is also estimated by the neighbor difference of adjacent sampling points. Let α and β be the two eigenvalues ​​of the matrix H, then we get:

[0082] tr(H)=D xx +D yy =α+β

[0083] det(H)=D xx +D yy -D xy 2 =α·β

[0084]

[0085] When α and β are equal, tr(H) 2 / det(H) takes the minimum; tr(H) 2 / det(H) is used as a criterion for judging edge points. When its value is less than 1.2, it is considered a key point and the position information of the key point is obtained;

[0086] c. Determine the direction of the key points; for the key points selected in step b, use the neighborhood gradient and direction to represent their directional characteristics. The specific calculation formula is:

[0087]

[0088] Among them, m(x,y) is the neighborhood gradient of the key point, and θ(x,y) is the direction of the key point. After calculating the neighborhood gradient at the key point, the gradient direction and amplitude corresponding to the pixels in the neighborhood of the feature point are counted using the histogram. The horizontal axis of the gradient direction histogram is the angle of the gradient direction.

[0089] d. Determine the descriptor of the key point; after obtaining the position and direction of the key point through steps ac, rotate the original coordinate axis according to the direction of the key point, calculate the gradient magnitude and direction of the pixels in a 16x16 window centered on the feature point, and divide the pixels in this area into 16 blocks, each of which is a histogram statistics of 8 directions within its pixels, forming a 128-dimensional feature vector;

[0090] e. Perform steps a through d on both the previous and current frames to obtain descriptor sets T1 and T2 of the key points of the two images, respectively. Use Euclidean distance to characterize the similarity of the key points in the two images, perform correlation matching based on the similarity, and solve the affine transformation matrix of the two images based on the corresponding points.

[0091] Preferably, the step (4.1) performs registration by extracting SURF features, ORB features, or AKAZE feature image key points of the image.

[0092] The advantages and positive effects of the present invention are as follows: Starting from the difficulties of the air-to-ground time-sensitive infrared target tracking task, the present invention introduces an interactive multi-model filtering algorithm to characterize the strong maneuverability of the moving target, which can effectively solve the problem that the high-speed maneuvering of the moving target causes the tracking algorithm to converge slowly and is easy to lose. By introducing an inter-frame registration algorithm to compensate for the errors caused by the transformation of the coordinate system in the field of view due to the rigid movement of the airborne platform, there is no need to use other sensors, the algorithm is simple and has great practical value. In summary, the present invention has stronger robustness and anti-interference capabilities than the original tracking method. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0094] Figure 1 It is a flow chart of the IMM filtering algorithm of the present invention;

[0095] Figure 2 This is a flow chart of the algorithm for inter-frame registration in step (4) of the present invention;

[0096] Figure 3 It is the overall algorithm flow chart of the tracking method of the present invention. DETAILED DESCRIPTION

[0097] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0098] Mounting various imaging payloads on unmanned aerial vehicle platforms enables effective detection of ground targets. This airborne detection method, combined with the stealth, anti-interference, and high-precision advantages of infrared imaging payloads, can play a crucial role in air-to-ground time-sensitive target detection missions. At the same time, there are many factors that hinder accurate target tracking in air-to-ground time-sensitive target imaging scenarios. For example, the complex background of the ground imaging environment results in low infrared target detection accuracy; the rigid motion of the airborne camera causes the coordinate system in the imaging field of view to constantly change, making the coordinates of the detected target inaccurate; and ground targets are mostly maneuverable, and the rapid motion of the airborne platform makes it difficult to establish a target motion model. These factors make it difficult to effectively detect and track ground targets using traditional methods.

[0099] The present invention proposes an interactive multi-model infrared time-sensitive target tracking method that integrates image features. Starting from the characteristics of air-to-ground infrared target imaging, based on the traditional Kalman filter target tracking algorithm, it introduces inter-frame registration to compensate for the errors caused by the rigid motion of the airborne platform resulting from the coordinate system transformation in the field of view, introduces an interactive multi-model filtering algorithm to characterize the strong maneuverability of ground moving targets, and makes real-time adjustments to the data association of the tracking target based on the correspondence between the predicted state and the observed state, and continuously iterates and continues the tracking process. The specific flow chart of the method is shown in the figure. Figure 3 The specific implementation method is as follows:

[0100] (1) Create a state observation model: When the tracking camera of the airborne platform captures a moving target on the ground, the target detection algorithm is used to obtain the pixel position of the moving target in the image, and the target detection result is used as the state observation model of the tracking system;

[0101] Preferably, the target detection algorithm uses an infrared target detection algorithm based on block matching to perform target detection on the image to obtain the target position;

[0102] Preferably, the target detection algorithm or the infrared target detection algorithm based on deep learning is used to detect the infrared target in the image to obtain the target position;

[0103] (2) Creating the state quantity of the target in the image, including the following steps:

[0104] (2.1) Input images: including the k-1th frame image and the kth frame image, where k = 2, 3, ..., n is the sequence number;

[0105] (2.2) Establish the target state: Perform target detection on the k-1th frame image and the kth frame image, and obtain the target frame set S of the k-1th frame image and the kth frame image respectively. k-1 and S k , establish the target state vector as

[0106]

[0107] Among them, x k and y k They refer to the horizontal and vertical position coordinates of the target k at the moment along the image, and are the pixel movement speeds along the horizontal and vertical directions of the image at time k, and are the accelerations of the horizontal and vertical pixel motions along the image at time k, respectively. Since the image sequence between frames is discretized, the velocity is calculated by the difference of the coordinates, and the acceleration is calculated by the difference of the velocity. The specific calculation formula is:

[0108]

[0109] Among them, x k-1 and y k-1 are the horizontal and vertical position coordinates of the target at time k-1 along the image, and are the pixel motion speeds along the horizontal and vertical directions of the image at time k-1, respectively. The construction of the initial state quantity requires the use of the first and second frames of image. Starting from the third frame, the calculation of the state quantity is performed using the current frame and the previous frame.

[0110] (3) Construct an interactive multi-model Kalman filter model and estimate the target state online. According to the target state established in step (2), predict the target state in the next frame image. The specific interactive multi-model algorithm flow chart is as follows: Figure 1 As shown;

[0111] (3.1) Constructing an interactive multi-model Kalman filter model set

[0112] The possible motion modes of the target are mapped into a Kalman filter model set. Each model in the model set represents a different motion mode. Assuming that the interactive multi-model filter contains r different Kalman filter models, the general formula for determining the model is:

[0113] X j (k+1)=F j (k)X j (k)+G j (k)W j (k) (1)

[0114] Z(k)=H j (k)X j (k)+V j (k) (2)

[0115] Wherein, formula (1) represents the state equation of the j-th motion model, formula (2) represents the measurement equation of the j-th motion model, X j (k) is the state vector at time k, F j (k) is the state transfer matrix at time k, G j (k) is the noise driving matrix at time k, W j (k) is the system noise at time k, V j (k) is the observation noise at time k, H j (k) is the observation matrix at time k, Z(k) is the observation quantity at time k; j = 1, 2..., r is the model number, and W j The system noise covariance matrix of (k) is expressed as Q j, V j The observation noise covariance matrix of (k) is expressed as R j ;

[0116] According to the actual motion conditions of the ground moving target, the motion mode of the ground moving target is set to target uniform linear motion and target uniform accelerated linear motion, and the state equations under the two motion modes are further determined; as an improvement, the variable acceleration motion model, or the uniform speed turning model, or the Singer model, or the combination of the variable acceleration motion model, the uniform speed turning model, and the Singer model is added to construct an interactive multi-model Kalman filter model set, and used as the state prediction model of the Kalman filter.

[0117] The state equation of the target uniform linear motion is:

[0118]

[0119] The state equation of the target uniformly accelerated linear motion is:

[0120]

[0121] Where, x(k) and y(k) are the horizontal and vertical coordinates of the target in the field of view at time k, respectively. are the target lateral velocity and longitudinal velocity at time k, are the lateral acceleration and longitudinal acceleration of the target at time k; x(k+1) and y(k+1) are the horizontal and vertical coordinates of the target in the field of view at time k+1, respectively. are the target lateral velocity and longitudinal velocity at time k+1, are the target lateral acceleration and longitudinal acceleration at time k+1 respectively; T is the sampling time interval, ω j (k) is the model noise, which is modeled as white noise;

[0122] The model is designed with n1 uniform linear motion equations and n2 uniform accelerated linear motion equations, satisfying n1+n2=r. Each motion equation corresponds to a different Q j and R j ;

[0123] (3.2) The uniform linear motion and uniformly accelerated linear motion models of the target are used to form a maneuvering target filter model. There are r Kalman filter models in total. According to the target state quantity constructed in step (2), the motion model parameters Q1, Q2, Q3...Q r 、R1,R2,R3...R r And combine the weight factors corresponding to each model to perform input interaction to obtain the estimated value of each target state quantity;

[0124] (4) Calculate the affine transformation matrix of the inter-frame image through inter-frame registration, and correct the estimated value of the target state quantity in step (3). The registration algorithm flow chart is as follows: Figure 2 As shown, the specific steps include:

[0125] (4.1) Extract key points from the previous and current frames based on their image features, and place the generated descriptors into two sets T1 and T2. Calculate the Euclidean distance between the descriptors in the two sets to characterize their similarity, perform key point matching, and after key point extraction, calculate the affine transformation matrix between the two images based on their corresponding relationship.

[0126] Preferably, the step (4.1) is performed by extracting SIFT features of the image for registration, and the specific operation steps are as follows:

[0127] a. Construct scale space; construct Gaussian difference scale space based on Gaussian kernel function filtering. The Gaussian kernel function G(x, y, σ) used for the image is

[0128]

[0129] Among them, x, y represent the pixel coordinates in the image, σ represents the size of the scale, and a Gaussian kernel function with different σ is selected to perform a convolution operation with the original image to obtain the value L(x, y, σ) of the image at different scales:

[0130] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0131] Among them, "*" is the convolution operation, I(x,y) is the pixel value at (x,y); after obtaining L(x,y,σ) at different scales, the L(x,y,σ) of adjacent scales is differentiated to obtain the information value at the original scale, and then the DOG scale space is constructed. The specific calculation method of the operation is:

[0132] D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)=L(x,y,kσ)-L(x,y,σ)

[0133] Among them, k is an integer, and the information at different scales can be obtained by adjusting the k value;

[0134] b. Extract key points from the image; after constructing the DOG scale space in step a, determine whether each point is a key point; select 8 points in the same scale space and 9 points in adjacent scales for calculation. Since there may be extreme points on the edge, they need to be eliminated. First, calculate the Hessian matrix H at the current pixel:

[0135]

[0136] Among them, D xx , D yy Indicates that the image of a certain scale in the DOG pyramid is differentiated twice in the x-direction and the y-direction, and the derivative is estimated by the neighbor difference of adjacent sampling points; D xy The image of a certain scale in the DOG pyramid is first differentiated in the x direction and then in the y direction. The derivative is also estimated by the neighbor difference of adjacent sampling points. Let α and β be the two eigenvalues ​​of the matrix H, then we get:

[0137] tr(H)=D xx +D yy =α+β

[0138] det(H)=D xx +D yy -D xy 2 =α·β

[0139]

[0140] When α and β are equal, tr(H) 2 / det(H) takes the minimum; tr(H) 2 / det(H) is used as a criterion for judging edge points. When its value is less than 1.2, it is considered a key point and the position information of the key point is obtained;

[0141] c. Determine the direction of the key points; for the key points selected in step b, use the neighborhood gradient and direction to represent their directional characteristics. The specific calculation formula is:

[0142]

[0143] Among them, m(x,y) is the neighborhood gradient of the key point, and θ(x,y) is the direction of the key point. After calculating the neighborhood gradient at the key point, the gradient direction and amplitude corresponding to the pixels in the neighborhood of the feature point are counted using the histogram. The horizontal axis of the gradient direction histogram is the angle of the gradient direction.

[0144] d. Determine the descriptor of the key point; after obtaining the position and direction of the key point through steps ac, rotate the original coordinate axis according to the direction of the key point, calculate the gradient magnitude and direction of the pixels in a 16x16 window centered on the feature point, and divide the pixels in this area into 16 blocks, each of which is a histogram statistics of 8 directions within its pixels, forming a 128-dimensional feature vector;

[0145] e. Perform steps a through d on both the previous and current frames to obtain descriptor sets T1 and T2 for the key points of the two images, respectively. Use Euclidean distance to characterize the similarity of the key points in the two images, perform correlation matching based on the similarity, and solve the affine transformation matrix of the two images based on the corresponding points.

[0146] (4.2) transforming the predicted bounding box from the coordinate system of the previous frame image to the coordinate system of the current frame image according to the affine transformation matrix to compensate for the effect of camera motion on the predicted position of the target;

[0147] (4.3) Correct the estimated value of the state quantity in step (3) by the affine transformation matrix obtained through inter-frame registration, and transform the estimated value into the coordinate system corresponding to the current image as the new state estimate;

[0148] (5) Perform state observation on the current frame image, i.e., perform target detection to obtain a state observation quantity; perform data association on the obtained state observation quantity and the new state estimation quantity obtained in step (4) to make a preliminary match between the state observation quantity and the state estimation quantity; and divide the targets into matched targets and unmatched targets according to the matching results; and divide the unmatched targets into unmatched prediction frames and unmatched detection frames, and then perform the following classification operations on different combinations of targets according to different situations:

[0149] (5.1) For an unmatched prediction box, if it is not at the edge of the image, it is considered to be an unoccluded target and the tracking box is still retained, but its observation noise R j The whole image is magnified 3 times. If it is at the edge of the image, it is considered that the target has moved out of the field of view, and the prediction box is directly deleted and no longer considered in the subsequent tracking process.

[0150] (5.2) For the unmatched detection box, it is considered as the target, a new ID is assigned to it, and a new Kalman filter is created for it to participate in the subsequent tracking process;

[0151] (5.3) For the correctly matched target, continue to retain its ID and continue tracking;

[0152] (6) Using state estimation and state observation to fuse information and update various parameters in the tracking system, specifically including the following steps:

[0153] (6.1) Hybrid model correction, Kalman filtering is performed on the model, and the input quantities are the initial state of the model at time k-1, X 0j (k-1|k-1), the covariance P of the model at time k-1 0j (k-1|k-1), the observed state Z(k) at time k; determine the predicted state at time k and the predicted covariance for

[0154]

[0155] Among them, F j (k|k-1) is the state transition matrix of the j-th motion model from time k-1 to time k, G j (k|k-1) is the noise driving matrix of the j-th motion model from time k-1 to time k, W j (k-1) is the system noise at time k-1, and W j The (k-1) system noise covariance matrix is ​​expressed as Q j (k-1);

[0156] The model transformation matrix is ​​expressed as

[0157]

[0158] Among them, p ij represents the model transition probability from the i-th motion model to the j-th motion model, 1≤i≤r,1≤j≤r;

[0159] The model transition probability μ at time k-1 ij (k-1|k-1) is expressed as

[0160]

[0161] r is the total number of models in the model set, μ i (k-1) is the probability corresponding to the i-th model at time k-1;

[0162] Determine the covariance matrix of measurement error and calculate S j (k) is:

[0163]

[0164] Among them, R j (k) is V in the observation equation j (k) The observation noise covariance matrix;

[0165] Determine the Kalman gain K j (k) is:

[0166]

[0167] Determine the state quantity at time k after filtering and covariance for:

[0168]

[0169] (6.2) Update the probability of each model; the update of the model will directly affect the accuracy of the algorithm. The maximum likelihood function method is used to update the model. The model weight is determined by calculating the similarity between the current model and the current motion state. The maximum likelihood function that best matches model j at time k is

[0170]

[0171] Among them, v j (k)=Z(k)-H j (k)X j (k|k-1)

[0172] Then we can get the update probability μ of model j j (k)

[0173]

[0174] Where c is the normalization constant, that is,

[0175]

[0176] (6.3) Output interaction: Output the result of information fusion and update the probability of each model; determine the state quantity after the model output fusion at time k And the covariance P(k|k) after fusion is

[0177]

[0178] The target state after information fusion is used as the target tracking result of the current frame image;

[0179] (7) When the camera captures the next frame of image, the above steps (2) to (6) are repeated to continuously obtain the target's motion position and realize air-ground time-sensitive infrared target tracking.

[0180] As an improvement, the step (4.1) performs registration by extracting SURF features, ORB features, or AKAZE feature image key points from the image.

[0181] This invention enables continuous tracking of ground maneuvering targets in air-to-ground imaging environments, where infrared target coordinate detection accuracy is insufficient. This method integrates an inter-frame registration algorithm with the traditional multi-target tracking algorithm based on interactive multi-model filtering to compensate for the effects of motion and oscillation on target tracking caused by the airborne imaging platform. It also effectively overcomes the effects of the high maneuverability of ground targets on target tracking, improving the robustness of infrared target tracking in air-to-ground imaging environments.

[0182] The air-to-ground infrared target tracking method proposed in the present invention has no requirements for the motion state of the aerial platform. Target tracking can be achieved whether the airborne platform is in a hovering, stationary state or in motion. The present invention does not involve other sensors and can perform target tracking normally under conditions of GNSS denial and IMU failure. The interactive multi-model model set construction and inter-frame image feature extraction method in the present invention are only preferred solutions for the interactive multi-model infrared target tracking method that integrates image features. Other model set selection schemes and feature extraction schemes also fall within the scope of the present invention.

[0183] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. All technical solutions within the scope of the present invention are within the scope of protection of the present invention. Any improvements and modifications that do not depart from the principles of the present invention are also within the scope of protection of the present invention.

Claims

1. An interactive multi-model infrared time-sensitive target tracking method integrating image features, characterized in that: The steps include: (1) Create a state observation model: When the tracking camera of the airborne platform captures a moving target on the ground, the target detection algorithm is used to obtain the pixel position of the moving target in the image, and the target detection result is used as the state observation model of the tracking system; (2) Creating the state quantity of the target in the image, including the following steps: (2.1) Input images: including the k-1th frame image and the kth frame image, where k = 2, 3, ..., n is the sequence number; (2.2) Establish the target state: Perform target detection on the k-1th frame image and the kth frame image, and obtain the target frame set S of the k-1th frame image and the kth frame image respectively. k-1 and S k , establish the target state vector as Among them, x k and y k They refer to the horizontal and vertical position coordinates of the target k at the moment along the image, and are the pixel movement speeds along the horizontal and vertical directions of the image at time k, and are the accelerations of the horizontal and vertical pixel motions along the image at time k, respectively. Since the image sequence between frames is discretized, the velocity is calculated by the difference of the coordinates, and the acceleration is calculated by the difference of the velocity. The specific calculation formula is: Among them, x k-1 and y k-1 are the horizontal and vertical position coordinates of the target at time k-1 along the image, and are the pixel motion speeds along the horizontal and vertical directions of the image at time k-1, respectively. The construction of the initial state quantity requires the use of the first and second frames of image. Starting from the third frame, the calculation of the state quantity is performed using the current frame image and the previous frame image. (3) constructing an interactive multi-model Kalman filter model and estimating the target state online, and predicting the target state in the next frame of the image based on the target state established in step (2); (3.1) Constructing an interactive multi-model Kalman filter model set The possible motion modes of the target are mapped into a Kalman filter model set. Each model in the model set represents a different motion mode. Assuming that the interactive multi-model filter contains r different Kalman filter models, the general formula for determining the model is: X j (k+1)=F j (k)X j (k)+G j (k)W j (k) (1) Z(k)=H j (k)X j (k)+V j (k) (2) Wherein, formula (1) represents the state equation of the j-th motion model, formula (2) represents the measurement equation of the j-th motion model, X j (k) is the state vector at time k, F j (k) is the state transfer matrix at time k, G j (k) is the noise driving matrix at time k, W j (k) is the system noise at time k, V j (k) is the observation noise at time k, H j (k) is the observation matrix at time k, Z(k) is the observation quantity at time k; j = 1, 2..., r is the model number, and W j The system noise covariance matrix of (k) is expressed as Q j (k), V j The observation noise covariance matrix of (k) is expressed as R j (k); According to the actual motion situation of the ground motion target, the motion mode of the ground motion target is set to target uniform linear motion and target uniform accelerated linear motion, and the state equations under the two motion modes are further determined; The state equation of the target uniform linear motion is: The state equation of the target uniformly accelerated linear motion is: Where, x(k) and y(k) are the horizontal and vertical coordinates of the target in the field of view at time k, respectively. are the target lateral velocity and longitudinal velocity at time k, are the lateral acceleration and longitudinal acceleration of the target at time k; x(k+1) and y(k+1) are the horizontal and vertical coordinates of the target in the field of view at time k+1, respectively. are the target lateral velocity and longitudinal velocity at time k+1 respectively, are the target lateral acceleration and longitudinal acceleration at time k+1 respectively; T is the sampling time interval, ω j (k) is the model noise, which is modeled as white noise; The model is designed with n1 uniform linear motion equations and n2 uniform accelerated linear motion equations, satisfying n1+n2=r. Each motion equation corresponds to a different Q j and R j ; (3.2) The uniform linear motion and uniformly accelerated linear motion models of the target are used to form a maneuvering target filter model. There are r Kalman filter models in total. According to the target state quantity constructed in step (2), the motion model parameters Q1, Q2, Q3...Q r 、R1,R2,R3...R r And combine the weight factors corresponding to each model to perform input interaction to obtain the estimated value of each target state quantity; (4) Calculating the affine transformation matrix of the inter-frame images through inter-frame registration and correcting the estimated value of the target state quantity in step (3) includes the following steps: (4.1) Extract key points from the previous and current frames based on their image features, and place the generated descriptors into two sets T1 and T2. Calculate the Euclidean distance between the descriptors in the two sets to characterize their similarity, perform key point matching, and after key point extraction, calculate the affine transformation matrix between the two images based on their corresponding relationship. (4.2) transforming the predicted bounding box from the coordinate system of the previous frame image to the coordinate system of the current frame image according to the affine transformation matrix to compensate for the effect of camera motion on the predicted position of the target; (4.3) The estimated value of the state quantity in step (3) is corrected by the affine transformation matrix obtained by inter-frame registration, and the estimated value is transformed into the coordinate system corresponding to the current image as the new state estimate; (5) Perform state observation on the current frame image, i.e., perform target detection to obtain a state observation quantity; perform data association on the obtained state observation quantity and the new state estimation quantity obtained in step (4) to make a preliminary match between the state observation quantity and the state estimation quantity; and divide the targets into matched targets and unmatched targets according to the matching results; and divide the unmatched targets into unmatched prediction frames and unmatched detection frames, and then perform the following classification operations on different combinations of targets according to different situations: (5.1) For an unmatched prediction box, if it is not at the edge of the image, it is considered to be an unoccluded target and the tracking box is still retained, but its observation noise R j The whole image is magnified 3 times. If it is at the edge of the image, it is considered that the target has moved out of the field of view, and the prediction box is directly deleted and no longer considered in the subsequent tracking process. (5.2) For the unmatched detection box, it is considered as the target, a new ID is assigned to it, and a new Kalman filter is created for it to participate in the subsequent tracking process; (5.3) For the correctly matched target, continue to retain its ID and continue tracking; (6) Using state estimation and state observation to fuse information and update various parameters in the tracking system, specifically including the following steps: (6.1) Hybrid model correction, Kalman filtering is performed on the model, and the input quantities are the initial state of the model at time k-1, X 0j (k-1|k-1), the covariance P of the model at time k-1 0j (k-1|k-1), the observed state Z(k) at time k; determine the predicted state at time k and the predicted covariance for Among them, F j (k|k-1) is the state transition matrix of the j-th motion model from time k-1 to time k, G j (k|k-1) is the noise driving matrix of the j-th motion model from time k-1 to time k, W j (k-1) is the system noise at time k-1, and W j The (k-1) system noise covariance matrix is ​​expressed as Q j (k-1); The model transformation matrix is ​​expressed as Among them, p ij represents the model transition probability from the i-th motion model to the j-th motion model, 1≤i≤r,1≤j≤r; The model transition probability μ at time k-1 ij (k-1|k-1) is expressed as r is the total number of models in the model set, μ i (k-1) is the probability corresponding to the i-th model at time k-1; Determine the covariance matrix of measurement error and calculate S j (k) is: Among them, R j (k) is V in the observation equation j (k) The observation noise covariance matrix; Determine the Kalman gain K j (k) is: Determine the state quantity at time k after filtering and covariance for: (6.2) Update the probability of each model; the update of the model will directly affect the accuracy of the algorithm. The maximum likelihood function method is used to update the model. The model weight is determined by calculating the similarity between the current model and the current motion state; the maximum likelihood function Λ that best matches model j at time k is j (k) in, Then we get the updated probability μ of model j j (k) Where c is the normalization constant, that is, (6.3) Output interaction: Output the result of information fusion and update the probability of each model; determine the state quantity after the model output fusion at time k And the covariance P(k|k) after fusion is The target state after information fusion is used as the target tracking result of the current frame image; (7) When the camera captures the next frame of image, the above steps (2) to (6) are repeated to continuously obtain the target's motion position and realize air-ground time-sensitive infrared target tracking.

2. The interactive multi-model infrared time-sensitive target tracking method integrating image features according to claim 1, characterized in that: The target detection algorithm in step (1) uses an infrared target detection algorithm based on block matching to perform target detection on the image to obtain the target position.

3. The interactive multi-model infrared time-sensitive target tracking method integrating image features according to claim 1, characterized in that: The target detection algorithm in step (1) uses an infrared target detection algorithm based on deep learning to detect infrared targets in the image and obtain the target position.

4. The interactive multi-model infrared time-sensitive target tracking method integrating image features according to claim 1, wherein: In the step (3.1), a variable acceleration motion model, a uniform speed turning model, a Singer model, or a combination of the variable acceleration motion model, the uniform speed turning model, and the Singer model is added to construct an interactive multi-model Kalman filter model set, and used as a state prediction model of the Kalman filter.

5. The interactive multi-model infrared time-sensitive target tracking method integrating image features according to claim 1, characterized in that: The step (4.1) performs registration by extracting SIFT features of the image. The specific steps are as follows: a. Construct scale space; construct Gaussian difference scale space based on Gaussian kernel function filtering. The Gaussian kernel function G(x, y, σ) used for the image is Among them, x, y represent the pixel coordinates in the image, σ represents the size of the scale, and a Gaussian kernel function with different σ is selected to perform a convolution operation with the original image to obtain the value L(x, y, σ) of the image at different scales: L(x,y,σ)=G(x,y,σ)*I(x,y) Among them, "*" is the convolution operation, I(x,y) is the pixel value at (x,y); after obtaining L(x,y,σ) at different scales, the L(x,y,σ) of adjacent scales are differentiated to obtain the information value at the original scale, and then the DOG scale space is constructed. The specific calculation method of the operation is: D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)=L(x,y,kσ)-L(x,y,σ) Among them, k is an integer, and information at different scales can be obtained by adjusting the k value; b. Extract key points from the image; after constructing the DOG scale space in step a, determine whether each point is a key point; select 8 points in the same scale space and 9 points in adjacent scales for calculation. Since there may be extreme points on the edge, they need to be eliminated. First, calculate the Hessian matrix H at the current pixel: Among them, D xx , D yy Indicates that the image of a certain scale in the DOG pyramid is differentiated twice in the x-direction and the y-direction, and the derivative is estimated by the neighbor difference of adjacent sampling points; D xy The image of a certain scale in the DOG pyramid is first differentiated in the x direction and then in the y direction. The derivative is also estimated by the neighbor difference of adjacent sampling points. Let α and β be the two eigenvalues ​​of the matrix H, then we get: tr(H)=D xx +D yy =α+β it(H)=D xx +D yy -D xy 2 =α·β When α and β are equal, tr(H) 2 / det(H) takes the minimum; tr(H) 2 / det(H) is used as a criterion for judging edge points. When its value is less than 1.2, it is considered a key point and the position information of the key point is obtained; c. Determine the direction of the key points; for the key points selected in step b, use the neighborhood gradient and direction to represent their directional characteristics. The specific calculation formula is: Among them, m(x,y) is the neighborhood gradient of the key point, and θ(x,y) is the direction of the key point. After calculating the neighborhood gradient at the key point, the gradient direction and amplitude corresponding to the pixels in the neighborhood of the feature point are counted using the histogram. The horizontal axis of the gradient direction histogram is the angle of the gradient direction. d. Determine the descriptor of the key point; after obtaining the position and direction of the key point through steps ac, rotate the original coordinate axis according to the direction of the key point, calculate the gradient magnitude and direction of the pixels in a 16x16 window centered on the feature point, and divide the pixels in this area into 16 blocks, each of which is a histogram statistics of 8 directions within its pixels, forming a 128-dimensional feature vector; e. Perform steps a through d on both the previous and current frames to obtain descriptor sets T1 and T2 of the key points of the two images, respectively. Use Euclidean distance to characterize the similarity of the key points in the two images, perform correlation matching based on the similarity, and solve the affine transformation matrix of the two images based on the corresponding points.

6. The interactive multi-model infrared time-sensitive target tracking method integrating image features according to claim 1, wherein: The step (4.1) performs registration by extracting SURF features, ORB features, or AKAZE feature image key points of the image.

Images