Interactive multi-model infrared time-sensitive target tracking method fusing image features

Abstract

The invention provides an interactive multi-model infrared time-sensitive target tracking method fusing image features, aiming at solving the problem that the motion state of a ground target is difficult to determine due to rigid motion generated by an airborne camera influenced by air flow and strong maneuverability of the ground target in infrared target tracking in an air-ground scene. According to the method, under the framework of Kalman filtering, a target detection result is used as a state observer, inter-frame registration is introduced to compensate for errors caused by coordinate system transformation due to shaking of an airborne platform, strong maneuverability of a ground moving target is represented by interactive multiple models, and finally, the moving target is accurately tracked through data association. The tracking method provided by the invention is low in complexity, can well adapt to a complex environment of air-ground imaging, and has important engineering significance, and the tracking precision is not influenced by the precision of a target detection result.

Description

Technical Field

[0001] The invention belongs to the field of image analysis, and in particular relates to an interactive multi-model infrared time-sensitive target tracking method integrating image features, which is suitable for infrared target tracking tasks under the background of air-to-ground imaging. Background Art

[0002] With the rapid development of UAV remote sensing technology, the use of UAVs carrying various payloads to perform reconnaissance, strike and other tasks can have a significant impact on the battlefield situation. By analyzing and processing the videos collected by UAVs, time-sensitive targets can be intelligently detected and tracked, thereby achieving efficient perception, monitoring and analysis of the battlefield situation. Since modern warfare is full of complex and fierce confrontations and various interference methods, infrared detection technology can play an important role due to its advantages of good concealment and strong anti-interference ability, and is widely used in early warning, reconnaissance, tracking, guidance and other fields.

[0003] As a passive detection method, infrared imaging technology has the advantages of concealment, anti-interference, and all-day operation. However, in complex battlefield environments, the target needs to be detected as far as possible, which results in a small imaging area of ​​the target in the infrared detection system, and lacks important information such as texture and structure. At the same time, it is easily affected by clutter and noise in the environment, making the target difficult to detect in the clutter and noise. There are many methods for studying infrared target detection. Although the existing technology can obtain the position coordinates of the target, there are certain errors.

[0004] For the target tracking problem in the background of air-to-ground imaging, the airborne platform is affected by airflow in the air and is always in irregular shaking. This shaking of the imaging platform will cause the field of view of the infrared camera to rotate, causing the background such as roads, trees, buildings in the image and the target to be detected on the ground to tilt, that is, the coordinate system corresponding to the two images changes, affecting the accurate representation of the target position. The error caused by the shaking of the imaging payload will further increase the error in the target detection link and affect the accurate tracking of air-to-ground time-sensitive targets.

[0005] At the same time, due to the mobility of the ground target itself and the relative motion between the UAV platform and the ground target, the motion model of the ground target is difficult to be well described by a single model. The traditional single model has great limitations in modeling the motion model of the target, which will greatly affect the accuracy of target tracking.

[0006] In view of the existing problems, the present invention proposes an interactive multi-model infrared time-sensitive target tracking method that integrates image features, which is suitable for the scenario where an airborne infrared platform detects and tracks a ground mobile target. Under the framework of Kalman filtering, the target detection algorithm is used as the observation model, the interactive multi-model is used as the prediction model, the image features are extracted for inter-frame registration, and the affine transformation matrix obtained by image registration is used to correct the coordinates of the detected target to compensate for the error caused by the coordinate system transformation caused by the shaking of the airborne platform. Finally, the Hungarian algorithm is used for data association, which can accurately track the ground infrared target. Summary of the invention

[0007] The purpose of the present invention is to solve the many problems that restrict target tracking accuracy in the above-mentioned air-to-ground imaging scenes. An interactive multi-model infrared time-sensitive target tracking method that integrates image features is proposed. The method can accurately track targets detected on the ground in air-to-ground imaging scenes with complex backgrounds, and can well overcome the significant impact of the shaking of the airborne platform and the strong maneuverability of the tracked target on the tracking accuracy.

[0008] The interactive multi-model infrared time-sensitive target tracking method of the present invention fused with image features specifically comprises the following steps:

[0009] (1) Create a state observation model: When the tracking camera of the airborne platform captures a moving target on the ground, the pixel position of the moving target in the image is obtained through the target detection algorithm, and the target detection result is used as the state observation model of the tracking system;

[0010] (2) Creating the state quantity of the target in the image includes the following steps:

[0011] (2.1) Input image: including the k-1th frame image and the kth frame image, where k = 2, 3, ..., n is a sequence number;

[0012] (2.2) Establishing the target state: Perform target detection on the k-1th frame image and the kth frame image, and obtain the target frame set S of the k-1th frame image and the kth frame image respectively. k-1 and S k , establish the target state vector as

[0013]

[0014] Among them, x k and k They refer to the horizontal and vertical position coordinates of the target k at the moment along the image, and are the horizontal and vertical pixel movement speeds along the image at time k, and are the accelerations of the horizontal and vertical pixel motions along the image at time k, respectively; since the image sequence between frames is discretized, the velocity is calculated by the difference of the coordinates, and the acceleration is calculated by the difference of the velocity. The specific calculation formula is:

[0015]

[0016] Among them, x k-1 and k-1 are the horizontal and vertical position coordinates of the target at time k-1 along the image, and are the pixel movement speeds along the horizontal and vertical directions of the image at time k-1 respectively; the construction of the initial state quantity requires the use of the first and second frames of images, and the calculation of the state quantity starting from the third frame of image is performed using the current frame of image and the previous frame of image;

[0017] (3) constructing an interactive multi-model Kalman filter and estimating the state of the target online, and predicting the target state in the next frame of the image based on the target state established in step (2);

[0018] (3.1) Constructing an interactive multi-model Kalman filter model set

[0019] The possible motion modes of the target are mapped to a Kalman filter model set, where each model represents a different motion mode. Assuming that the interactive multi-model filter contains r different Kalman filter models, the general model formula is determined as:

[0020] X j (k+1)=F j (k)X j (k)+G j (k)W j (k) (1)

[0021] Z(k)=H j (k)X j (k)+V j (k) (2)

[0022] Wherein, formula (1) represents the state equation of the j-th motion model, formula (2) represents the measurement equation of the j-th motion model, and X j (k) is the state vector at time k, F j (k) is the state transfer matrix at time k, G j (k) is the noise driving matrix at time k, W j (k) is the system noise at time k, V j (k) is the observation noise at time k, H j(k) is the observation matrix at time k, Z(k) is the observation quantity at time k; j = 1, 2..., r is the model number, and W j The system noise variance matrix of (k) is expressed as Q j , V j The observation noise variance matrix of (k) is expressed as R j ;

[0023] According to the actual motion situation of the ground motion target, the motion mode of the ground motion target is set to target uniform linear motion and target uniform accelerated linear motion, and the state equations under the two motion modes are further determined;

[0024] The state equation of the target uniform motion is:

[0025]

[0026] The state equation of the target uniformly accelerated motion is:

[0027]

[0028] In the formula, x(k) and y(k) are the horizontal and vertical coordinates of the target in the field of view at time k, respectively. are the target lateral velocity and longitudinal velocity at time k, are the lateral acceleration and longitudinal acceleration of the target at time k; x(k+1) and y(k+1) are the horizontal and vertical coordinates of the target in the field of view at time k+1, are the target lateral velocity and longitudinal velocity at time k+1 respectively, are the lateral acceleration and longitudinal acceleration of the target at time k+1 respectively; T is the sampling time interval, ω j (k) is the model noise, which is modeled as white noise;

[0029] The model is designed with n1 uniform motion equations and n2 uniform acceleration motion equations, satisfying n1+n2=r. Each motion equation corresponds to a different Q j and R j ;

[0030] (3.2) The uniform motion and uniform acceleration motion models of the target are used to form a maneuvering target filter model. There are a total of r Kalman filter models. According to the target state quantity constructed in step (2), the motion model parameters Q1, Q2, Q3...Q r 、R1,R2,R3...R r And combine the weight factors corresponding to each model for input interaction to obtain the estimated value of each target state quantity;

[0031] (4) Calculating the affine transformation matrix of the inter-frame images by inter-frame registration and correcting the estimated value of the target state quantity in step (3), including the following steps:

[0032] (4.1) Extract key points from the previous frame and the current frame according to the image features, and put the generated descriptors into two sets T1 and T2; calculate the Euclidean distance between the descriptors in the two sets to characterize their similarity, perform key point matching, and after key point purification, calculate the affine transformation matrix between the two images according to their corresponding relationship;

[0033] (4.2) transforming the predicted bounding box from the coordinate system of the previous frame image to the coordinate system of the current frame image according to the affine transformation matrix to compensate for the effect of camera motion on the predicted position of the target;

[0034] (4.3) The estimated value of the state quantity in step (3) is corrected by the affine transformation matrix obtained by inter-frame registration, and the estimated value is transformed into the coordinate system corresponding to the current image as a new state estimate;

[0035] (5) Perform state observation on the current frame image, that is, perform target detection to obtain state observations; perform data association on the obtained state observations and the new state estimates obtained in step (4) to make preliminary matching of the new state estimates of each target; and divide the targets into matched targets and unmatched targets according to the matching results; divide the unmatched targets into unmatched prediction frames and unmatched detection frames; and then perform the following classification operations on targets of different combinations according to different situations:

[0036] (5.1) For an unmatched prediction box, if it is not at the edge of the image, it is considered to be an unoccluded target and the tracking box is still retained, but its observation noise R j The whole image is magnified 3 times; if it is at the edge of the image, it is considered that the target has moved out of the field of view, and the prediction box is directly deleted and no longer considered in the subsequent tracking process;

[0037] (5.2) For the unmatched detection box, it is regarded as the target, a new ID is assigned to it, and a new Kalman filter is created for it to participate in the subsequent tracking process;

[0038] (5.3) For the correctly matched target, continue to retain its ID and continue tracking;

[0039] (6) Using the state estimation and state observation to fuse information and update various parameters in the tracking system, the specific steps include:

[0040] (6.1) Hybrid model correction, Kalman filtering is performed on the model, and the input quantities are the initial state of the model at time k-1, X 0j(k-1|k-1), the covariance P of the model at time k-1 0j (k-1|k-1), the observed state Z(k) at time k; determine the predicted state at time k and the predicted covariance for

[0041]

[0042] Among them, F j (k|k-1) is the state transfer matrix of the jth motion model from time k-1 to time k, G j (k|k-1) is the noise driving matrix of the jth motion model from time k-1 to time k, W j (k-1) is the system noise at time k-1, and W j The system noise covariance matrix of (k-1) is expressed as Q j (k-1);

[0043] The model transformation matrix is ​​expressed as

[0044]

[0045] Among them, p ij represents the model transition probability from the i-th motion model to the j-th motion model, 1≤i≤r,1≤j≤r;

[0046] The model transition probability μ at time k-1 ij (k-1|k-1) is expressed as

[0047]

[0048] r is the total number of models in the model set, μ i (k-1) is the probability corresponding to the i-th model at time k-1;

[0049] Determine the covariance matrix of the measurement error and calculate S j (k) is:

[0050]

[0051] Among them, R j (k) is V in the observation equation j (k) The observation noise covariance matrix;

[0052] Determine the Kalman gain K j (k) is:

[0053]

[0054] Determine the state quantity at time k after filtering and covariance for:

[0055]

[0056]

[0057] (6.2) Update the probability of each model; the update of the model will directly affect the accuracy of the algorithm. The maximum likelihood function method is used to update the model. The model weight is determined by calculating the similarity between the current model and the current motion state; the maximum likelihood function Λ that best matches model j at time k j (k)

[0058]

[0059] in,

[0060] Then we get the updated probability μ of model j j (k)

[0061]

[0062] Where c is the normalization constant, that is

[0063]

[0064] (6.3) Output interaction: Output the result of information fusion and update the probability of each model; determine the state quantity after model output fusion at time k And the fused covariance P(k|k) is

[0065]

[0066]

[0067] The target state quantity after information fusion is used as the target tracking result of the current frame image;

[0068] (7) When the camera captures the next frame of image, the above steps (2) to (6) are repeated to continuously obtain the target's moving position and realize air-to-ground time-sensitive infrared target tracking.

[0069] Preferably, the target detection algorithm in step (1) uses an infrared target detection algorithm based on block matching to perform target detection on the image to obtain the target position.

[0070] Preferably, the target detection algorithm in step (1) uses an infrared target detection algorithm based on deep learning to detect infrared targets in the image to obtain the target position.

[0071] Preferably, in the step (3.1), a variable acceleration motion model, or a uniform speed turning model, or a Singer model, or a combination of a variable acceleration motion model, a uniform speed turning model, and a Singer model is added to construct an interactive multi-model Kalman filter model set, and used as a state prediction model of the Kalman filter.

[0072] Preferably, the step (4.1) performs registration by extracting SIFT features of the image, and the specific operation steps are:

[0073] a. Construct scale space; construct Gaussian difference scale space based on Gaussian kernel function filtering. The Gaussian kernel function G(x, y, σ) used for the image is

[0074]

[0075] Among them, x and y represent the pixel coordinates in the image, σ represents the size of the scale, and a Gaussian kernel function with different σ is selected to perform a convolution operation with the original image to obtain the value L(x, y, σ) of the image at different scales:

[0076] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0077] Among them, "*" is the convolution operation, I(x,y) is the pixel value at (x,y); after obtaining L(x,y,σ) at different scales, the L(x,y,σ) at adjacent scales is differentiated to obtain the information value at the original scale, and then the DOG scale space is constructed. The specific calculation method of the operation is:

[0078] D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)=L(x,y,kσ)-L(x,y,σ)

[0079] Among them, k is an integer, and the information at different scales can be obtained by adjusting the k value;

[0080] b. Extract key points in the image; after constructing the DOG scale space in step a, determine whether each point is a key point; select 8 points in the same scale space and 9 points in adjacent scales for calculation. Since there may be extreme points on the edge, the extreme points on the edge need to be eliminated. First, calculate the Hessian matrix H at the current pixel point:

[0081]

[0082] Among them, D xx , D yy Indicates that the image of a certain scale in the DOG pyramid is differentiated twice in the x direction and the y direction, and the derivative is estimated by the neighbor difference of adjacent sampling points; D xyIt means that the image of a certain scale in the DOG pyramid is first differentiated in the x direction and then in the y direction. The derivative is also estimated by the neighbor difference of adjacent sampling points. Let α and β be the two eigenvalues ​​of the matrix H, then we get:

[0083] tr(H)=D xx +D yy =α+β

[0084] det(H)=D xx +D yy -D xy 2 =α·β

[0085]

[0086] When α and β are equal, tr(H) 2 / det(H) takes the minimum; tr(H) 2 / det(H) is used as a criterion for judging edge points. When its value is less than 1.2, it is considered a key point and the position information of the key point is obtained;

[0087] c. Determine the direction of the key points; for the key points selected in step b, use the neighborhood gradient and direction to represent their directional characteristics. The specific calculation formula is:

[0088]

[0089] Among them, m(x, y) is the neighborhood gradient of the key point, and θ(x, y) is the direction of the key point. After calculating the neighborhood gradient at the key point, the histogram is used to count the gradient direction and amplitude corresponding to the pixels in the neighborhood of the feature point. The horizontal axis of the histogram of the gradient direction is the angle of the gradient direction.

[0090] d. Determine the descriptor of the key point; after obtaining the position and direction of the key point through steps ac, rotate the original coordinate axis according to the direction of the key point, calculate the gradient amplitude and direction of the pixels of the 16x16 window centered on the feature point, and divide the pixels in this area into 16 blocks, each of which is the histogram statistics of 8 directions within the pixel, forming a 128-dimensional feature vector;

[0091] e. Perform steps a to d on both the previous frame image and the current frame image to obtain descriptor sets T1 and T2 of the key points of the two images respectively, characterize the similarity of the key points in the two images using Euclidean distance, perform association matching based on the similarity, and solve the affine transformation matrix of the two images based on the corresponding points.

[0092] Preferably, the step (4.1) performs registration by extracting SURF features, ORB features, or AKAZE feature image key points of the image.

[0093] The advantages and positive effects of the present invention are as follows: Starting from the difficulty of the air-to-ground time-sensitive infrared target tracking task, the present invention introduces an interactive multi-model filtering algorithm to characterize the strong maneuverability of the moving target, which can effectively solve the problem that the high-speed maneuverability of the moving target causes the tracking algorithm to converge slowly and is easy to lose. By introducing an inter-frame registration algorithm to compensate for the error caused by the transformation of the coordinate system in the field of view due to the rigid movement of the airborne platform, there is no need to use other sensors, the algorithm is simple and has great practical value. In summary, the present invention has stronger robustness and anti-interference capabilities than the original tracking method. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0095] Figure 1 It is a flow chart of the IMM filtering algorithm of the present invention;

[0096] Figure 2 is an algorithm flow chart of inter-frame registration in step (4) of the present invention;

[0097] Figure 3 It is the overall algorithm flow chart of the tracking method of the present invention. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0099] Mounting various imaging payloads on UAV platforms can effectively detect ground targets. This airborne detection method, combined with the advantages of infrared imaging payloads such as concealment, anti-interference, and high precision, can play a very important role in air-to-ground time-sensitive target detection tasks. At the same time, there are many factors that restrict accurate target tracking in air-to-ground time-sensitive target imaging scenarios. For example: the ground imaging environment background is complex, and the infrared target detection accuracy is not high; the rigid motion of the airborne camera causes the coordinate system in the imaging field of view to constantly change, making the coordinates of the detected target inaccurate; most ground targets are in a maneuvering state, and the rapid motion of the airborne platform makes it difficult to establish the target's motion model. The above factors make it difficult to effectively detect and track ground targets using traditional methods.

[0100] The present invention proposes an interactive multi-model infrared time-sensitive target tracking method that integrates image features. Starting from the characteristics of air-to-ground infrared target imaging, based on the traditional Kalman filter target tracking algorithm, it introduces inter-frame registration to compensate for the errors caused by the rigid motion of the airborne platform resulting from the transformation of the coordinate system in the field of view, introduces an interactive multi-model filtering algorithm to characterize the strong maneuverability of ground moving targets, and adjusts the data association of the tracking target in real time according to the corresponding relationship between the predicted state and the observed state, and continuously iterates and continues the tracking process. The specific flow chart of the method is as follows: Figure 3 The specific implementation method is as follows:

[0101] (1) Create a state observation model: When the tracking camera of the airborne platform captures a moving target on the ground, the pixel position of the moving target in the image is obtained through the target detection algorithm, and the target detection result is used as the state observation model of the tracking system;

[0102] Preferably, the target detection algorithm uses an infrared target detection algorithm based on block matching to perform target detection on the image to obtain the target position;

[0103] Preferably, the target detection algorithm or an infrared target detection algorithm based on deep learning is used to detect infrared targets in the image to obtain the target position;

[0104] (2) Creating the state quantity of the target in the image includes the following steps:

[0105] (2.1) Input image: including the k-1th frame image and the kth frame image, where k = 2, 3, ..., n is a sequence number;

[0106] (2.2) Establishing the target state: Perform target detection on the k-1th frame image and the kth frame image, and obtain the target frame set S of the k-1th frame image and the kth frame image respectively. k-1 and S k , establish the target state vector as

[0107]

[0108] Among them, x k and k They refer to the horizontal and vertical position coordinates of the target k at the moment along the image, and are the horizontal and vertical pixel movement speeds along the image at time k, and are the accelerations of the horizontal and vertical pixel motions along the image at time k, respectively; since the image sequence between frames is discretized, the velocity is calculated by the difference of the coordinates, and the acceleration is calculated by the difference of the velocity. The specific calculation formula is:

[0109]

[0110] Among them, x k-1 and k-1 are the horizontal and vertical position coordinates of the target at time k-1 along the image, and are the pixel movement speeds along the horizontal and vertical directions of the image at time k-1 respectively; the construction of the initial state quantity requires the use of the first and second frames of images, and the calculation of the state quantity starting from the third frame of image is performed using the current frame of image and the previous frame of image;

[0111] (3) Construct an interactive multi-model Kalman filter and estimate the state of the target online. According to the target state established in step (2), predict the target state in the next frame of the image. The specific algorithm flow chart of the interactive multi-model is as follows: Figure 1 As shown;

[0112] (3.1) Constructing an interactive multi-model Kalman filter model set

[0113] The possible motion modes of the target are mapped to a Kalman filter model set, where each model represents a different motion mode. Assuming that the interactive multi-model filter contains r different Kalman filter models, the general model formula is determined as:

[0114] X j (k+1)=F j (k)X j (k)+G j (k)W j (k) (1)

[0115] Z(k)=H j (k)X j (k)+V j (k) (2)

[0116] Wherein, formula (1) represents the state equation of the j-th motion model, formula (2) represents the measurement equation of the j-th motion model, and X j (k) is the state vector at time k, F j (k) is the state transfer matrix at time k, G j (k) is the noise driving matrix at time k, W j (k) is the system noise at time k, V j (k) is the observation noise at time k, H j (k) is the observation matrix at time k, Z(k) is the observation quantity at time k; j = 1, 2..., r is the model number, and W j The system noise variance matrix of (k) is expressed as Q j, V j The observation noise variance matrix of (k) is expressed as R j ;

[0117] According to the actual movement of the ground moving target, the movement mode of the ground moving target is set to target uniform linear motion and target uniform accelerated linear motion, and the state equations under the two movement modes are further determined; as an improvement, a variable acceleration motion model, or a uniform turning model, or a Singer model, or a combination of a variable acceleration motion model, a uniform turning model, and a Singer model is added to construct an interactive multi-model Kalman filter model set, and used as the state prediction model of the Kalman filter.

[0118] The state equation of the target uniform motion is:

[0119]

[0120] The state equation of the target uniformly accelerated motion is:

[0121]

[0122] In the formula, x(k) and y(k) are the horizontal and vertical coordinates of the target in the field of view at time k, respectively. are the target lateral velocity and longitudinal velocity at time k, are the lateral acceleration and longitudinal acceleration of the target at time k; x(k+1) and y(k+1) are the horizontal and vertical coordinates of the target in the field of view at time k+1, are the target lateral velocity and longitudinal velocity at time k+1, are the lateral acceleration and longitudinal acceleration of the target at time k+1 respectively; T is the sampling time interval, ω j (k) is the model noise, which is modeled as white noise;

[0123] The model is designed with n1 uniform motion equations and n2 uniform acceleration motion equations, satisfying n1+n2=r. Each motion equation corresponds to a different Q j and R j ;

[0124] (3.2) The uniform motion and uniform acceleration motion models of the target are used to form a maneuvering target filter model. There are a total of r Kalman filter models. According to the target state quantity constructed in step (2), the motion model parameters Q1, Q2, Q3...Q r 、R1,R2,R3...R r And combine the weight factors corresponding to each model for input interaction to obtain the estimated value of each target state quantity;

[0125] (4) The affine transformation matrix of the inter-frame images is calculated through inter-frame registration, and the estimated value of the target state quantity in step (3) is corrected. The registration algorithm flow chart is as follows: Figure 2 As shown, the specific steps include:

[0126] (4.1) Extract key points from the previous frame and the current frame according to the image features, and put the generated descriptors into two sets T1 and T2; calculate the Euclidean distance between the descriptors in the two sets to characterize their similarity, perform key point matching, and after key point purification, calculate the affine transformation matrix between the two images according to their corresponding relationship;

[0127] Preferably, the step (4.1) performs registration by extracting SIFT features of the image, and the specific operation steps are:

[0128] a. Construct scale space; construct Gaussian difference scale space based on Gaussian kernel function filtering. The Gaussian kernel function G(x, y, σ) used for the image is

[0129]

[0130] Among them, x and y represent the pixel coordinates in the image, σ represents the size of the scale, and a Gaussian kernel function with different σ is selected to perform a convolution operation with the original image to obtain the value L(x, y, σ) of the image at different scales:

[0131] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0132] Among them, "*" is the convolution operation, I(x,y) is the pixel value at (x,y); after obtaining L(x,y,σ) at different scales, the L(x,y,σ) at adjacent scales is differentiated to obtain the information value at the original scale, and then the DOG scale space is constructed. The specific calculation method of the operation is:

[0133] D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)=L(x,y,kσ)-L(x,y,σ)

[0134] Among them, k is an integer, and the information at different scales can be obtained by adjusting the k value;

[0135] b. Extract key points in the image; after constructing the DOG scale space in step a, determine whether each point is a key point; select 8 points in the same scale space and 9 points in adjacent scales for calculation. Since there may be extreme points on the edge, the extreme points on the edge need to be eliminated. First, calculate the Hessian matrix H at the current pixel point:

[0136]

[0137] Among them, D xx , D yy Indicates that the image of a certain scale in the DOG pyramid is differentiated twice in the x direction and the y direction, and the derivative is estimated by the neighbor difference of adjacent sampling points; D xy It means that the image of a certain scale in the DOG pyramid is first differentiated in the x direction and then in the y direction. The derivative is also estimated by the neighbor difference of adjacent sampling points. Let α and β be the two eigenvalues ​​of the matrix H, then we get:

[0138] tr(H)=D xx +D yy =α+β

[0139] det(H)=D xx +D yy -D xy 2 =α·β

[0140]

[0141] When α and β are equal, tr(H) 2 / det(H) takes the minimum; tr(H) 2 / det(H) is used as a criterion for judging edge points. When its value is less than 1.2, it is considered as a key point and the position information of the key point is obtained;

[0142] c. Determine the direction of the key points; for the key points selected in step b, use the neighborhood gradient and direction to represent their directional characteristics. The specific calculation formula is:

[0143]

[0144] Among them, m(x, y) is the neighborhood gradient of the key point, and θ(x, y) is the direction of the key point. After calculating the neighborhood gradient at the key point, the histogram is used to count the gradient direction and amplitude corresponding to the pixels in the neighborhood of the feature point. The horizontal axis of the histogram of the gradient direction is the angle of the gradient direction.

[0145] d. Determine the descriptor of the key point; after obtaining the position and direction of the key point through steps ac, rotate the original coordinate axis according to the direction of the key point, calculate the gradient amplitude and direction of the pixels of the 16x16 window centered on the feature point, and divide the pixels in this area into 16 blocks, each of which is the histogram statistics of 8 directions within the pixel, forming a 128-dimensional feature vector;

[0146] e. Performing the operations of step a to step d on both the previous frame image and the current frame image, respectively obtaining the descriptor sets T1 and T2 of the key points of the two images, representing the similarity of the key points in the two images using Euclidean distance, performing association matching based on the similarity, and solving the affine transformation matrix of the two images based on the corresponding points;

[0147] (4.2) transforming the predicted bounding box from the coordinate system of the previous frame image to the coordinate system of the current frame image according to the affine transformation matrix to compensate for the effect of camera motion on the predicted position of the target;

[0148] (4.3) Correcting the estimated value of the state quantity in step (3) by the affine transformation matrix obtained through inter-frame registration, transforming the estimated value into the coordinate system corresponding to the current image as a new state estimate;

[0149] (5) Perform state observation on the current frame image, that is, perform target detection to obtain state observations; perform data association on the obtained state observations and the new state estimates obtained in step (4) to make preliminary matching of the new state estimates of each target; and divide the targets into matched targets and unmatched targets according to the matching results; divide the unmatched targets into unmatched prediction frames and unmatched detection frames; and then perform the following classification operations on targets of different combinations according to different situations:

[0150] (5.1) For an unmatched prediction box, if it is not at the edge of the image, it is considered to be an unoccluded target and the tracking box is still retained, but its observation noise R j The whole image is magnified 3 times; if it is at the edge of the image, it is considered that the target has moved out of the field of view, and the prediction box is directly deleted and no longer considered in the subsequent tracking process;

[0151] (5.2) For the unmatched detection box, it is regarded as the target, a new ID is assigned to it, and a new Kalman filter is created for it to participate in the subsequent tracking process;

[0152] (5.3) For the correctly matched target, continue to retain its ID and continue tracking;

[0153] (6) Using the state estimation and state observation to fuse information and update various parameters in the tracking system, the specific steps include:

[0154] (6.1) Hybrid model correction, Kalman filtering is performed on the model, and the input quantities are the initial state of the model at time k-1, X 0j (k-1|k-1), the covariance P of the model at time k-1 0j (k-1|k-1), the observed state Z(k) at time k; determine the predicted state at time k and the predicted covariance for

[0155]

[0156] Among them, F j (k|k-1) is the state transfer matrix of the jth motion model from time k-1 to time k, G j (k|k-1) is the noise driving matrix of the jth motion model from time k-1 to time k, W j (k-1) is the system noise at time k-1, and W j The system noise covariance matrix of (k-1) is expressed as Q j (k-1);

[0157] The model transformation matrix is ​​expressed as

[0158]

[0159] Among them, p ij represents the model transition probability from the i-th motion model to the j-th motion model, 1≤i≤r,1≤j≤r;

[0160] The model transition probability μ at time k-1 ij (k-1|k-1) is expressed as

[0161]

[0162] r is the total number of models in the model set, μ i (k-1) is the probability corresponding to the i-th model at time k-1;

[0163] Determine the covariance matrix of the measurement error and calculate S j (k) is:

[0164]

[0165] Among them, R j (k) is V in the observation equation j (k) The observation noise covariance matrix;

[0166] Determine the Kalman gain K j (k) is:

[0167]

[0168] Determine the state quantity at time k after filtering and covariance for:

[0169]

[0170]

[0171] (6.2) Update the probability of each model; the update of the model will directly affect the accuracy of the algorithm. The maximum likelihood function method is used to update the model. The model weight is determined by calculating the similarity between the current model and the current motion state. The maximum likelihood function that best matches model j at time k is

[0172]

[0173] Among them, v j (k)=Z(k)-H j (k)X j (k|k-1)

[0174] Then we can get the update probability μ of model j j (k)

[0175]

[0176] Where c is the normalization constant, that is

[0177]

[0178] (6.3) Output interaction: Output the result of information fusion and update the probability of each model; determine the state quantity of the model output after fusion at time k And the fused covariance P(k|k) is

[0179]

[0180] The target state quantity after information fusion is used as the target tracking result of the current frame image;

[0181] (7) When the camera captures the next frame of image, the above steps (2) to (6) are repeated to continuously obtain the target's moving position and realize air-to-ground time-sensitive infrared target tracking.

[0182] As an improvement, the step (4.1) performs registration by extracting SURF features, ORB features, or AKAZE feature image key points of the image.

[0183] The present invention can realize continuous tracking of ground mobile targets under the condition that the detection accuracy of infrared target coordinates is not enough in the background of air-to-ground imaging. This method introduces an inter-frame registration algorithm on the traditional multi-target tracking algorithm based on interactive multiple model filtering to compensate for the influence of the movement and shaking of the airborne imaging platform on target tracking. At the same time, it can well overcome the influence of the strong mobility of ground targets on target tracking, and improve the robustness of infrared target tracking in the background of air-to-ground imaging.

[0184] The air-to-ground infrared target tracking method proposed in the present invention has no requirements on the motion state of the aerial platform. Target tracking can be achieved when the airborne platform is in a hovering static state or a moving state. The present invention does not involve other sensors and can perform target tracking normally under the conditions of GNSS denial and IMU failure. The model set construction of the interactive multi-model and the feature extraction method of the inter-frame image in the present invention are only the preferred solutions of the interactive multi-model infrared target tracking method that integrates image features. Other model set selection schemes and feature extraction schemes also belong to the scope of the present invention.

[0185] The above is only a preferred embodiment of the present invention and is not intended to limit the present invention. All technical solutions within the scope of the present invention belong to the protection scope of the present invention. Some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An interactive multi-model infrared time-sensitive target tracking method integrating image features, characterized in that: The steps include: (1) Create a state observation model: When the tracking camera of the airborne platform captures a moving target on the ground, the pixel position of the moving target in the image is obtained through the target detection algorithm, and the target detection result is used as the state observation model of the tracking system; (2) Creating the state quantity of the target in the image includes the following steps: (2.1) Input image: including the k-1th frame image and the kth frame image, where k = 2, 3, ..., n is a sequence number; (2.2) Establishing the target state: Perform target detection on the k-1th frame image and the kth frame image, and obtain the target frame set S of the k-1th frame image and the kth frame image respectively. k-1 and S k , establish the target state vector as Among them, x k and k They refer to the horizontal and vertical position coordinates of the target k at the moment along the image, and are the horizontal and vertical pixel movement speeds along the image at time k, and are the accelerations of the horizontal and vertical pixel motions along the image at time k, respectively; since the image sequence between frames is discretized, the velocity is calculated by the difference of the coordinates, and the acceleration is calculated by the difference of the velocity. The specific calculation formula is: Among them, x k-1 and k-1 are the horizontal and vertical position coordinates of the target at time k-1 along the image, and are the pixel movement speeds along the horizontal and vertical directions of the image at time k-1 respectively; the construction of the initial state quantity requires the use of the first and second frames of images, and the calculation of the state quantity starting from the third frame of image is performed using the current frame of image and the previous frame of image; (3) constructing an interactive multi-model Kalman filter and estimating the state of the target online, and predicting the target state in the next frame of the image based on the target state established in step (2); (3.1) Constructing an interactive multi-model Kalman filter model set The possible motion modes of the target are mapped to a Kalman filter model set, where each model represents a different motion mode. Assuming that the interactive multi-model filter contains r different Kalman filter models, the general model formula is determined as: X j (k+1)=F j (k)X j (k)+G j (k)W j (k) (1) Z(k)=H j (k)X j (k)+V j (k) (2) Wherein, formula (1) represents the state equation of the j-th motion model, formula (2) represents the measurement equation of the j-th motion model, and X j (k) is the state vector at time k, F j (k) is the state transfer matrix at time k, G j (k) is the noise driving matrix at time k, W j (k) is the system noise at time k, V j (k) is the observation noise at time k, H j (k) is the observation matrix at time k, Z(k) is the observation quantity at time k; j = 1, 2..., r is the model number, and W j The system noise variance matrix of (k) is expressed as Q j , V j The observation noise variance matrix of (k) is expressed as R j ; According to the actual motion situation of the ground motion target, the motion mode of the ground motion target is set to target uniform linear motion and target uniform accelerated linear motion, and the state equations under the two motion modes are further determined; The state equation of the target uniform motion is: The state equation of the target uniformly accelerated motion is: In the formula, x(k) and y(k) are the horizontal and vertical coordinates of the target in the field of view at time k, respectively. are the target lateral velocity and longitudinal velocity at time k, are the lateral acceleration and longitudinal acceleration of the target at time k; x(k+1) and y(k+1) are the horizontal and vertical coordinates of the target in the field of view at time k+1, are the target lateral velocity and longitudinal velocity at time k+1 respectively, are the lateral acceleration and longitudinal acceleration of the target at time k+1 respectively; T is the sampling time interval, ω j (k) is the model noise, which is modeled as white noise; The model is designed with n1 uniform motion equations and n2 uniform acceleration motion equations, satisfying n1+n2=r. Each motion equation corresponds to a different Q j and R j ; (3.2) The uniform motion and uniform acceleration motion models of the target are used to form a maneuvering target filter model. There are a total of r Kalman filter models. According to the target state quantity constructed in step (2), the motion model parameters Q1, Q2, Q3...Q r 、R1,R2,R3...R r And combine the weight factors corresponding to each model for input interaction to obtain the estimated value of each target state quantity; (4) Calculating the affine transformation matrix of the inter-frame images by inter-frame registration and correcting the estimated value of the target state quantity in step (3), including the following steps: (4.1) Extract key points from the previous frame and the current frame according to the image features, and put the generated descriptors into two sets T1 and T2; calculate the Euclidean distance between the descriptors in the two sets to characterize their similarity, perform key point matching, and after key point purification, calculate the affine transformation matrix between the two images according to their corresponding relationship; (4.2) transforming the predicted bounding box from the coordinate system of the previous frame image to the coordinate system of the current frame image according to the affine transformation matrix to compensate for the effect of camera motion on the predicted position of the target; (4.3) The estimated value of the state quantity in step (3) is corrected by the affine transformation matrix obtained by inter-frame registration, and the estimated value is transformed into the coordinate system corresponding to the current image as a new state estimate; (5) Perform state observation on the current frame image, that is, perform target detection to obtain state observations; perform data association on the obtained state observations and the new state estimates obtained in step (4) to make preliminary matching of the new state estimates of each target; and divide the targets into matched targets and unmatched targets according to the matching results; divide the unmatched targets into unmatched prediction frames and unmatched detection frames; and then perform the following classification operations on targets of different combinations according to different situations: (5.1) For an unmatched prediction box, if it is not at the edge of the image, it is considered to be an unoccluded target and the tracking box is still retained, but its observation noise R j The whole image is magnified 3 times; if it is at the edge of the image, it is considered that the target has moved out of the field of view, and the prediction box is directly deleted and no longer considered in the subsequent tracking process; (5.2) For the unmatched detection box, it is regarded as the target, a new ID is assigned to it, and a new Kalman filter is created for it to participate in the subsequent tracking process; (5.3) For the correctly matched target, continue to retain its ID and continue tracking; (6) Using the state estimation and state observation to fuse information and update various parameters in the tracking system, the specific steps include: (6.1) Hybrid model correction, Kalman filtering is performed on the model, and the input quantities are the initial state of the model at time k-1, X 0j (k-1|k-1), the covariance P of the model at time k-1 0j (k-1|k-1), the observed state Z(k) at time k; determine the predicted state at time k and the predicted covariance for Among them, F j (k|k-1) is the state transfer matrix of the jth motion model from time k-1 to time k, G j (k|k-1) is the noise driving matrix of the jth motion model from time k-1 to time k, W j (k-1) is the system noise at time k-1, and W j The system noise covariance matrix of (k-1) is expressed as Q j (k-1); The model transformation matrix is ​​expressed as Among them, p ij represents the model transition probability from the i-th motion model to the j-th motion model, 1≤i≤r,1≤j≤r; The model transition probability μ at time k-1 ij (k-1|k-1) is expressed as r is the total number of models in the model set, μ i (k-1) is the probability corresponding to the i-th model at time k-1; Determine the covariance matrix of the measurement error and calculate S j (k) is: Among them, R j (k) is V in the observation equation j (k) The observation noise covariance matrix; Determine the Kalman gain K j (k) is: Determine the state quantity at time k after filtering and covariance for: (6.2) Update the probability of each model; the update of the model will directly affect the accuracy of the algorithm. The maximum likelihood function method is used to update the model. The model weight is determined by calculating the similarity between the current model and the current motion state; the maximum likelihood function Λ that best matches model j at time k j (k) in, Then we get the updated probability μ of model j j (k) Where c is the normalization constant, that is (6.3) Output interaction: Output the result of information fusion and update the probability of each model; determine the state quantity of the model output after fusion at time k And the fused covariance P(k|k) is The target state quantity after information fusion is used as the target tracking result of the current frame image; (7) When the camera captures the next frame of image, the above steps (2) to (6) are repeated to continuously obtain the target's moving position and realize air-to-ground time-sensitive infrared target tracking.

2. The interactive multi-model infrared time-sensitive target tracking method integrating image features as claimed in claim 1, characterized in that: The target detection algorithm in step (1) uses an infrared target detection algorithm based on block matching to perform target detection on the image to obtain the target position.

3. The interactive multi-model infrared time-sensitive target tracking method integrating image features as claimed in claim 1, characterized in that: The target detection algorithm in step (1) uses an infrared target detection algorithm based on deep learning to detect infrared targets in the image and obtain the target position.

4. The interactive multi-model infrared time-sensitive target tracking method integrating image features as claimed in claim 1, characterized in that: In the step (3.1), a variable acceleration motion model, a uniform turning model, a Singer model, or a combination of a variable acceleration motion model, a uniform turning model, and a Singer model is added to construct an interactive multi-model Kalman filter model set, and used as a state prediction model of the Kalman filter.

5. The interactive multi-model infrared time-sensitive target tracking method integrating image features as claimed in claim 1, characterized in that: The step (4.1) performs registration by extracting SIFT features of the image, and the specific operation steps are as follows: a. Construct scale space; construct Gaussian difference scale space based on Gaussian kernel function filtering. The Gaussian kernel function G(x, y, σ) used for the image is Among them, x and y represent the pixel coordinates in the image, σ represents the size of the scale, and a Gaussian kernel function with different σ is selected to perform a convolution operation with the original image to obtain the value L(x, y, σ) of the image at different scales: L(x,y,σ)=G(x,y,σ)*I(x,y) Among them, "*" is the convolution operation, I(x,y) is the pixel value at (x,y); after obtaining L(x,y,σ) at different scales, the L(x,y,σ) at adjacent scales is differentiated to obtain the information value at the original scale, and then the DOG scale space is constructed. The specific calculation method of the operation is: D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)=L(x,y,kσ)-L(x,y,σ) Among them, k is an integer, and the information at different scales can be obtained by adjusting the k value; b. Extract key points in the image; after constructing the DOG scale space in step a, determine whether each point is a key point; select 8 points in the same scale space and 9 points in adjacent scales for calculation. Since there may be extreme points on the edge, the extreme points on the edge need to be eliminated. First, calculate the Hessian matrix H at the current pixel point: Among them, D xx , D yy Indicates that the image of a certain scale in the DOG pyramid is differentiated twice in the x direction and the y direction, and the derivative is estimated by the neighbor difference of adjacent sampling points; D xy It means that the image of a certain scale in the DOG pyramid is first differentiated in the x direction and then in the y direction. The derivative is also estimated by the neighbor difference of adjacent sampling points. Let α and β be the two eigenvalues ​​of the matrix H, then we get: tr(H)=D xx +D yy =α+β it(H)=D xx +D yy -D xy 2 =α·β When α and β are equal, tr(H) 2 / det(H) takes the minimum; tr(H) 2 / det(H) is used as a criterion for judging edge points. When its value is less than 1.2, it is considered a key point and the position information of the key point is obtained; c. Determine the direction of the key points; for the key points selected in step b, use the neighborhood gradient and direction to represent their directional characteristics. The specific calculation formula is: Among them, m(x, y) is the neighborhood gradient of the key point, and θ(x, y) is the direction of the key point. After calculating the neighborhood gradient at the key point, the histogram is used to count the gradient direction and amplitude corresponding to the pixels in the neighborhood of the feature point. The horizontal axis of the histogram of the gradient direction is the angle of the gradient direction. d. Determine the descriptor of the key point; after obtaining the position and direction of the key point through steps ac, rotate the original coordinate axis according to the direction of the key point, calculate the gradient amplitude and direction of the pixels of the 16x16 window centered on the feature point, and divide the pixels in this area into 16 blocks, each of which is the histogram statistics of 8 directions within the pixel, forming a 128-dimensional feature vector; e. Perform steps a to d on both the previous frame image and the current frame image to obtain descriptor sets T1 and T2 of the key points of the two images respectively, characterize the similarity of the key points in the two images using Euclidean distance, perform association matching based on the similarity, and solve the affine transformation matrix of the two images based on the corresponding points.

6. The interactive multi-model infrared time-sensitive target tracking method integrating image features as claimed in claim 1, characterized in that: The step (4.1) performs registration by extracting SURF features, ORB features, or AKAZE feature image key points of the image.

Images