An infrared image target tracking method based on a kernel correlation filter

Abstract

The application discloses an infrared image target tracking method based on a kernel correlation filter, which comprises the following steps: S1, extracting features of a target region to obtain a plurality of feature points and a descriptor of the target; S2, training and updating filter parameters according to the target features; S3, calculating a response map of a search region by using the parameters in step S2; S4, judging whether the tracking result is valid according to an APCE index of the response map in step S3, if yes, outputting the result and returning to step S1, if not, executing step S5; S5, matching the feature points of a current frame image with the target feature points, if the matching is successful, calculating the target position and size and outputting the same, and then returning to step S1, otherwise, reading a new frame image and executing step S5 again. The application can avoid parameter degradation when the target is seriously occluded or beyond the field of view, and re-track the target after the target appears.

Description

Technical Field

[0001] This invention relates to the field of infrared image processing technology, and in particular to an infrared image target tracking method based on a kernel correlation filter. Background Technology

[0002] Existing infrared target tracking technologies can be divided into three main categories: feature matching-based, correlation filtering-based, and deep learning-based.

[0003] (1) Feature matching

[0004] Feature-based matching methods typically require first extracting target features, then comparing them within the current frame to find the region most similar to the target features as the target location. Depending on the features used, feature-based matching methods can be categorized into three types: contour matching, template matching, and feature point matching.

[0005] 1) Contour matching: Contour matching-based methods first obtain all contours in the image through preprocessing algorithms such as image enhancement, image segmentation, and edge extraction. Then, based on information such as the perimeter, aspect ratio, and compactness of the target contour, the target contour is identified and locked from all contours. Finally, the centroid of the target contour is calculated to obtain the target position.

[0006] 2) Template matching: Template matching involves sliding a target template within a search area to find the position with the highest correlation to the template image, thus obtaining the target position in the current frame. The correlation between the target template and the sub-image of the search area is usually defined as the pixel difference between the sub-image of the search area and the target template.

[0007] 3) Feature point matching. Commonly used feature points include SIFT, SURF, ORB, etc. Based on the feature point descriptor, the same feature point in two adjacent frames can be matched. Thus, the target in the current frame can be matched based on the target's feature points to obtain the target's position. Feature matching-based tracking methods perform well in simple backgrounds, but in complex backgrounds, they are easily interfered with by similar targets, resulting in tracking errors or tracking failures.

[0008] (2) Correlation Filtering

[0009] Correlation filtering originated in signal processing and was later introduced into target tracking. Due to its high real-time performance and reliability, it has gradually become a mainstream method in target tracking. Correlation filtering-based tracking methods train a filter that can distinguish between the target and the background using several image samples. The response map of the search area is then calculated in real time, and the location with the largest response value is the target's position. The most representative algorithm is the KCF algorithm. KCF obtains a large number of virtual samples by "cyclically shifting" the reference samples, greatly expanding the richness of the training samples and effectively improving the filter's discriminative performance. Simultaneously, by utilizing the diagonalization property of the circulant matrix, it effectively reduces the computational complexity of the training process, improving tracking accuracy while maintaining high real-time performance. Furthermore, based on KCF, improvements have been made in overcoming boundary effects, multi-feature fusion, and scale adaptation, leading to algorithms such as SRDCF, DSST, and ECO, further enhancing tracking accuracy and reliability. However, all of the aforementioned tracking methods based on correlation filtering lack the ability to judge the validity of the tracking results. When the target is occluded in extreme cases, the tracking results are obviously invalid, but the algorithm will still use the incorrect results to train and update the filter parameters, causing irreversible degradation of the model parameters. As a result, the algorithm will not be able to track the target again when it reappears.

[0010] (3) Deep Learning

[0011] As deep learning begins to demonstrate its capabilities in the field of computer vision, researchers have gradually introduced deep learning methods into visual object tracking tasks, achieving considerable research results. Currently, deep learning-based tracking algorithms are classified into three types according to network type: classification networks, Siamese networks, and Transformer networks.

[0012] 1) Classification networks: Tracking algorithms based on classification networks treat target tracking as a classification problem between the target and the background. By fine-tuning the convolutional neural network in image classification, a classification network more suitable for target tracking is obtained. Representative algorithms include MDNet and RT-MDNe.

[0013] 2) Siamese Network: Tracking algorithms based on the Siamese network have two branches. One branch extracts features from the target, and the other branch extracts features from the search area. Then, cross-correlation is used to calculate a response map describing the similarity between the two. The peak of the response map indicates the target's position in the image. Representative algorithms include SiamFC++ and SiamRPN++.

[0014] 3) Transformer Networks: Transformer networks consist entirely of attention modules, which can fully consider global information of input features. Therefore, they are widely used in tasks such as speech recognition and machine translation. Since visual object tracking also requires consideration of global image information, Transformer networks have also been widely applied to object tracking tasks in recent years. Representative algorithms include STARK and Trans.T.

[0015] In summary, feature matching algorithms offer no significant advantages in terms of complexity or robustness to interference, and have been gradually replaced by correlation filtering and deep learning algorithms. Correlation filtering algorithms offer advantages in high tracking accuracy and good interference resistance, along with excellent real-time performance. However, they lack mechanisms for judging and correcting the validity of tracking results, making them ill-suited for situations where the target is occluded or out of view. While deep features can better represent the target's appearance model, resulting in higher tracking accuracy and reliability, the complex network models make it difficult to guarantee real-time performance, hindering deployment on some low-performance embedded devices.

[0016] The existing technology has at least the following technical problems:

[0017] Existing technologies struggle to handle complex background interference, target occlusion, or targets outside the field of view. Feature matching algorithms are ill-suited for complex background interference and are only applicable to target tracking tasks with simple backgrounds. Correlation filtering algorithms significantly improve the ability to handle complex background interference, but their model update mechanisms prevent them from handling severely occluded targets. Deep learning algorithms are more robust to interference and occlusion than the aforementioned methods, but their complex network models require more computational resources, making it difficult to meet the high real-time requirements of tracking algorithms and limiting their deployment on embedded devices. Summary of the Invention

[0018] The purpose of this invention is to provide an infrared image target tracking method based on a kernel correlation filter to solve the technical problem that the target cannot be tracked again when it is occluded or reappears after being out of the field of view.

[0019] This invention is achieved using the following technical solution: an infrared image target tracking method based on a kernel correlation filter, comprising the following steps:

[0020] S1: Feature point extraction, performing feature extraction on the target region to obtain multiple feature points and descriptors of the target;

[0021] S2: Training, based on the target features, train and update the filter parameters;

[0022] S3: Detection, using the parameters in step S2 to calculate the response map of the search area;

[0023] S4: Determine whether the tracking result is valid based on the APCE index of the response graph in step S3. If valid, output the result and return to step S1; if invalid, proceed to step S5.

[0024] S5: Re-detection, extract the feature points of the current frame image and match them with the target feature points. If the match is successful, calculate the target position and size and output them, then return to step S1; otherwise, read the new frame image and execute step S5 again.

[0025] Furthermore, step S1 specifically involves: extracting features from the region within the target rectangle to obtain multiple feature points and descriptors of the target, wherein each feature point corresponds to a descriptor.

[0026] Furthermore, step S2 includes the following sub-steps:

[0027] S21: Extract features, extract the features Z of the region of interest roi, which is obtained by expanding the target rectangle;

[0028] S22: Calculate the kernel correlation matrix K ZZ Kernel correlation matrix K ZZ The calculation formula is:

[0029] In the formula, σ = 0.6, The symbol represents the inverse Fourier transform, ^ represents the Fourier transform, * represents the complex conjugate, and Z represents the inverse Fourier transform. c This represents the c-th row of the characteristic matrix Z;

[0030] S23: Calculate the filter parameters;

[0031] S24: Update parameters.

[0032] Furthermore, the formula for calculating the filter parameters is as follows:

[0033]

[0034] In the formula, Let represent the filter parameters for frame t, λ = 0.0001, and y be an m x n matrix. The values ​​of the elements in the i-th row and j-th column of matrix y are calculated using the following formula:

[0035]

[0036]

[0037] Furthermore, the formula for calculating the updated parameters is as follows:

[0038]

[0039] Z = a * Z t +(1-a)*Z.

[0040] Furthermore, step S3 includes the following sub-steps:

[0041] S31: Extract the features of the search region. The search region is exactly the same as the region of interest (ROI) of the previous frame. Use the feature extraction algorithm to extract the features X of the search region.

[0042] S32: Calculate the kernel correlation matrix K XZ Kernel correlation matrix K XZ The calculation formula is:

[0043] In the formula, σ = 0.6, The symbol ^ represents the inverse Fourier transform, ^ represents the Fourier transform, and * represents the complex conjugate. X c Z represents the c-th row of the characteristic matrix X. c This represents the c-th row of the characteristic matrix Z;

[0044] S33: Calculate the response diagram.

[0045] Furthermore, the calculation formula for the response map is as follows:

[0046] In the formula, f(Z) represents the response map, and the value of f(Z) represents the response of the corresponding pixel in the search area. The pixel with the largest response value is the target location (x', y').

[0047] Furthermore, step S4 specifically involves introducing the Average Peak Correlation Energy (APCE) index, calculated using the following formula:

[0048] In the formula, F max and F min These are the maximum and minimum values ​​of the response graph, F. i,j It is the value in the i-th row and j-th column of the response graph; if the APCE of the current frame is lower than the β proportion of the historical average APCE, the tracking result is invalid and step S5 needs to be executed.

[0049] Furthermore, step S5 includes the following sub-steps:

[0050] S51: Use the ORB algorithm to extract feature points and descriptors of the current frame image;

[0051] S52: Use the nearest neighbor matching algorithm to perform a coarse match between the feature points and descriptors of the current frame image and the target feature points and descriptors;

[0052] S53: Use the RANSAC or GMS algorithm to refine the coarse matching results, remove erroneous matching relationships, and obtain the final matching results.

[0053] Furthermore, if the match is successful, the target location is:

[0054]

[0055]

[0056] In the formula, n is the total number of matched feature points, and x i and y i These are the x and y coordinates of the i-th matched feature point, respectively. If the match fails, step S5 is executed until a match is successful.

[0057] The beneficial effects of this invention are as follows: In view of the problem that existing algorithms are difficult to deal with when the target is severely occluded or out of the field of view, this invention introduces the APCE index and re-detection mechanism, which can avoid parameter degradation when the target is severely occluded or out of the field of view, and re-track the target after the target appears.

[0058] Compared with deep learning-based tracking algorithms, this invention has higher real-time performance, achieving a tracking speed of 100fps with only a single-core CPU, and is easier to deploy in practical applications. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the architecture shown in these drawings without creative effort.

[0060] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0062] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0063] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0064] Example 1

[0065] See Figure 1 An infrared image target tracking method based on kernel correlation filter includes the following steps:

[0066] Step 1: Extract feature points within the target rectangle. Feature point extraction algorithms include, but are not limited to, SURF and ORB algorithms.

[0067] Step 2: Extract target features and train filter parameters. When calculating the kernel correlation matrix, σ = 0.6. When calculating the filter parameters, λ = 0.0001. When updating the parameters, the update rate a = 0.012.

[0068] Step 3: Read a new frame of image and use filter parameters to detect the position and size of the target in the current frame.

[0069] Step 4: Determine whether the tracking result is valid based on the APCE index of the response graph. If valid, output the target position and size, represented by a rectangle, and return to Step 1; otherwise, proceed to Step 5.

[0070] Step 5: Extract the feature points of the current frame image and match them with the target feature points. If the match is successful, calculate the target position and size as the tracking result of the current frame. Otherwise, read the new frame image and execute Step 5 again.

[0071] In this embodiment, the target to be tracked needs to be selected with a rectangle in the first frame. The rectangle is represented by (x, y, w, h), where (x, y) are the coordinates of the center point of the rectangle, and w and h are the width and height of the rectangle, respectively. At the same time, while keeping the position of the target rectangle unchanged, its size is increased by 2.5 times to obtain the region of interest (ROI), which is represented by (x, y, 2.5*w, 2.5*h).

[0072] Step 1 specifically involves feature point extraction. Feature extraction is performed on the region within the target rectangle to obtain several feature points obj_keypoints and descriptors obj_descriptor. Each feature point corresponds to a descriptor.

[0073] Step 2 specifically involves training. The purpose of training is to train and update the filter parameters based on the target features, which includes four steps: feature extraction, calculation of the kernel correlation matrix, calculation of filter parameters, and parameter updating.

[0074] Feature extraction: Using the improved HOG feature extraction algorithm in the DPM algorithm, the features Z of roi are extracted. Z is a matrix with 31 rows and m*n columns.

[0075] Calculate the kernel correlation matrix K ZZ :

[0076] In the formula, σ = 0.6, The symbol represents the inverse Fourier transform, ^ represents the Fourier transform, * represents the complex conjugate, and Z represents the inverse Fourier transform. c This represents the c-th row of the characteristic matrix Z.

[0077] Calculate the filter parameters: Assume the filter parameters for frame t are α. t , then α t The calculation formula is as follows:

[0078] In the formula, λ = 0.0001, y is an m x n matrix, and the values ​​of the elements in the i-th row and j-th column of matrix y are calculated using the following formula:

[0079]

[0080] The matrix element values ​​are actually Gaussian distributed with the target as the center.

[0081] Update parameters: The calculation formula is as follows:

[0082]

[0083] Z = a * Z t +(1-a)*Z.

[0084] Step 3 specifically involves detection, which means using the filter parameters calculated in step 2 in the next frame. The response map of the search region is calculated, and the pixel where the peak of the response map is located is the target location. This process includes three steps: extracting features of the search region, calculating the kernel correlation matrix, and calculating the response map.

[0085] Extracting search region features: The search region is exactly the same as the ROI of the previous frame. Using the improved HOG feature extraction algorithm in the DPM algorithm, the feature X of the search region is extracted. X is also a matrix with 31 rows and m*n columns.

[0086] Calculate the kernel correlation matrix K XZThis section calculates the kernel correlation matrix K between the target features and the search region features. XZ The calculation formula is as follows:

[0087] In the formula, σ = 0.6, The symbol ^ represents the inverse Fourier transform, ^ represents the Fourier transform, and * represents the complex conjugate. X c Z represents the c-th row of the characteristic matrix X. c This represents the c-th row of the characteristic matrix Z.

[0088] Calculating the response plot: Assuming the response plot is f(Z), the formula for calculating f(Z) is as follows:

[0089] f(Z) is a matrix with the same width and height as the search region. The value of f(Z) represents the response of the corresponding pixel in the search region. The pixel with the largest response value is the target location (x', y').

[0090] It should be noted that the three steps of extracting search region features, calculating the kernel correlation matrix, and calculating the response map need to be performed once at each of the three scales: scale=1.05, scale=1, and scale=0.95. This means that three response maps can be calculated. By comparing the peak values ​​of the three response maps, the scale corresponding to the response map with the largest peak value represents the scale change of the target compared to the previous frame. Therefore, the target's width w′ and height h′ are:

[0091] w′=w*scale;

[0092] h′ = h * scale;

[0093] At this point, we can obtain the bounding box (x', y', w', h') containing the target.

[0094] Step 4 specifically involves determining whether the target is completely occluded or outside the field of view during actual tracking. In such cases, the tracking result should be considered invalid. To determine whether the target is occluded or outside the field of view, the average peak-to-correlation energy (APCE) index is introduced, and the calculation formula is as follows:

[0095] F max and F min These are the maximum and minimum values ​​of the response graph, F. i,j It is the value in the i-th row and j-th column of the response graph.

[0096] When a target is occluded or outside the field of view, the APCE value decreases significantly. Therefore, by calculating the APCE index, it can be determined whether a target is occluded or outside the field of view. Generally, when the APCE of the current frame is lower than a certain percentage β of the historical average APCE, the target is considered to be severely occluded or outside the field of view.

[0097] APCE < β * average(APCE); If the target is severely occluded or out of the field of view, continue to step 5; otherwise, directly output the tracking result (x', y', w', h') and jump to step 1.

[0098] Step 5 specifically involves: using the ORB algorithm to extract the feature points `cur_keypoints` and descriptors `cur_descriptors` of the current frame image; using the nearest neighbor matching algorithm to perform a coarse match between the current frame's feature points `cur_keypoints` and descriptors `cur_descriptors` and the target feature points `obj_keypoints` and descriptors `obj_descriptors`; then using the RANSAC or GMS algorithm to refine the coarse match results, removing erroneous matches to obtain the final matching result. If the match is successful, the target location is:

[0099]

[0100]

[0101] n is the total number of matched feature points, x i and y i These are the x and y coordinates of the i-th matched feature point, respectively. At this point, the target's dimensions remain unchanged from the previous frame, and the process jumps to step 1; otherwise, it continues executing step 5 until a match is successful.

[0102] Based on the above embodiments, the present invention has at least the following technical effects:

[0103] This invention addresses the problem that existing algorithms struggle to handle targets that are severely occluded or out of the field of view. It introduces the APCE metric and a re-detection mechanism, which can prevent parameter degradation when targets are severely occluded or out of the field of view, and re-track the target after it reappears.

[0104] Compared with deep learning-based tracking algorithms, this invention has higher real-time performance, achieving a tracking speed of 100fps with only a single-core CPU, and is easier to deploy in practical applications.

[0105] This invention can track the position of a specified target in an image in real time and is mainly applied to infrared imaging guidance systems for various air-to-air and ground-to-air missiles.

[0106] For the foregoing embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0107] The above embodiments describe the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the invention should be within the protection scope of the appended claims.

Claims

1. A method for tracking an infrared image target based on a kernel correlation filter, characterized in that, Comprise the following steps: S1: feature point extraction, the target region is extracted, and a plurality of feature points and descriptors of the target are obtained; S2: training, training and updating the filter parameters according to the target features; step S2 comprises the following sub-steps: S21: extracting features, extracting the features Z of the region of interest roi, wherein the region of interest roi is obtained by expanding the target rectangular frame; S22: Calculate the kernel correlation matrix , the kernel correlation matrix The calculation formula is: wherein = 0.6, denotes the inverse Fourier transform, denotes the Fourier transform, denotes the complex conjugate, denotes the eigenmatrix of the first row; S23: calculating filter parameters; S24: updating parameters; The formula for calculating the filter parameters is: ； In the formula, denotes the filter parameter of the t-th frame, = 0.0001, y is an m-row n-column matrix, and the element value of the i-th row and the j-th column of the matrix y is calculated by the following formula: ； ； S3: detection, using the parameters in step S2, the response map of the search area is calculated; step S3 comprises the following sub-steps: S31: extracting search area features, the search area and the region of interest roi of the previous frame are completely the same, using the feature extraction algorithm, the features X of the search area are extracted; S32: Calculate the kernel correlation matrix , the kernel correlation matrix The calculation formula is: wherein , denotes the inverse Fourier transform, denotes the Fourier transform, denotes the complex conjugate, denotes the eigenmatrix of the first row, denotes the eigenmatrix of the first row; S33: calculating the response map; S4: judging, according to the APCE index of the response map in step S3, whether the tracking result is valid, if valid, output the result and return to step S1; if invalid, execute step S5; S5: re-detection, extract the matching between the feature points of the current frame image and the target feature points, if the matching is successful, calculate the target position and size and output, and then return to step S1; otherwise, read a new frame of image and execute step S5 again.

2. The kernel correlation filter based infrared image target tracking method of claim 1, wherein, Step S1 is specifically: extracting features in the region within the target rectangular frame, obtaining a plurality of feature points and descriptors of the target, wherein each feature point corresponds to a descriptor.

3. The kernel correlation filter based infrared image target tracking method of claim 1, wherein, The formula for updating the parameters is: ； 。 4. The kernel correlation filter based infrared image target tracking method of claim 3, wherein, The formula for calculating the response map is: wherein, represents a response map, The value of the response value of the search area corresponding to the pixel point represents the response of the target. The pixel point with the maximum response value is the position (x', y') of the target.

5. The kernel correlation filter based infrared image target tracking method of claim 1, wherein, Step S4 is specifically: introducing the average peak correlation energy index APCE, the formula is: ; where, are the maximum and minimum values of the response map, respectively, is the value of the response map at the i-th row and j-th column; if the APCE of the current frame is lower than the APCE historical average by a proportion, then the tracking result is invalid and step S5 needs to be continued.

6. The kernel correlation filter based infrared image target tracking method of claim 5, wherein, Step S5 comprises the following sub-steps: S51: using ORB algorithm to extract the feature points and descriptors of the current frame image; S52: using the nearest neighbor matching algorithm to perform coarse matching according to the feature points and descriptors of the current frame image and the target feature points and descriptors; S53: using RANSAC algorithm or GMS algorithm to purify the coarse matching result, and eliminating the wrong matching relationship to obtain the final matching result.

7. The kernel correlation filter based infrared image target tracking method of claim 6, wherein, If the matching is successful, the target position is: ； ； In the formula, n is the total number of matched feature points, and are the horizontal and vertical coordinates of the i-th matched feature point, respectively; if the matching is unsuccessful, step S5 is executed until the matching is successful.

Images