An infrared target tracking method based on deep feature and key point matching

Abstract

This invention discloses an infrared target tracking method based on depth features and keypoint matching, applied in the field of target technology, comprising the following steps: S1, constructing a feature extraction network model; S2, acquiring an infrared dataset as the network training dataset; S3, calculating depth feature similarity and appearance feature similarity; S4, training the feature extraction network model by minimizing the network's loss function; S5, performing target tracking and detection. This invention can simultaneously extract and fuse depth features from two frames of images, enhancing the utilization rate of image depth features. Furthermore, using depth feature similarity and appearance feature similarity as matching criteria improves the algorithm's resistance to occlusion and interference.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and more specifically to an infrared target tracking method based on depth features and key point matching. Background Technology

[0002] Visual tracking is a hot research topic in the field of computer vision, and it is widely used in civilian applications such as video surveillance and intelligent transportation. In recent years, with the rapid development of computer vision, the overall performance of tracking algorithms has been significantly improved. Meanwhile, because infrared imaging systems utilize the energy emitted by targets for detection and identify targets by acquiring their energy information, they possess both passive and all-weather detection capabilities and are widely used in target perception devices. Tracking targets of interest is a primary task of infrared detection systems; therefore, infrared target tracking is a current research hotspot.

[0003] Chinese patent publication number CN113470071A, entitled "A Two-Layer Cascaded Infrared Pedestrian Tracking Method," describes a method that first extracts the depth features of the target from an infrared image. To reduce target tracking box drift, it constructs normalized joint confidence scores and normalized joint confidence responses, and finally matches the target trajectory using the joint confidence score. However, because this method only performs normalized matching on the depth features during feature extraction, it may experience trajectory loss when the target is occluded, resulting in a low tracking success rate. Furthermore, the process is complex and slow.

[0004] Therefore, how to propose an infrared target tracking method with high success rate and fast tracking speed is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an infrared target tracking method based on depth features and key point matching, which improves tracking speed and reduces target trajectory loss due to occlusion while ensuring high tracking accuracy.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] An infrared target tracking method based on depth features and key point matching includes the following steps:

[0008] S1. Constructing the Feature Extraction Network Model: The constructed feature extraction network model includes a deep feature extraction network and an appearance feature extraction network; the deep feature extraction network is used to extract the feature map of the input target to obtain the depth features of the target; the appearance feature extraction network is used to extract the key point feature information of the input target to obtain the appearance features of the input target.

[0009] S2. Obtain the infrared dataset as the network training dataset;

[0010] S3. Calculate feature similarity: Input the training dataset obtained in S2 into the feature extraction network model built in S1, extract features and calculate feature similarity;

[0011] S4. Train the feature extraction network model by minimizing the network's loss function. When the number of training iterations reaches a set threshold or the value of the loss function reaches a set range, save the model parameters.

[0012] S5. Perform target tracking and detection: Fix the model parameters saved in S4, input the video sequence to be tested into the feature extraction network model with fixed parameters, and obtain the feature matching results.

[0013] Preferably, the deep feature extraction network includes convolutional layers, residual blocks, max pooling layers, normalization layers, average pooling layers, and activation functions; the residual blocks include convolutional layers, normalization layers, and activation functions, wherein the activation function uses the rectified linear function ReLU; all convolutional kernels are n×n in size.

[0014] Preferably, the appearance feature extraction network includes the following five steps: establishing the image scale space, constructing the Gaussian difference pyramid, interpolation lookup, specifying orientation parameters, and generating appearance descriptors.

[0015] Preferably, the infrared dataset in S2 is the infrared image dataset CVC-09.

[0016] Preferably, S3 specifically involves: inputting the first frame image of the video in the infrared dataset into the feature extraction network model to obtain depth features and appearance features, and recording the depth features and appearance features as the depth features and appearance features of the determined target;

[0017] The nth frame image and the (n+1)th frame image are processed by a deep feature extraction network to extract deep features and then fused to obtain fused deep features. The similarity between the fused deep features and the deep features of the determined target is calculated. When the similarity is greater than a preset threshold, the nth frame image matches the determined target image, and the fused deep features are updated to the new determined target deep features. When the similarity is less than the preset threshold, the nth frame image and the determined target image cannot match. The deep features of the (n+1)th frame image and the (n+2)th frame image are fused to obtain new fused deep features, and the similarity between the new fused deep features and the deep features of the determined target is calculated.

[0018] The appearance features of the nth frame image are extracted using an appearance feature extraction network, and the similarity between the appearance features of the nth frame image and the appearance features of the identified target is calculated.

[0019] Preferably, S4 specifically involves: summing the similarity of deep features and the similarity of appearance features according to their weights to obtain a total similarity function; using the total similarity function as the network loss function; training the feature extraction network model by minimizing the network loss function; and saving the model parameters when the number of training iterations reaches a set threshold or the value of the loss function reaches a set range.

[0020] Preferably, after saving the model parameters in S4, the following steps are also included: testing the feature extraction network model using the PTB-TIR dataset.

[0021] As can be seen from the above technical solution, compared with the prior art, the present invention discloses an infrared target tracking method based on depth features and key point matching, which has the following beneficial effects:

[0022] 1. The deep feature extraction network uses a residual block structure. The residual block has many bypass branches that directly connect the input to the subsequent layers, so that the subsequent layers can directly learn the residual. Traditional convolutional layers or fully connected layers will have problems such as information loss and loss when transmitting information. This invention directly transmits the input information to the output through a bypass, protecting the integrity of the information. The entire network only needs to learn the difference between the input and the output, simplifying the learning objective and difficulty.

[0023] 2. The deep feature extraction network uses two frames of images as input and extracts their depth features, which can maximize the use of similar feature information in the images and enhance the anti-occlusion ability of the tracking process.

[0024] 3. The appearance features used in this invention have high tolerance to changes in light, noise, and micro-viewpoint. Furthermore, the appearance features are highly significant, easy to obtain, and have good distinguishability and rich information content, making them suitable for fast and accurate matching in massive feature databases.

[0025] 4. In the feature matching part, this invention combines the evaluation criteria of motion features and appearance features, which is more reasonable than other algorithms that only use motion features or appearance features. The matching results are more accurate and faster. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1 This is a flowchart of the infrared target tracking method based on depth features and key point matching of the present invention;

[0028] Figure 2 This is a diagram of the deep feature extraction network structure of the present invention;

[0029] Figure 3 This is a diagram of the residual block structure in the deep feature extraction network of this invention;

[0030] Figure 4 This is a flowchart of the appearance feature extraction algorithm of the present invention;

[0031] Figure 5 This is a block diagram illustrating the feature similarity calculation principle of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] This invention discloses an infrared target tracking method based on depth features and key point matching, such as... Figure 1 As shown, it includes the following steps:

[0034] S1. Constructing the Feature Extraction Network Model: The constructed feature extraction network model includes a deep feature extraction network and an appearance feature extraction network; the deep feature extraction network is used to extract the feature map of the input target to obtain the depth features of the target; the appearance feature extraction network is used to extract the key point feature information of the input target to obtain the appearance features of the input target.

[0035] S2. Obtain the infrared dataset as the network training dataset;

[0036] S3. Calculate feature similarity: Input the training dataset obtained in S2 into the feature extraction network model built in S1, extract features and calculate feature similarity;

[0037] S4. Model Training Judgment: When the number of training iterations reaches a set threshold or the value of the loss function reaches a set range, stop model training and save the model parameters.

[0038] S5. Perform target tracking and detection: Fix the model parameters saved in S4, input the video sequence to be tested into the feature extraction network model with fixed parameters, and obtain the feature matching results.

[0039] Furthermore, such as Figure 2As shown, the deep feature extraction network includes convolutional layers, residual blocks, max pooling layers, normalization layers, average pooling layers, and activation functions, where the activation function uses the rectified linear function ReLU. In this embodiment, convolutional layers two and three in the deep feature extraction network are both convolutional blocks with a kernel size of 1×1, a stride and padding of 1, and 256 channels. The first two layers of the deep feature extraction network use convolutional layer one with 64 output channels, a stride of 2, and a kernel size of 7×7, and max pooling layer one with a stride of 2 (3×3).

[0040] The residual block consists of convolutional layers, normalization layers, and activation functions, where the activation function uses the rectified linear function ReLU; all convolutional kernels are n×n in size. Figure 3 As shown, when inputting the residual block of the deep feature extraction network, it is divided into two branches. The first branch passes through convolutional layer 1, normalization layer, activation function, convolutional layer 2, and normalization layer. The convolutional kernels of convolutional layer 1 and convolutional layer 2 are both 3×3, the stride of convolutional layer 1 is 2, and the stride of convolutional layer 2 is 1. The second branch is input through a convolutional layer with a 1×1 kernel and a stride of 2 and a normalization layer. The output of the second branch is superimposed with the output of the first branch and passed through the activation function to obtain the total output of the residual block. Residual block 1, residual block 2, residual block 3, residual block 4, residual block 5, and residual block 6 in the deep feature extraction network all have the above residual block structure. The number of channels in convolutional layers 1 and 2 in residual blocks 1 and 2 is 64 each; the number of channels in convolutional layers 1 and 2 in residual blocks 3 and 4 is 128 each; and the number of channels in convolutional layers 1 and 2 in residual blocks 5 and 6 is 256 each.

[0041] Furthermore, such as Figure 4 As shown, the appearance feature extraction network includes the following five steps: establishing the image scale space, constructing a Gaussian difference pyramid, interpolation lookup, specifying orientation parameters, and generating appearance descriptors. Specifically, appearance features contain multiple scale-invariant key points. The first step of the appearance feature extraction algorithm is to establish the image scale space. The scale space L(x,y,σ) of a two-dimensional image can be obtained by convolving a Gaussian function G(x,y,σ) with the original image I(x,y) with a varying scale, as shown below:

[0042] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0043]

[0044] In the formula: (x,y) are pixel coordinates, and σ is the scale space factor;

[0045] To detect stable keypoints, a Gaussian difference pyramid D(x,y,σ) needs to be constructed. Essentially, it represents the difference between two adjacent scale images, calculated using the following formula:

[0046] D(x,y,σ)=L(x,y,kσ)-L(x,y,σ)

[0047] In the formula, k is the ratio of adjacent scale factors.

[0048] The scale space is not continuous between different layers, meaning that the extreme points obtained in the previous step do not represent the true scale and location of the keypoints. To obtain more accurate keypoints, we need to use the Taylor series expansion of the difference-of-Gaussian pyramid function in the scale space for interpolation, while removing keypoints with low contrast. The expansion is shown below:

[0049]

[0050] Where: X = (x, y, σ) T

[0051] After the above two steps, the keypoints still lack orientation. By utilizing the gradient direction characteristics of the neighboring pixels of each keypoint, orientation parameters are assigned to each keypoint, making the feature descriptor rotation-invariant. For each sampling point L(x,y) within the window, the formulas for calculating its gradient value m(x,y) and direction θ(x,y) are as follows:

[0052]

[0053]

[0054] Finally, feature points are described by generating descriptor vectors: First, a square pixel region of 16×16 grid is selected around the feature point; second, a 4×4 grid is divided into sub-regions; finally, the gradient accumulation values ​​in 8 directions (one direction every 45°) are calculated in each sub-region, so that a 4×4×8=128-dimensional feature descriptor F can be generated for each feature point. s (i).

[0055] Furthermore, the infrared dataset in S2 is the CVC-09 infrared image dataset. The CVC-09 infrared dataset includes infrared video sequences captured in nighttime and daytime road scenes. In this embodiment, four video sequences with a total of 427 frames are selected as the training set, and two other video sequences with a total of 263 frames are selected as the test set.

[0056] Furthermore, S3 specifically refers to: such as Figure 5 As shown, the first frame image of the video in the infrared dataset is input into the feature extraction network model to obtain depth features and appearance features. The depth features and appearance features are denoted as the depth features and appearance features of the determined target.

[0057] Images in the dataset are processed by a deep feature extraction network to obtain two-dimensional feature maps. These two-dimensional feature maps are then mapped to a vector space to obtain a 1×256 one-dimensional deep feature F. t (i) Secondly, the scale-invariant key points extracted by the appearance feature extraction algorithm are used with F s (i) represents the i-th feature of the target as F s (i)=[y1(i),y2(i),…y n (i)], where y n (i) represents the appearance feature descriptor of the nth keypoint. Each descriptor consists of a 128-dimensional sub-vector, therefore F s (i) is represented in latitudinal dimension as a 128×n two-dimensional matrix.

[0058] The depth features of the nth frame and the (n+1)th frame are extracted using a depth feature extraction network and then fused to obtain fused depth features. The similarity between the fused depth features and the depth features of the identified target is calculated. If the similarity is greater than a preset threshold, the nth frame matches the identified target image, and the fused depth features are updated to reflect the new depth features of the identified target. If the similarity is less than the preset threshold, the nth frame does not match the identified target image, and the depth features of the (n+1)th and (n+2)th frames are fused to obtain new fused depth features. The similarity between the new fused depth features and the depth features of the identified target is then calculated. The depth feature similarity calculation formula is as follows:

[0059]

[0060] F t (i) represents the deep features obtained after passing through the deep feature extraction network, and F represents the deep features of the identified target.

[0061] The appearance features of the nth frame image are extracted using an appearance feature extraction network, and the similarity between the appearance features of the nth frame image and the appearance features of the identified target is calculated.

[0062] Calculate the appearance feature similarity A based on the appearance feature descriptor. s There are n key points in the i-th appearance feature. That is, F s (i)=[y1(i),y2(i),…y n (i)], each y n (i) represents the appearance feature vector at the nth point. The appearance features corresponding to the target have been determined as follows: The Euclidean distance between the nth keypoint of the detected target i and the nth keypoint of the determined target j is: When the calculated Euclidean distance is less than a set threshold σ, the two keypoints are considered to match. The ratio of the number of matching keypoints to the total number of keypoints is the appearance feature similarity A. s .

[0063] Furthermore, S4 specifically involves: summing the similarity of deep features and the similarity of appearance features according to their weights to obtain a total similarity function; using the total similarity function as the network loss function; training the feature extraction network model by minimizing the network loss function; and saving the model parameters when the number of training iterations reaches a set threshold or the value of the loss function reaches a set range.

[0064] Based on the experimental results, assign a depth feature similarity A. t Similarity to appearance features A s Assigning different weights and summing the results, the total similarity is as follows:

[0065] A f =αA t +(1-α)A s .

[0066] Where α is the weight of similarity.

[0067] The loss function of the algorithm in S4 is defined as:

[0068] T loss =1-A f

[0069] The loss function has a value range between [0, 1]. The closer the value is to 0, the greater the target similarity and the smaller the difference between the two images. In general, the size of the loss function can be reduced directly through network optimization.

[0070] In this embodiment, the training iterations are set to 100, with a total of 5 video sequences in both the training and test sets. Each video sequence input to the network contains 1000 images. The upper limit of the number of images input to the network each time is mainly determined by the performance of the computer's graphics processing unit; generally, the larger the number of images input to the network each time, the better, making the network more stable. The learning rate during training is set to 0.001, which ensures that the network fits quickly without causing overfitting. The advantage of choosing the adaptive moment estimation algorithm as the network parameter optimizer is that after bias correction, the learning rate has a defined range for each iteration, making the parameters relatively stable. The threshold value of the loss function is set to around 0.005; a value less than 0.005 indicates that the training of the entire network is basically complete.

[0071] Furthermore, after saving the model parameters in S4, the feature extraction network model is also tested using the PTB-TIR dataset. The test results are shown in Table 1.

[0072] Table 1

[0073]

[0074] As can be seen from the table, the method proposed in this invention has higher accuracy and success rate than existing methods. These indicators further demonstrate that the method proposed in this invention has better matching accuracy.

[0075] The implementation of convolution, activation functions, fusion operations, and normalization used in this invention are algorithms well known to those skilled in the art, and the specific processes and methods can be found in relevant textbooks or technical documents.

[0076] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0077] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An infrared target tracking method based on depth features and key point matching, characterized in that, Includes the following steps: S1. Constructing a feature extraction network model: The constructed feature extraction network model includes a deep feature extraction network and an appearance feature extraction network; Deep feature extraction networks are used to extract feature maps of input targets to obtain the depth features of the targets; The appearance feature extraction network is used to extract key point feature information of the input target to obtain the appearance features of the input target; S2. Obtain the infrared dataset as the network training dataset; S3. Calculate feature similarity: Input the training dataset obtained in S2 into the feature extraction network model built in S1, extract features and calculate feature similarity; S4. Model Training Judgment: When the number of training iterations reaches a set threshold or the value of the loss function reaches a set range, stop model training and save the model parameters. S5. Perform target tracking and detection: Fix the model parameters saved in S4, input the video sequence to be tested into the feature extraction network model with fixed parameters, and obtain the feature matching results; The deep feature extraction network includes convolutional layers, residual blocks, max pooling layers, normalization layers, average pooling layers, and activation functions. The residual blocks consist of convolutional layers, normalization layers, and activation functions, where the activation function uses the rectified linear function ReLU. All convolutional kernels are n×n in size. S3 specifically involves inputting the first frame of the video from the infrared dataset into the feature extraction network model to obtain depth features and appearance features, and recording the depth features and appearance features as the depth features and appearance features of the determined target. The nth frame image and the (n+1)th frame image are processed by a deep feature extraction network to extract deep features and then fused to obtain fused deep features. The similarity between the fused deep features and the deep features of the determined target is calculated. When the similarity is greater than a preset threshold, the nth frame image matches the determined target image, and the fused deep features are updated to the new determined target deep features. When the similarity is less than the preset threshold, the nth frame image and the determined target image cannot match. The deep features of the (n+1)th frame image and the (n+2)th frame image are fused to obtain new fused deep features, and the similarity between the new fused deep features and the deep features of the determined target is calculated. The appearance features of the nth frame image are extracted through an appearance feature extraction network, and the similarity between the appearance features of the nth frame image and the appearance features of the identified target is calculated. S4 specifically involves: summing the similarity of deep features and the similarity of appearance features according to their weights to obtain a total similarity function; using the total similarity function as the network loss function; training the feature extraction network model by minimizing the network loss function; and saving the model parameters when the number of training iterations reaches a set threshold or the value of the loss function reaches a set range.

2. The infrared target tracking method based on depth features and key point matching according to claim 1, characterized in that, The appearance feature extraction network includes the following five steps: establishing the image scale space, constructing the Gaussian difference pyramid, interpolation lookup, specifying orientation parameters, and generating appearance descriptors.

3. The infrared target tracking method based on depth features and key point matching according to claim 2, characterized in that, The infrared dataset in S2 is the infrared image dataset CVC-09.

4. The infrared target tracking method based on depth features and key point matching according to claim 1, characterized in that, After saving the model parameters in S4, it also includes testing the feature extraction network model using the PTB-TIR dataset.

Images