A visual tracking method based on target dynamic appearance alignment

Abstract

This invention discloses a visual tracking method based on target dynamic appearance alignment. The method utilizes a target appearance alignment module to estimate the morphological alignment parameters between a template image and the image to be processed. Target alignment is decomposed into three sub-modules: feature alignment Jacobian parameter estimation, morphological residual estimation, and inverse iterative combination. Trainable forward propagation networks are designed to perform the corresponding operations for each sub-module. The target template matching module uses a deep neural network to extract the semantic features of the tracked object in the template image and the image to be processed. Then, the alignment parameters iteratively predicted by the target appearance alignment module are used to appropriately adjust the features of the image to be processed, ensuring that the target morphology in the template image and the image to be processed is as consistent as possible. Finally, relevant convolution operations are used to calculate the similarity between the features of the template image and the adjusted features of the image to be processed, outputting a similarity matching heatmap sensitive to the target position. The position with the highest response in the heatmap is the target state position.

Description

Technical Field

[0001] This invention belongs to the field of moving object tracking in computer vision, and relates to a fast twin network tracking method based on target dynamic appearance alignment. Background Technology

[0002] Siamese neural network-based target tracking technology has attracted significant attention from researchers in recent years due to its fast processing speed, high tracking accuracy, and ease of portability and deployment. Siamese neural network tracking algorithms typically describe target tracking as an appearance matching problem in video sequences. They acquire deep features of the target template and search region through two weight-sharing Siamese neural networks, and then calculate the relevant convolutional responses of these feature pairs to obtain an appearance similarity response map between them. Bertinetto et al. first proposed the SiamFC algorithm, which employs an end-to-end fully convolutional architecture. With the help of large-scale training, the algorithm can run in real-time and achieve high accuracy. Building upon this, many researchers have proposed various effective improvements, focusing primarily on learning more semantically rich features, adaptively updating template appearance, and dynamically searching network hyperparameters. Representative methods, such as the SA-Siam network, use two Siamese network branches to extract CNN features with different semantics to suppress background interference. Wang et al. introduced an attention mechanism based on SiamFC, establishing a stronger correlation between template features and the target region to be matched. Zhang et al. proposed a reasonable Siamese network receptive field and detail parameter design strategy, significantly improving the algorithm's performance in deeper network structures. Choi et al. introduced the concept of meta-learning into the Siamese network framework, using the meta-learning module to predict hyperparameters in the network that originally needed to be manually preset, enabling the target tracker to adapt more flexibly to complex changes in target shape.

[0003] Furthermore, inspired by the RPN network structure, which has garnered significant attention in object detection, Siamese networks, which combine RPN with Siamese networks, have gradually become a research hotspot in the field of object tracking in recent years. SiamRPN pioneered the introduction of the RPN structure into the Siamese network framework and treated video object tracking as a single-sample detection task, significantly improving the accuracy of target scale prediction during the tracking phase. DaSiamRPN analyzed and further expanded the training data, designing a new negative sample extraction strategy during the SiamRPN training phase to handle out-of-view and full occlusion issues in video sequences. SiamRPN++, based on SiamRPN, conducted in-depth research on different network structures, focusing on the impact of translation invariance of convolution operations on tracking and localization, achieving excellent performance on public databases. Ocean, based on SiamRPN, combined an anchor-free target state inference strategy, improving the positive-negative sample imbalance problem in previous SiamRPN algorithms. Siam-RCNN analyzed the object detection mechanism in Siamese networks, proposing a more reasonable target window re-detection mechanism, and using dynamic programming to construct the temporal motion trajectory of target objects, thereby suppressing target loss in complex environments.

[0004] However, overall, current tracking algorithms based on Siamese neural networks still rely on training on pre-specified template-search image pairs, resulting in poor tracking performance for target shapes not present in the training set. The dual-path weight-sharing Siamese network for feature extraction lacks an effective temporal alignment / update mechanism for template-search images, leading to highly unstable matching responses in complex scenes. Furthermore, during the pre-training phase, existing methods only select template-backward image pairs within short time segments for network learning, failing to dynamically adjust for cumulative changes in target appearance over long periods, which easily causes tracking drift. Summary of the Invention

[0005] To address the technical problems existing in the prior art, the purpose of this invention is to provide a visual tracking method based on dynamic appearance alignment of targets. This invention combines an adaptive appearance alignment target tracking method, designs a dedicated target appearance alignment network to adjust the geometric pose of the search image, and predicts the target's state position in the video through depth-correlated convolution operations. This method can effectively overcome the problem of the Siamese network matching tracking method's difficulty in handling drastic appearance changes, thus improving the tracker's localization accuracy.

[0006] The technical solution of this invention is as follows:

[0007] A visual tracking method based on target dynamic appearance alignment includes the following steps:

[0008] Training phase:

[0009] Select a set of calibrated target video sequences from the dataset, arbitrarily sample video frames, and determine multiple pairs of images from the video frames to train a feature extraction network module; wherein each pair of images includes a template image T and a search image I; input each pair of images into the feature extraction network module to obtain the template image features corresponding to the template image T and the search image features corresponding to the search image I; then perform a correlation convolution on the extracted template image features and the search image features to obtain the correlation response R;

[0010] The template image T is scaled up to obtain a target region image of the same size as the search image I. The image formed by the target region image and the corresponding search image I is used to train the appearance alignment network module.

[0011] Online tracking phase:

[0012] For a video to be tracked, extract the template image T(z) and the target region image I0(x) from a frame before time t. Then, combine the template image T(z), the target region image I0(x), and the search image I0(x) at time t. t (x) Input the trained feature extraction network module respectively to extract the corresponding image features φ(T(z)), φ(I0(x)) and φ(I t (x)); convert φ(I0(x)), φ(I t (x) is input into the trained appearance alignment network module to predict the search image I. t Transformation parameters between (x) and template image T(z); adjust the search image I at time t according to the transformation parameters. t Image features φ(I) of (x) t (x)), and then perform correlation convolution between the adjusted features and the image features φ(T(z)) to obtain the search image I. t The target position in (x).

[0013] Furthermore, the apparent alignment network module includes a Jacobian layer, a residual layer, and a reverse combination layer; wherein, the Jacobian layer is used to estimate the Jacobian gradient J of the target region image I0(x); the residual layer is used to calculate the search image I t Image feature residuals of I0(x) and the target region image I0(x); the inverse combination layer is used to iteratively calculate the search image I0(x). t The transformation parameter change element Δp between I0(x) and the target region image I0(x) is a differential element.

[0014] Furthermore, the method for predicting the transformation parameters is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] t Image features φ(I) of (x) t(x) is input into the transform network to obtain the initially estimated transform parameters p0; then, the image features φ(I) are estimated using these transform parameters p0. t (x)) Spatial coordinate mapping relationship during adjustment process, based on bilinear interpolation method for image features φ(I) t (x) is interpolated to obtain the adjusted search image features; the k-th adjusted search image features φ(I) are then used to obtain the adjusted search image features. t (w(x;p k Residual iterative estimation is performed using the image features φ(I0(x)) and image features φ(I0(x)) to obtain r. k =φ(I t (w(x;p k )))-φ(I0(x)) and input it into the inverse combination layer; the inverse combination layer is based on the Jacobian gradient J output by the Jacobian layer and the residual r output by the residual layer. k Calculate the parameter adjustment Δp at the k-th iteration. k =H -1 Jr k The transformation parameters of the k-th iteration output are updated. When the set number of iterations K is reached, the transformation parameter p is output. K Where H is the Hessian matrix and w is the image deformation function.

[0015] Furthermore, the search image I is obtained. t The method for determining the target location in (x) is as follows: based on the predicted transformation parameter p K Adjust the search image I at time t t Image features φ(I) of (x) t (x)), and perform correlation convolution between the adjusted features and the image features φ(T(z)) to predict the final matching response R = φ(T(z)) * φ(I). t (w(x;p K The position with the highest response value in R is taken as the optimal matching position. An inverse geometric transformation operation is then performed on this optimal matching position to obtain the search image I. t The target position in (x).

[0016] Furthermore, the Hessian matrix Ω represents the heatmap area covered by the output response R.

[0017] Furthermore, the cost function L used in training the appearance alignment network module pos =||B K -B gt || 2 ; of which B gt It searches for the target location already labeled in image I, B KPredict the position of the output for the appearance alignment network module.

[0018] Furthermore, the loss function used when training the feature extraction network module... Where Ω represents the heatmap region covered by the output response R, X i Let g be the i-th pixel in the heatmap region. i ∈{-1, +1} is a pre-generated X i The actual label corresponding to the location, r i It is X i The confidence score of the response corresponding to the location.

[0019] A server is characterized by comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the methods described above.

[0020] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the above-described method.

[0021] Specifically, this method includes the following steps:

[0022] 1. Target appearance alignment parameter prediction

[0023] The core objective of this invention is to effectively track a target object within any video segment. Given a target template image T, the target search region image in any subsequent frame is I. This invention employs the inverse Lucas alignment method to predict the transformation parameter p between the search region image I and the template image T:

[0024]

[0025] In formula (1), E(p) represents the objective function for optimizing the transformation parameter p, Δp represents the first-order incremental element of the transformation parameter after Taylor expansion, w represents the image deformation function, which can be used to estimate the geometric transformation from image I to image T. x represents each pixel in the image. Taking the derivative of the above formula, we can obtain:

[0026]

[0027] Setting the first derivative to zero, we can obtain a closed-form solution to the above problem:

[0028] Δp=H -1 J[I(w(x;p))-T(x)] (3)

[0029] in H is the Jacobian matrix, and H is the corresponding Hessian matrix.

[0030] This invention introduces the reverse Lucas alignment method into the deep Siamese network framework, utilizing the powerful feature representation capabilities of CNNs to extract semantic features of the target object, and using the ICLK algorithm to align the depth features, thereby obtaining a more accurate target-related response. Since the input template image T and the search image I in each frame are not the same size (the search image I is obtained by cropping a fixed-size expanded region centered on the predicted position of the previous frame in the current frame), this method locally expands the template image T to obtain a target local region image of the same size as the search image I for the alignment operation. Therefore, by rewriting formula (1) in conjunction with depth features, we have:

[0031]

[0032] Among them I t Let represent the search image of frame t, and φ() represent the mapping function of the CNN feature extraction network. I0 represents the image cropped from the template image T of the starting frame of the video, centered on the target object, and aligned with I. t The target local region images are of the same size. The algorithm first searches the image I according to formula (4). t Alignment is performed with the target local region image I0. The closed-form solution of this equation is:

[0033] Δp=H -1 J[φ(I t (w(x;p)))-φ(I0(x))] (5)

[0034] Based on formula (5), this invention designs an appearance alignment network and uses the Lucas alignment algorithm to predict the alignment parameters between the template and search image pairs' depth features. The method decomposes the appearance alignment into three network layers: a Jacobi layer, a residual layer, and a reverse combination layer. Specifically, the Jacobi layer is used to estimate the Jacobi gradient J of the target region image I0 in the initial frame; the residual layer is used to calculate the search image I... t The depth feature residual φ(I) of the target region image I0 in the starting frame t (w(x;p)))-φ(I0(x)); The inverse combination layer is used for iterative computation of I. t The transformation parameter change Δp between the target region image I0 and the initial frame is defined. The design of each network layer is as follows:

[0035] (1) Jacobi layer

[0036] The purpose of the Jacobian layer is to calculate the Jacobian matrix parameters of the depth features φ(I0(x)) corresponding to the target region image I0 in the initial frame. Given the input target region image I0 in the initial frame, a depth feature extraction network (such as ResNet or AlexNet) is used to obtain depth features of size C×H×W, where H and W are the length and width of the depth features, and C is the number of channels of the depth features. The Jacobian parameters can be obtained using the following chain rule:

[0037]

[0038] The Feature Jacobian layer extracts key edge information from the target region image features in the initial frame for subsequent matching comparison. As can be seen from equation (6), this is achieved by calculating the gradient of the depth feature φ(I0(x)). and deformation function The product of these factors yields a feature Jacobian output of size C×(H×W)×4. The corresponding Hessian matrix H can also be obtained using the following formula:

[0039]

[0040] Here, x represents each element in the Jacobian matrix.

[0041] (2) Residual layer

[0042] The residual layer is mainly used for the initial frame target region image features φ(I0(x)) and the pose-adjusted search image features φ(I). t The similarity between (w(x; p))) is determined. It mainly involves three steps: First, the search image I... t Depth features φ(I) t (x) The input is a transform network (comprising a two-layer 3×3 convolution and a fully connected layer) used to predict the transform parameters, which predicts the 4-dimensional transform parameters p. Then, the transform parameters p are used to estimate the search image features φ(I). t (x)) The spatial coordinate mapping relationship is adjusted during the process. Finally, the search image features are interpolated using the bilinear interpolation method to obtain the adjusted search image features. The Lucas algorithm applies each adjusted search image feature φ(I) to the search image features. t (w(x;p k The residuals of the template image features φ(I0(x)) and template image features are repeatedly estimated iteratively. For the k-th iteration, the similarity residual between the adjusted search image features and the target region image features of the starting frame can be obtained by the following residual calculation method:

[0043] r k =φ(I t (w(x;p k )))-φ(I0(x)) (8)

[0044] Here, we use the following linear similarity transformation mapping to predict changes in target pose:

[0045]

[0046] (3) Reverse Combination Layer

[0047] The purpose of the inverse combination layer is to iteratively update and obtain the optimal target apparent pose adjustment parameters. Based on the predictions of the Jacobian layer and the residual layer, we can calculate the parameter adjustment Δp at each update iteration:

[0048] Δp k =H -1 Jr k (10)

[0049] The final target apparent attitude adjustment parameters can be achieved by continuously updating p k We obtain p. For each iteration, p k The adjustment calculation method is as follows:

[0050]

[0051] By setting a fixed number of iterations, attitude adjustment parameters with high accuracy can be obtained.

[0052] 2. Correlation Convolution Matching

[0053] After K iterations, the target appearance alignment module obtains the initial frame target region image I0 and the search image I. t The pose change parameter p predicted in the CNN network K Apply it to the search image feature φ(I) t In (x)), the features of the search image after apparent pose adjustment are φ(I t (w(x;p K For the original target template features φ(T(z)) and the search image features φ(I) t (w(x;p K By performing relevant convolutions, we can obtain target matching outputs that are sensitive to the target's location.

[0054] R=φ(T(z))*φ(I t (w(x;p K (12)

[0055] The position with the highest response value in the output R is considered the position with the highest similarity to the template target in the currently processed video frame. This process is performed on every frame of the video. This allows for the prediction of the target object's trajectory throughout the entire video.

[0056] Compared with existing technologies, this invention achieves significant positive results. When the algorithm employs the SiamRPN framework and the AlexNet feature extraction network, the tracking accuracy of this invention on the OTB dataset is improved from 0.851 to 0.882. Attached Figure Description

[0057] Figure 1 Target tracking flowchart. Detailed Implementation

[0058] This invention addresses the problems of existing Siamese network-based tracking methods, which rely entirely on large-scale data training and lack a template-target image alignment mechanism. It proposes a Siamese network tracking algorithm with adaptive dynamic appearance alignment. The invention and specific embodiments are further described below with reference to the accompanying drawings.

[0059] Specifically, this invention addresses the problem that existing Siamese network tracking models lack an alignment mechanism and cannot dynamically adjust the target matching posture over long time intervals, proposing a dynamic appearance learning-based target tracking method. The specific implementation steps are as follows:

[0060] (1) The training of the network is divided into two stages. Taking the AlexNet network structure as an example, in the first stage of training, a set of video sequences with the vertex coordinates of the target object's bounding box are randomly selected from the dataset. Video frames are sampled arbitrarily, and template target images and pairs of images to be processed are batch-selected (i.e., multiple pairs of images are obtained, each pair including a template image T and a corresponding search image I). These are then standardized to a fixed input size (template image T(z) is 127×127, and search image I(x) is 255×255) as the network input. Then, AlexNet is used to extract the template image features φ(T(z)) of size 7×7×128 and the search image features φ(I(x)) of size 22×22×128. Correlation convolution is directly performed on the two features to obtain the correlation response R (i.e., the feature heatmap output by the network):

[0061] R=φ(T(z))*φ(I(x)) (13)

[0062] In the first stage of training, we use the following loss function to train the feature extraction network.

[0063]

[0064] Where Ω represents the region covered by the output feature heatmap R (Ω is considered the domain, and R is the value on the domain), g i ∈{-1, +1} is a pre-generated X i The actual label corresponding to the location, r i It is a response to X in R iConfidence score of the response at the location.

[0065] (2) Then, in the second stage of training, the parameters of the AlexNet feature extraction network trained in the first stage are fixed. The network parameters in the appearance alignment network module are specifically trained using the ImageNet and Youtube-BB datasets. The target region image is obtained by scale augmentation based on the template image T(z), and feature alignment is performed on the target region image and the search image pair. Figure 1 The process of the apparent alignment network is as follows: assuming the target state position estimated at the (k-1)th iteration is B. k-1 =(x 1,k-1 ,y 1,k-1 ,x 2,k-1 ,y 2,k-1 The alignment parameter output by the apparent alignment network is p. k A new state position estimate B for the target can be obtained by using a simple similarity transformation. k :

[0066] B k =w(B k-1 ,p k (15)

[0067] To train the parameters of the appearance alignment network module, the following cost function L, which is sensitive to the target location, is used. pos Train the network to make the aligned target state position as close as possible to the real target region:

[0068] L pos =||B K -B gt || 2 (16)

[0069] Among them B gt It is the rectangular annotation of the target location already labeled on image I in the training dataset;

[0070] (3) In the final online tracking phase, you can refer to the complete process. Figure 1 The tracking process shown estimates the target position frame by frame. Specifically, firstly, the template image T(z) extracted from the first frame of the video, the target region image I0(x) from the starting frame, and the search image I at time t are used... t Using (x) as input, the deep network features φ(T(z)), φ(I0(x)), and φ(I) are obtained through the feature extraction network. t (x)). Then, φ(I) t (x) is input into the transform network to obtain the initially estimated appearance transform parameters p0. z represents each pixel in the template image.

[0071] Next, based on the aforementioned Lucas alignment algorithm, in the k-th iteration, the Jacobian matrix J and Hessian matrix of the target domain image I0(x) of the initial frame are obtained using formulas (6) and (7), and then the residual r between the search image and the initial feature is calculated. k =φ(I t (w(x;p k )))-φ(I0(x)). Therefore, the apparent transformation parameter increment Δp can be calculated using formula (10). k The new apparent parameters can be obtained through update operations. By repeatedly performing the above iterative process K times, better target tracking alignment parameters can be obtained.

[0072] Finally, based on the predicted transformation parameters p K Adjust the depth features of the search image at time t to predict the final matching response R = φ(T(z)) * φ(I) t (w(x;p K The position with the highest response value in R is considered the optimal matching position. After a simple geometric transformation inverse operation, the target position estimated by the original search image at time t can be obtained.

[0073] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A visual tracking method based on target dynamic appearance alignment, comprising the following steps: Training phase: Select a set of calibrated target video sequences from the dataset, arbitrarily sample video frames, and determine multiple pairs of images from the video frames to train a feature extraction network module; wherein each pair of images includes a template image T and a search image I; input each pair of images into the feature extraction network module respectively to obtain the template image features corresponding to the template image T and the search image features corresponding to the search image I; then perform a correlation convolution on the extracted template image features and the search image features to obtain the correlation response R; The template image T is scaled up to obtain a target region image of the same size as the search image I. The image formed by the target region image and the corresponding search image I is used to train the appearance alignment network module. Online tracking phase: For a video to be tracked, extract the template image T(z) and the target region image I0(x) from a frame before time t. Then, combine the template image T(z), the target region image I0(x), and the search image I0(x) at time t. t (x) Input the trained feature extraction network module respectively to extract the corresponding image features φ(T(z)), φ(I0(x)) and φ(I t (x)); convert φ(I0(x)), φ(I t (x) is input into the trained appearance alignment network module to predict the search image I. t Transformation parameters between (x) and template image T(z); adjust the search image I at time t according to the transformation parameters. t Image features φ(I) of (x) t (x)), and then perform correlation convolution between the adjusted features and the image features φ(T(z)) to obtain the search image I. t The target position in (x).

2. The method according to claim 1, characterized in that, The apparent alignment network module includes a Jacobian layer, a residual layer, and a reverse combination layer; wherein, the Jacobian layer is used to estimate the Jacobian gradient J of the target region image I0(x); the residual layer is used to calculate the search image I t Image feature residuals of I0(x) and the target region image I0(x); the inverse combination layer is used to iteratively calculate the search image I0(x). t The transformation parameter change element Δp between I0(x) and the target region image I0(x) is a differential element.

3. The method according to claim 2, characterized in that, The method for predicting the transformation parameters is as follows: The search image I... t Image features φ(I) of (x) t (x) is input into the transform network to obtain the initially estimated transform parameters p0; then, the image features φ(I) are estimated using these transform parameters p0. t (x)) Spatial coordinate mapping relationship during adjustment process, based on bilinear interpolation method for image features φ(I) t (x) is interpolated to obtain the adjusted search image features; the k-th adjusted search image features φ(I) are then used to obtain the adjusted search image features. t (w(x;p k Residual iterative estimation is performed using the image features φ(I0(x)) and image features φ(I0(x)) to obtain r. k =φ(I t (w(x;p k )))-φ(I0(x)) and input it into the inverse combination layer; The inverse combination layer is based on the Jacobian gradient J output by the Jacobian layer and the residual r output by the residual layer. k Calculate the parameter adjustment Δp at the k-th iteration. k =H -1 Jr k The transformation parameters of the k-th iteration output are updated. When the set number of iterations K is reached, the transformation parameter p is output. K Where H is the Hessian matrix and w is the image deformation function.

4. The method according to claim 3, characterized in that, Get search image I t The method for determining the target location in (x) is as follows: based on the predicted transformation parameter p K Adjust the search image I at time t t Image features φ(I) of (x) t (x)), and perform correlation convolution between the adjusted features and the image features φ(T(z)) to predict the final matching response R = φ(T(z)) * φ(I). t (w(x;p K The position with the highest response value in R is taken as the optimal matching position. An inverse geometric transformation operation is then performed on this optimal matching position to obtain the search image I. t The target position in (x).

5. The method according to claim 3, characterized in that, Hessian matrix Ω represents the heatmap area covered by the output response R.

6. The method according to claim 1, 2, or 3, characterized in that, The cost function L used to train the appearance alignment network module pos =||B K -B gt || 2 ; of which B gt It searches for the target location already labeled in image I, B K Predict the position of the output for the appearance alignment network module.

7. The method according to claim 1, 2, or 3, characterized in that, The loss function used when training the feature extraction network module Where Ω represents the heatmap region covered by the output response R, X i Let g be the i-th pixel in the heatmap region. i ∈{-1, +1} is a pre-generated X i The actual label corresponding to the location, r i It is X i The confidence score of the response corresponding to the location.

8. A server, characterized in that, The method includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing each step of the method of any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images