A tracking method based on object-level transform neural network

Abstract

A tracking method based on an object-level transform neural network comprises: 1) obtaining a tracking template Z and a dynamic local template Z of a video sequence to be tracked according to a given video sequence and annotation information Box1 of the first frame; dl ; Among them, the dynamic local template Z dl Obtained after processing by the tracking template Z, including 9 local templates; 2) According to the annotation information of the previous frame, obtain the search image S of the video sequence to be tracked i ; 3) Based on step 1) and step 2), the tracking template Z and the dynamic local template Z dl and search images i As the input of the object-level transformation neural network, obtain the search image feature X after feature encoding; 4) Based on step 3), the encoded search image feature X is used as the input of the corner prediction network to obtain the target positioning information Box in the current frame i 5) Update the local template through a dynamic local template update strategy. The present invention designs object-level attention as window attention, thereby effectively avoiding background information from distracting attention.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a tracking method based on an object-level transformation neural network. Background Art

[0002] Visual object tracking is a fundamental task in computer vision. Its goal is to estimate the future state of any object of interest based on initial human annotations. Consequently, visual object tracking has been widely used in autonomous driving, human-computer interaction systems, and intelligent surveillance. While the widespread use of visual transformer neural networks has further improved the tracking performance of visual object tracking, many challenges remain, such as severe deformation, partial occlusion, complex backgrounds, and scale variations.

[0003] Most trackers based on Siamese networks use a correlation method to calculate the affinity between the template and the search area. This calculation process is susceptible to local optimal problems. Recently, trackers based on transformer neural networks have improved the performance of visual tracking by introducing self-attention modules or cross-attention modules of transformer neural networks. MixFormer and SBT further describe a single-stream single-stage framework, which produces a more compact and neat tracking paradigm. However, the global perspective of these paradigms ignores the distinction between foreground and background. A large number of background markers inevitably cause the attention module to lose focus on the discriminative information of the object. OSTrack proposes an early elimination module that eliminates background markers through a priori values of similarity scores obtained early. SparseTT introduces a sparse transformer neural network, and the score of each token of the feature is determined only by the top-K token scores to improve the ability to distinguish between foreground and background.

[0004] Furthermore, changes in target appearance during the test phase are a key factor in tracking drift. Addressing this issue requires an effective online update strategy. However, most update strategies involve several complete historical tracking results (of the same size as the initial template) to obtain temporal information. This creates an inherent contradiction between the richness of temporal information and computational complexity. Summary of the Invention

[0005] The purpose of this application is to provide a tracking method based on an object-level transformation neural network, which introduces an object-level transformation neural network and a dynamic local template update strategy into the existing technical solutions to overcome the technical problems raised in the above-mentioned background technology.

[0006] The present invention comprises the following steps:

[0007] A tracking method based on an object-level transform neural network is provided, wherein a target is tracked by constructing an object-level transform neural network, and the method is characterized by comprising the following steps:

[0008] 1) According to the given video sequence and the annotation information Box1 of the first frame, obtain the tracking template Z and dynamic local template Z of the video sequence to be tracked dl ; Among them, the dynamic local template Z dl Obtained after processing by tracking template Z, it contains 9 local templates;

[0009] 2) According to the annotation information of the previous frame, obtain the search image S of the video sequence to be tracked i ;

[0010] 3) Based on step 1) and step 2), the tracking template Z and the dynamic local template Z dl and search images i As the input of the object-level transformation neural network, obtain the search image feature X after feature encoding;

[0011] 4) Based on step 3), the encoded search image feature X is used as the input of the corner prediction network to obtain the target positioning information Box in the current frame i ;

[0012] 5) Update the local template through the dynamic local template update strategy; according to the Box predicted in step 4) i , use the same method as step 1) to obtain the current dynamic local template Z dli At the same time, according to Box i Get the corresponding Z through ProRoIPooling dli The feature vectors of the 9 local templates are input into the quality assessment network to obtain the reliability score; according to the obtained reliability score and the dynamic local template Z dli , for the historical dynamic local template Z dl Updates are made to ensure the stability and accuracy of subsequent tracking.

[0013] Note that step 1) of the entire tracking process is only implemented in the first frame, and steps 2) to 5) are continuously looped until the tracking is completed.

[0014] In step 1), the tracking template Z and the dynamic local template Z of the video sequence to be tracked are obtained. dl , including the following sub-steps:

[0015] (11) Enlarging the manually marked tracking target bounding box information by a preset first multiple, and cutting out a tracking template Z from the first frame of the video sequence to be used;

[0016] (12) Copy a tracking template Z and segment it into nine local templates Z using a sliding window dl ; The width and height of the sliding window are half of the tracking template size, and the step size is one quarter of the tracking template size.

[0017] In step 2), the search image S of the video sequence to be tracked is obtained. i , the specific method is:

[0018] Box the tracking result information of the previous frame i-1 Zoom in by the preset second magnification and crop the search image from the current frame to be processed.

[0019] In step 3), obtaining the object-level transformation neural network includes the following sub-steps:

[0020] (31)Tracking template Dynamic partial templates With search features Spliced together as input to the object transformation neural network

[0021] (32) The input X is split and deformed and then downsampled through a fully convolutional embedding layer;

[0022] (33) Encoding the downsampled features;

[0023] (34) The object-level transformation neural network adopts a layered multi-stage architecture divided into three stages. Each stage is preceded by a fully convolutional embedding layer to downsample the features. In each stage, the same operation is repeated several times to encode the object discriminative information in the input features. The first and second stages are repeated once, and the third stage is repeated nine times.

[0024] (35) The feature encoding part includes cascaded object-level attention and channel attention; the object-level attention uses an iterative binary clustering algorithm to transform the input After k iterations, it is divided into 2k object clusters. In each iteration, two cluster centers are randomly initialized. and The formula is as follows:

[0025]

[0026] in represents a vector in the input X, m = N / 2. Then the cluster center is moved twice to ensure reliability, using the following operations:

[0027] [c1;c2]=Softmax([c1;c2]X T )X (2)

[0028] in represents the concatenation of the two cluster centers, and T represents the transpose. Then the cosine distance rate r is calculated as follows:

[0029]

[0030] Where D(a, b) represents the cosine distance between a and b. Finally, the cosine distance ratio is sorted to divide the input X into two clusters. The specific formula is as follows:

[0031]

[0032] in and Represents two clusters after division, and argsort represents obtaining the sorted subscripts. After k iterations, we will have the feature sequence Perform spatial attention encoding. The specific method is as follows:

[0033]

[0034] in Depend on After passing through the linear layer, h represents the number of heads in the calculation process, d = C / h, and T represents transposition. Finally, the features after object-level attention encoding are obtained.

[0035] Channel attention encodes the features after object-level attention Encoding, the specific formula is as follows:

[0036]

[0037] in By X SA After the linear layer, h represents the number of heads in the calculation process, d = C / h, and T represents transposition. Finally, the features after channel attention encoding are obtained.

[0038] In step 5), the dynamic local template update strategy includes the following sub-steps:

[0039] (51) According to step 1), a new dynamic local template is obtained;

[0040] (52) Use ProRoIPooling to extract the corresponding nine feature vectors X roi and tracking template features As input to the reliability score calculation process;

[0041] (53) The specific formula of the quality assessment network is as follows:

[0042]

[0043] S roi =σ(W1Rroi ) (8)

[0044] in Represents the reliability score. By X roi After the fully connected layer, By X Z Generated by the fully connected layer. W1 fully connected layer operation, σ represents the activation function ReLU, and T represents transpose.

[0045] (54) In order to avoid introducing too many intermediate local templates of the tracking target, only the local template is updated at the corresponding position, as follows:

[0046]

[0047] Among them, argsort means getting the sorted index, topk means selecting the first k, and Represents the historical and current dynamic partial templates respectively.

[0048] (55) The dynamic local template is updated every 20 frames.

[0049] The advantages of the present invention are: introducing object-level attention to capture subtle differences between similar objects for tracking; first, by clustering similar object vectors, the object-level attention is designed as window attention, which effectively avoids the interference of external background information; second, a new dynamic local template update strategy is designed to introduce timing information for visual target tracking, achieving a trade-off between the richness of temporal information and the amount of computation; the dynamic local template update strategy maintains nine local templates, which are independently sampled from historical tracking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of the tracking method based on object-level transformation neural network in this application.

[0051] Figure 2 Schematic diagram of the object-level transformation neural network structure of this application.

[0052] Figure 3 This is a schematic diagram of the network structure for quality assessment of this application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] In one embodiment, Figure 1As shown in the figure, a tracking method based on object-level transformation neural network is proposed, including:

[0055] Step S1: Obtain a tracking template and a dynamic local template of a video sequence to be tracked.

[0056] For a video sequence requiring target tracking, this embodiment first obtains its tracking template Z. Specifically, the manually marked tracking target bounding box information Box1 is magnified by a preset first multiple, and the tracking template Z is cropped from the first frame image I1 of the video sequence to be tracked.

[0057] For example, the bounding box information Box1 is magnified 2 times, and then the tracking template Z is cropped from the first frame image I1. The bounding box information Box1 includes the coordinates of the upper left corner and the lower right corner of the tracking target bounding box.

[0058] After obtaining the tracking template Z, this instance initializes its dynamic local template Z dl Specifically, a sliding window with a step size of one quarter of Z and a size of one half of Z is designed to segment Z and obtain a dynamic local template Z containing nine local templates. dl .

[0059] For example, the width and height of Z1 are W respectively. Z and H Z , then the step size of the sliding window is and The window size is and Finally, nine local templates are obtained, with a size of and

[0060] Step S2: Obtain the current search image from the video sequence to be tracked.

[0061] In this example, the tracking result information of the previous frame is magnified by a preset second multiple, and the search image is cropped from the current frame to be processed.

[0062] For example, according to the tracking result Box of the previous frame i-1 , for the input image I i , i∈[2,n], according to the bounding box information of the tracking result of the previous frame, zoom in 5 times and then crop the search image S i Similarly, the tracking result Box of the previous frame i-1 , including the coordinates of the upper left corner and lower right corner of the tracking target bounding box.

[0063] Step S3: Input the target template, dynamic local template and search image into the object-level transform neural network to encode the discriminative features of the tracking target and assist in predicting the target location Box i .

[0064] The object-level transformation neural network adopts a layered, multi-stage architecture divided into three stages. Each stage is preceded by a fully convolutional embedding layer to downsample the features. In each stage, the input features are encoded with object attention and channel attention several times. The first and second stages are repeated once, and the third stage is repeated nine times.

[0065] In the first stage, the target template Dynamic partial templates With search features Each is downsampled through a shared fully convolutional embedding layer to obtain tracking template features Dynamic local template features With search features And spliced together as input Encode object information.

[0066] It should be noted that It is usually used to represent the size of the feature, C represents the number of channels, H represents the height, W represents the width, and B represents the batch size of the input. In the process of encoding object information, object-level attention and channel-level attention are used to encode discriminative information.

[0067] The object-level attention encoding process first transforms the input After k iterations, it is divided into 2k object clusters. In each iteration, two cluster centers are randomly initialized. and The formula is as follows:

[0068]

[0069] in represents a vector in the input X, m = N / 2. Then the cluster center is moved twice to ensure reliability, using the following operations:

[0070] [c1;c2]=Softmax([c1;c2]X T )X (2)

[0071] in represents the concatenation of the two cluster centers, and T represents the transpose. Then the cosine distance rate r is calculated as follows:

[0072]

[0073] Where D(a, b) represents the cosine distance between a and b. Finally, the cosine distance ratio is sorted to divide the input X into two clusters. The specific formula is as follows:

[0074]

[0075] in and Represents two clusters after division, and argsort represents obtaining the sorted subscripts. After k iterations, we will have the feature sequence Perform general spatial attention calculations as follows:

[0076]

[0077] in Depend on After passing through the linear layer, h represents the number of heads in the attention calculation, d = C / h, and T represents transposition. Finally, the features after object-level attention encoding are obtained.

[0078] The channel attention encoding process encodes the features after object-level attention encoding Continuing with channel attention encoding, the specific formula is as follows:

[0079]

[0080] in By X SA After passing through the linear layer, h represents the number of heads in the attention calculation, d = C / h, and T represents transposition. Finally, the features after object-level attention encoding are obtained. It should be noted that the feature encoding is repeated once in this stage.

[0081] In the second stage, the features encoded in the first stage are After splitting and deformation, the shared full convolution embedding layer is input again for downsampling to obtain the tracking template features. Dynamic local template features With search features And spliced together as input Perform feature encoding again. Note that this stage only includes one feature encoding calculation.

[0082] In the third stage, the features encoded in the second stage are After splitting and deformation, the shared full convolution embedding layer is input again for downsampling to obtain the tracking template features. Dynamic local template features With search features And spliced together as input Perform feature encoding. It should be noted that the feature encoding calculation is repeated nine times in this stage.

[0083] Step S4: Output the features from step 3 Split into tracking template features Dynamic local template features With search features right Perform dimensionality reduction to predict the corner points of the tracking target and obtain the bounding box information of the tracking target.

[0084] This step will search for features Transformed into a new feature map To facilitate subsequent corner prediction, the process needs to go through three layers of fully connected layers (FCN) and normalize through soft-argmax. F obtains the probability distribution P of the upper left corner and lower right corner of the prediction result through three layers of fully connected layers. tl (x, y) and P br (x, y), and then pass the soft-argmax function to get the coordinates of the upper left corner and lower right corner of the bounding box

[0085]

[0086] in, Indicates: the coordinates of the upper left corner of the predicted bounding box; Indicates: the coordinate of the upper left corner of the predicted bounding box; P tl (x, y) represents the probability distribution of the coordinates of the upper left corner of the predicted bounding box; P br (x, y) represents: the probability distribution of the coordinates of the lower right corner of the predicted bounding box.

[0087] According to the expected value of the probability distribution, the coordinates of the upper left and lower right corners of the prediction result are obtained, and then the bounding box result is obtained, and finally the tracking result Box is obtained. i .

[0088] Step S5: According to the predicted tracking result Box i , using ProRoIPooling from Extract the corresponding nine eigenvectors X roi and tracking template features As input, the reliability score of each local template is obtained by cross attention. At the same time, a new dynamic local template is obtained by cutting from the i-th frame The dynamic local template is updated by comparing the reliability scores.

[0089] This step will be X roi and tracking template features The reliability score of each local template is obtained as input, and the specific formula is as follows:

[0090]

[0091] S roi =σ(W1R roi ) (8)

[0092] in Represents the reliability score. By X roi After the fully connected layer, Depend on Generated through the fully connected layer. W1 fully connected layer operation, σ represents the activation function ReLU, and T represents the transpose. Then, the dynamic local template is updated every 20 frames. To avoid introducing too many intermediate local templates of the tracking target, only the local template at the corresponding position is updated, as follows:

[0093]

[0094] Among them, argsort means getting the sorted index, topk means selecting the first k, and Represents the historical and current dynamic partial templates respectively and

[0095] For Figure 2 The entire deep learning model shown requires two stages of training before it can be used in practice. The training process is as follows:

[0096] In the first stage, the training dataset is preprocessed. Two frames separated by an interval of T are selected from the video sequence. Based on the annotation information, the template image and the search image are cropped to 128×128 and 320×320 sizes. The dynamic local template contains nine local templates of 64×64 size.

[0097] The preprocessed training data set is input into the deep learning model for training. During training, the joint loss is calculated, backpropagation is performed, and the model parameters are updated to complete the training.

[0098] The joint loss is expressed as follows:

[0099]

[0100] Among them, L iou represents the intersection-over-union loss, which is used to measure the distance between the true value and the predicted value. L1 represents the mean absolute error loss, and λ iou 、 Indicates the weights of the corresponding loss function, for example, 5 and 2, b i and Bounding boxes representing the true value and prediction.

[0101] During the training process, the batch size is 80, the learning rate is reduced from 0.0001 to 0.00001, and the AdamW algorithm is used for 500 iterations and the results of each iteration are saved. The last 100 iterations are trained at one tenth of the overall network learning rate. It should be noted that only the parameters of the transformation neural network are fine-tuned at this stage.

[0102] In the second stage, the parameters of the quality assessment network are fine-tuned. The training dataset is preprocessed by selecting two frames separated by T from the video sequence. Based on the annotation information, the template image and search image are cropped to 128×128 and 320×320 sizes. The dynamic local template contains nine local templates of size 64×64.

[0103] The preprocessed training data set is input into the deep learning model for training. During the training, the cross entropy loss is calculated, back propagation is performed, the model parameters are updated, and the training is completed.

[0104] The cross entropy loss is expressed as follows:

[0105]

[0106] Among them, y i Represents the true value, 1 if the tracking target exists, and 0 if it does not exist. i Represents the reliability score of the final prediction.

[0107] During training, the batch size was 256, and the learning rate was decreased from 0.0001 to 0.00001. The AdamW algorithm was used for 40 iterations, with the results of each iteration saved. The last 10 iterations were trained at one-tenth of the overall network learning rate. It is important to note that only the parameters of the quality assessment network were fine-tuned during this phase; the parameters of the transform neural network were frozen throughout.

[0108] This paper utilizes object-level transformation neural networks to encode tracking templates, dynamic local templates, and discriminative target information in search images, clustering similar object tokens and designing object-level attention as window attention, thereby effectively avoiding distraction from background information. Secondly, a new dynamic local template update strategy is designed to introduce temporal information for single-target tracking, achieving a trade-off between rich temporal information and computational burden. The dynamic local template update strategy maintains nine local templates that are independently sampled during testing.

[0109] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A tracking method based on an object-level transform neural network, wherein a target is tracked by constructing an object-level transform neural network, characterized in that: The following steps are involved: 1) According to the given video sequence and the annotation information Box1 of the first frame, obtain the tracking template Z and dynamic local template Z of the video sequence to be tracked dl ; Among them, the dynamic local template Z dl Obtained after processing by tracking template Z, it contains 9 local templates; 2) According to the annotation information of the previous frame, obtain the search image S of the video sequence to be tracked i ; 3) Based on step 1) and step 2), the tracking template Z and the dynamic local template Z dl and search images i As the input of the object-level transformation neural network, obtain the search image feature X after feature encoding; 4) Based on step 3), the encoded search image feature X is used as the input of the corner prediction network to obtain the target positioning information Box in the current frame i ; 5) Update the local template through the dynamic local template update strategy; according to the Box predicted in step 4) i , use the same method as step 1) to obtain the current dynamic local template Z dli At the same time, according to Box i Get the corresponding Z through ProRoIPooling dli The feature vectors of the 9 local templates are input into the quality assessment network to obtain the reliability score; according to the obtained reliability score and the dynamic local template Z dli , for the historical dynamic local template Z dl Replace the low-scoring parts.

2. The tracking method based on object-level transform neural network according to claim 1, characterized in that: Step 1) Obtain the tracking template Z and dynamic local template Z of the video sequence to be tracked dl , including the following sub-steps: (11) Enlarging the manually marked tracking target bounding box information by a preset first multiple, and cutting out a tracking template Z from the first frame of the video sequence to be used; (12) Copy a tracking template Z and segment it into nine local templates Z using a sliding window dl ; The width and height of the sliding window are half of the tracking template size, and the step size is one quarter of the tracking template size.

3. The tracking method based on object-level transform neural network according to claim 1, characterized in that: Step 2) Obtain the search image S of the video sequence to be tracked i , the specific method is: Box the tracking result information of the previous frame i-1 Zoom in by the preset second magnification and crop the search image from the current frame to be processed.

4. The tracking method based on object-level transform neural network according to claim 1, characterized in that: The step 3) of obtaining the object-level transformation neural network includes the following sub-steps: (31)Tracking template Dynamic partial templates With search features Spliced together as input to the object transformation neural network (32) The input X is split and deformed and then downsampled through a fully convolutional embedding layer; (33) Encoding the downsampled features; (34) The object-level transformation neural network adopts a hierarchical multi-stage architecture, which is divided into three stages. Before each stage, the features are downsampled through a fully convolutional embedding layer; In each stage, the same operation is repeated several times to encode the target discriminative information in the input features, where the first and second stages are repeated once, and the third stage is repeated nine times; (35) The feature encoding part includes cascaded object-level attention and channel attention; the object-level attention uses an iterative binary clustering algorithm to transform the input After k iterations, it is divided into 2 k object clusters; at each iteration, two cluster centers are randomly initialized and The formula is as follows: in Represents a vector in the input X, m = N / 2; then the cluster center is moved twice to ensure reliability, using the following operations: [c1;c2]=Softmax([c1;c2]X T )X (2) in represents the concatenation of the two cluster centers, and T represents the transpose; then the cosine distance rate r is calculated as follows: Where D(a,b) represents the cosine distance between a and b. Finally, the cosine distance ratios are sorted to divide the input X into two clusters. The specific formula is as follows: in and Represents two divided clusters, argsort represents obtaining the sorted subscripts; after k iterations, the feature sequence Perform spatial attention encoding. The specific method is as follows: in Depend on After the linear layer, h represents the number of heads in the calculation process, d = C / h, and T represents transposition. Finally, the features after object-level attention encoding are obtained. Channel attention will encode the features Channel encoding is performed, and the specific formula is as follows: in By X SA After the linear layer, h represents the number of heads in the calculation process, d = C / h, and T represents transposition. Finally, the features after channel attention encoding are obtained.

5. The tracking method based on object-level transform neural network according to claim 1, characterized in that: The dynamic local template update strategy described in step 5) includes the following sub-steps: (51) According to step 1), a new dynamic local template is obtained; (52) Use ProRoIPooling to extract the corresponding nine feature vectors X roi and tracking template features As input to the reliability score calculation process; (53) The specific formula of the quality assessment network is as follows: S roi =σ(W1R roi ) (8) in represents the reliability score; By X roi After the fully connected layer, By X Z Generated by the fully connected layer; W1 fully connected layer operation, σ represents the activation function ReLU, T represents transpose; (54) In order to avoid introducing too many intermediate local templates of the tracking target, only the local template is updated at the corresponding position, as follows: Among them, argsort means getting the sorted index, topk means selecting the first k, and Respectively represent the historical and current dynamic local templates; (55) The dynamic local template is updated every 20 frames.

Images