Feature adaptive selection target tracking method based on twin network

Abstract

The invention belongs to the field of deep learning and single target tracking, and particularly relates to a feature adaptive selection target tracking method based on a twin network. In the method, a template feature filter is provided, and the optimal feature of a current template frame is dynamically selected and constructed by using historical frame information; the search feature divider divides the search features into related features and independent features, the related features are used for calculating cross attention with template features, and the independent features are used for calculating self-attention of search frames; in consideration of the space-time characteristics of tracking, a space-time information transmission encoder is provided, and cross-frame transmission of the space-time characteristics is realized by generating space-time information tokens frame by frame. Under actual challenges such as illumination change, similarity interference and re-appearing after shielding of the target, the tracker provided by the method can still keep enough precision.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and single target tracking, and in particular to a method for feature adaptive selection of target tracking based on a twin network. Background Art

[0002] In the field of computer vision, visual target tracking is the cornerstone of many tasks, such as visual analysis, autonomous driving, and posture estimation. The main process of the target tracking task is to first mark the target location box in the first frame of the video or image sequence, and then in each subsequent frame, find the candidate area that matches the target and determine their coordinates in the image to locate the target. In this way, a continuously changing state related to the initially marked target can be obtained. However, the target tracking process faces many challenges, including illumination changes, target deformation, occlusion, background interference, and appearance changes caused by target motion.

[0003] With the help of deep neural networks, the emergence of Siamese networks has significantly improved the tracking performance. In these methods, robust target representation learning plays an important role in improving tracking performance. However, existing tracking methods always use all template features and search features to interact freely, which is beneficial for target-specific feature extraction. However, most of the background in the search area may lead to inappropriate cross-correlations because highly discriminative feature representations have not been extracted in some shallow features. Therefore, it may not be optimal for the template to always interact with all parts in the search area.

[0004] In the visual tracker of the Siamese network framework, reference features from templates or historical frames are crucial for visual object tracking. However, due to the dynamic characteristics of videos, the reference history information required for different search areas at different times is inconsistent. Therefore, using all features from templates and historical frames may lead to redundancy and impair tracking performance.

[0005] The use of spatiotemporal information is a core issue in the field of target tracking. Both spatial information and temporal information are important for target tracking. The former contains the appearance information of the object used for target positioning, and the latter contains the state changes of the object across frames. How to effectively use the characteristics of spatiotemporal information to achieve cross-frame propagation of target trajectory and motion state information is also a problem worthy of attention.

[0006] Therefore, a new technical solution is urgently needed in the prior art to solve these problems. Summary of the invention

[0007] 1. Technical issues to be resolved

[0008] In order to solve the problem that the existing twin network framework tracker always allows the template to interact with all parts of the search frame features after the twin subnet features are extracted, when the extracted feature representation does not have sufficient discriminability, erroneous noise area interactions will be introduced, affecting the tracking results, the present invention provides a method for target tracking based on feature adaptive selection of twin networks, which solves the problems raised in the above background technology by adaptively selecting the interaction features of the template and the search frame.

[0009] (II) Technical solution

[0010] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0011] A method for target tracking based on feature adaptive selection of Siamese network includes the following four modules: Siamese network, template feature filter, search feature divider, spatiotemporal information transfer encoder, including the following steps, and the following steps are performed in sequence:

[0012] Step 1: Input the target template frame Z and target search frame X t Then, the Siamese Network with ResNet50 as the backbone is used to extract features and obtain the feature f of the template frame Z. z and search frame X t The characteristic f x , while initializing the iterative feature memory, the Siamese Network is divided into two template branches and a search branch with shared parameters for deep feature extraction;

[0013] Step 2: The feature f of the template frame Z obtained in step 1 z , Search frame X t The characteristic f x and the stored features f of the iterative feature memory h After flattening, they are input into the template feature filter together to obtain the current frame template feature R for subsequent feature interaction, and the output template feature R of the template feature filter is fed into the iterative feature memory.

[0014] The template feature filter includes multiple multi-head attention blocks connected in series to select the template features that are most suitable for the current search area from the current frame template features and the historical frame template features. The importance of the template frame features is evaluated by the score of the template feature attention weight, and the top K template features are selected as the most representative current moment features for subsequent interaction with the search area. The iterative process is expressed as:

[0015]

[0016] in, are the first K feature markers selected, and the size of m is N max Through multiple rounds of iterative updates, we can eventually obtain the most representative template markers in the entire video sequence.

[0017] Step 3: The feature f of the template frame Z obtained in step 1 z , Search frame X t The features of are flattened and input into the search feature partitioner together. The search feature partitioner can divide the search feature tokens into E A and E S Two categories, E A The partial search feature token that represents the need to model the relationship with the template region, E S Represents the remaining search tokens. Self-attention mechanism and cross-attention mechanism are used to model the relations of different categories respectively.

[0018] The search feature partitioner first aggregates all template information of the template feature iterator in a maximum pooling manner to generate a perceptual representation of the target, then connects the target perception with the search token, and applies a lightweight multi-layer perceptron to predict the category probability of the search feature token. The overall process is expressed as:

[0019]

[0020] Among them, η i,0 and η i,1 denotes the i-th search feature token assigned to E A and E S The probability of the search area is determined by selecting the category with higher probability to achieve the division of search area features.

[0021] Step 4: R and E divided by step 2 and step 3 A and E S The three types of template frames and search frame features use the separated attention mechanism for feature interaction, so as to adapt the attention calculation operation after adaptive feature division, and initialize the spatiotemporal information token T empty Used for subsequent space-time information transmission.

[0022] The split attention mechanism is improved on the basis of the traditional attention mechanism, which decomposes the original cross attention operation into the self-attention of the template frame R, the template frame R and the search frame E A Cross attention between the search frames E A and E S of self-attention.

[0023] Step 5: Combine the feature interaction results with the initialized spatiotemporal information token T emptyAt the same time, it is input into the spatiotemporal information transfer cross attention encoder to obtain the search area response f r and spatiotemporal information embedding T t .

[0024] The spatiotemporal information transfer encoder receives the aggregation result after the separation attention mechanism and the initialized spatiotemporal information token T empty As input, the encoder autoregressively transforms the spatiotemporal information token from the tth th Frame propagated to (t+1)th th frame, the tth th The space-time token T of the frame t Add to (t+1) th The empty token T of the frame empty In order to update (t+1) th The space-time token T of the frame t+1 , and continue to be used as input for cross-attention calculation and subsequent spatiotemporal information propagation. The overall process of the spatial information transmission encoder can be described as follows:

[0025] T t+1 =T t +T empty

[0026] f t+1 =Attn([MHSA(R),MHCA(Q=R,K=V=E A ),MHSA(S),T t+1 ])

[0027] Step 6: The search area response f obtained in step 5 r In the prediction subnet, the score head is used to determine the confidence of the current tracking result, and the response result with high confidence is selected for template update. The corner point prediction head is used to obtain the expected value of the corner point probability distribution to obtain the tracking box coordinates, thereby completing the accurate tracking of the target.

[0028] Search area response f r Input into a two-branch fully convolutional network (FCN), which consists of L stacked Conv-BN-ReLU layers. The overall process can be expressed as follows:

[0029]

[0030] Where P tl (x,y) and P br (x,y) represents the two probability maps of the upper left corner and lower right corner of the tracked target. Indicates the coordinates of the upper left corner of the tracking box. It represents the coordinate of the lower right corner of the tracking box, H represents the height of the response map, and W represents the width of the response map.

[0031] (III) Beneficial effects

[0032] Compared with the prior art, the present invention provides a method for feature adaptive selection of target tracking based on a twin network, which has the following beneficial effects:

[0033] The present invention proposes a template feature filter, which uses historical frame information to dynamically select and construct the best features of the current template frame; designs a search feature divider to divide the search features into related features and independent features, where the related features are used to calculate the cross-attention with the template features, and the independent features are used to calculate the self-attention of the search frame; considering the spatiotemporal characteristics of tracking, a spatiotemporal information transfer encoder is proposed to achieve cross-frame transfer of spatiotemporal features by generating spatiotemporal information tokens frame by frame. Through the above measures, the robustness of the tracker to cope with complex challenges such as target deformation, partial occlusion, and illumination changes is effectively enhanced, making up for the limitations of the existing twin network framework tracker. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a method for adaptively selecting target tracking based on a twin network of the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of the template feature filter of the present invention;

[0036] Figure 3 This is a schematic diagram of the structure of the search feature divider of the present invention;

[0037] Figure 4 Schematic diagram of the separation attention mechanism used in the present invention;

[0038] Figure 5 A schematic diagram of the structure of a spatiotemporal information transmission encoder used in the present invention;

[0039] Figure 6 The overall block diagram of a method for adaptively selecting target tracking based on a twin network of the present invention is shown in FIG. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Example

[0042] This paper proposes a method for target tracking based on feature adaptive selection of twin networks, which enhances the robustness of the tracker to cope with complex challenges by adaptively selecting the interactive features of the template and the search frame. A template feature filter is proposed to dynamically select and construct the best features of the current template frame using historical frame information; a search feature divider is designed to divide the search features into related features and independent features. The related features are used to calculate the cross attention with the template features, and the independent features are used to calculate the self-attention of the search frame; considering the spatiotemporal characteristics of tracking, a spatiotemporal information transfer encoder is proposed to realize the cross-frame transfer of spatiotemporal features by generating spatiotemporal information tokens frame by frame.

[0043] like Figure 1 As shown, a flowchart of a method for adaptively selecting target tracking based on a twin network according to an embodiment of the present invention is provided. The specific implementation process of the method includes the following steps:

[0044] A. Input target template frame Z and target search frame X t Then, the Siamese network with ResNet50 as the backbone is used to extract features and obtain the feature f of the template frame Z. z and search frame X t The characteristic f x , while initializing the iterative feature memory, the Siamese Network is divided into two template branches and a search branch with shared parameters for deep feature extraction;

[0045] Specifically: the twin network covers two feature extraction subnetworks. Both subnetworks use the weight-shared convolutional neural network ResNet50 as the backbone network. Compared with the traditional convolutional neural network, the twin subnetwork of this method introduces residual blocks and normalization layers, and abandons the random inactivation layer to solve the problems of gradient vanishing and gradient exploding faced by deep networks.

[0046] B. The feature f of the template frame Z obtained in the above steps z , Search frame X t The characteristic f x and the stored features f of the iterative feature memory h After flattening, they are input into the template feature filter together to obtain the current frame template feature R for subsequent feature interaction, and the output template feature R of the template feature filter is fed into the iterative feature memory. Figure 2 Schematic diagram of the structure of the template feature filter in the present invention;

[0047] Specifically, the template feature filter includes multiple multi-head attention blocks connected in series to select the most suitable template features for the current search area from the current frame template features and the historical frame template features. The importance of the template frame features is evaluated by the score of the template feature attention weight, and the top K template features are selected as the most representative current moment features for subsequent interaction with the search area. The iterative process is expressed as:

[0048]

[0049] in, are the first K feature markers selected, and the size of m is N max Through multiple rounds of iterative updates, we can eventually obtain the most representative template markers in the entire video sequence.

[0050] C. The feature f of the template frame Z obtained in the above steps z , Search frame X t The features of are flattened and input into the search feature partitioner together. The search feature partitioner can divide the search feature tokens into E A and E S Two categories, E A The partial search feature token that represents the need to model the relationship with the template region, E S Represents the remaining search tokens. Self-attention mechanism and cross-attention mechanism are used to model the relations of different categories respectively. Figure 3 Schematic diagram of the structure of the search feature divider in the present invention;

[0051] Specifically, the search feature partitioner first aggregates all template information of the template feature iterator using the maximum pooling method to generate the perceptual representation of the target, then connects the target perception with the search token, and applies a lightweight multi-layer perceptron to predict the category probability of the search feature token. The overall process is expressed as:

[0052]

[0053] Among them, η i,0 and η i,1 denotes the i-th search feature token assigned to E A and E S The probability of the search area is determined by selecting the category with higher probability to achieve the division of search area features.

[0054] D. Divide the above steps into R, E A and E S The three types of template frames and search frame features use the separated attention mechanism for feature interaction, so as to adapt the attention calculation operation after adaptive feature division, and initialize the spatiotemporal information token T emptyUsed for subsequent space-time information transmission. Figure 4 Schematic diagram of the separation attention mechanism in the present invention;

[0055] Specifically, the split attention mechanism is improved on the basis of the traditional attention mechanism, which decomposes the original cross attention operation into the self-attention of the template frame R, the template frame R and the search frame E A Cross attention between the search frames E A and E S of self-attention.

[0056] E. Combine the feature interaction results with the initialized spatiotemporal information token T empty At the same time, it is input into the spatiotemporal information transfer cross attention encoder to obtain the search area response f r and spatiotemporal information embedding T t . Figure 5 Schematic diagram of the cross-attention encoder for spatiotemporal information transfer in the present invention;

[0057] Specifically: The spatiotemporal information transfer encoder receives the aggregation result after the separation attention mechanism and the initialized spatiotemporal information token T empty As input, the encoder autoregressively transforms the spatiotemporal information token from the tth th Frame propagated to (t+1)th th frame, the tth th The space-time token T of the frame t Add to (t+1) th The empty token T of the frame empty In order to update (t+1) th The space-time token T of the frame t+1 , and continue to be used as input for cross-attention calculation and subsequent spatiotemporal information propagation. The overall process of the spatial information transmission encoder can be described as follows:

[0058] T t+1 =T t +T empty

[0059] f t+1 =Attn([MHSA(R),MHCA(Q=R,K=V=E A ),MHSA(S),T t+1 ])

[0060] F. The search area response f obtained in the above steps r In the prediction subnet, the score head is used to determine the confidence of the current tracking result, and the response result with high confidence is selected for template update. The corner point prediction head is used to obtain the expected value of the corner point probability distribution to obtain the tracking box coordinates, thereby completing the accurate tracking of the target.

[0061] Specifically: the search area response f r Input into a two-branch fully convolutional network (FCN), which consists of L stacked Conv-BN-ReLU layers. The overall process can be expressed as follows:

[0062]

[0063] Where P tl (x,y) and P br (x,y) represents the two probability maps of the upper left corner and lower right corner of the tracked target. Indicates the coordinates of the upper left corner of the tracking box. It represents the coordinate of the lower right corner of the tracking box, H represents the height of the response map, and W represents the width of the response map.

[0064] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for a person skilled in the art to modify the technical solutions described in the aforementioned embodiments or to replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for target tracking based on feature adaptive selection of Siamese network, the method is based on the following four modules: Siamese network, template feature filter, search feature divider, spatiotemporal information transfer encoder, characterized in that: The process includes the following steps, and is performed in sequence: Step 1: Input the target template frame Z and target search frame X t Then, the Siamese network with ResNet50 as the backbone is used to extract features and obtain the feature f of the template frame Z. z and search frame X t The characteristic f x , while initializing the iterative feature memory, the Siamese Network is divided into two template branches and a search branch with shared parameters for deep feature extraction; Step 2: The feature f of the template frame Z obtained in step 1 z , Search frame X t The characteristic f x and the stored features f of the iterative feature memory h After flattening, they are input into the template feature filter together to obtain the current frame template feature R for subsequent feature interaction, and the output template feature R of the template feature filter is fed into the iterative feature memory. Step 3: The feature f of the template frame Z obtained in step 1 z , Search frame X t The features of are flattened and input into the search feature partitioner together. The search feature partitioner can divide the search feature tokens into E A and E S Two categories, E A The partial search feature token that represents the need to model the relationship with the template region, E S Represents the remaining search tokens. Self-attention mechanism and cross-attention mechanism are used to model the relations of different categories respectively. Step 4: R and E divided by step 2 and step 3 A and E S The three types of template frames and search frame features use the separated attention mechanism for feature interaction, so as to adapt the attention calculation operation after adaptive feature division, and initialize the spatiotemporal information token T empty Used for subsequent space-time information transmission. Step 5: Combine the feature interaction results with the initialized spatiotemporal information token T empty At the same time, it is input into the spatiotemporal information transfer cross attention encoder to obtain the search area response f r and spatiotemporal information embedding T t . Step 6: The search area response f obtained in step 5 r In the prediction subnet, the score head is used to determine the confidence of the current tracking result, and the response result with high confidence is selected for template update. The corner point prediction head is used to obtain the expected value of the corner point probability distribution to obtain the tracking box coordinates, thereby completing the accurate tracking of the target.

2. The method for target tracking based on feature adaptive selection of twin networks according to claim 1 is characterized in that: The twin network in step one covers two feature extraction subnetworks. Both subnetworks use the weight-shared convolutional neural network ResNet50 as the backbone network. Compared with the traditional convolutional neural network, the twin subnetwork of this method introduces residual blocks and normalization layers, and abandons the random inactivation layer to solve the problems of gradient vanishing and gradient exploding faced by deep networks.

3. The method for target tracking based on feature adaptive selection of twin networks according to claim 1 is characterized in that: The template feature filter described in step 2 includes multiple multi-head attention blocks connected in series to select the template features that are most suitable for the current search area from the current frame template features and the historical frame template features. The importance of the template frame features is evaluated by the score of the template feature attention weight, and the top K template features are selected as the most representative current moment features for subsequent interaction with the search area. The iterative process is expressed as: in, are the first K feature markers selected, and the size of m is N max Through multiple rounds of iterative updates, the most representative template markers in the entire video sequence can be obtained.

4. The method for target tracking based on feature adaptive selection of twin networks according to claim 1 is characterized in that: The search feature partitioner described in step 3 first aggregates all template information of the template feature iterator using the maximum pooling method to generate the perceptual representation of the target, then connects the target perception with the search token, and applies a lightweight multi-layer perceptron to predict the category probability of the search feature token. The overall process is expressed as: Among them, η i,0 and η i,1 denotes the i-th search feature token assigned to E A and E S The probability of the search area is determined by selecting the category with higher probability to achieve the division of search area features.

5. The method for target tracking based on feature adaptive selection of twin networks according to claim 1 is characterized by: The split attention mechanism described in step 4 is an improvement on the traditional attention mechanism. It decomposes the original cross attention operation into the self-attention of the template frame R, the template frame R and the search frame E. A Cross attention between the search frames E A and E S of self-attention.

6. The method for target tracking based on feature adaptive selection of twin networks according to claim 1 is characterized by: The spatiotemporal information transfer encoder described in step 5 receives the aggregation result after the separation attention mechanism and the initialized spatiotemporal information token T empty As input, the encoder autoregressively transforms the spatiotemporal information token from the tth th Frame propagated to (t+1)th th frame, the tth th The space-time token T of the frame t Add to (t+1) th The empty token T of the frame empty In order to update (t+1) th The space-time token T of the frame t+1 , and continue to be used as input for cross-attention calculation and subsequent spatiotemporal information propagation. The overall process of the spatial information transmission encoder can be described as follows: T t+1 =T t +T empty f t+1 =Attn([MHSA(R),MHCA(Q=R,K=V=E A ),MHSA(S),T t+1 ]) 7. The method for target tracking based on feature adaptive selection of twin networks according to claim 1 is characterized by: The specific steps for obtaining the tracking frame coordinates described in step 6 are: The search area response f output from step 5 r Input into a two-branch fully convolutional network (FCN), which consists of L stacked Conv-BN-ReLU layers. The overall process can be expressed as follows: Where P tl (x,y) and P br (x,y) represents the two probability maps of the upper left corner and lower right corner of the tracked target. Indicates the coordinates of the upper left corner of the tracking box. It represents the coordinate of the lower right corner of the tracking box, H represents the height of the response map, and W represents the width of the response map.

Images