Target tracking method based on hierarchical transformer

Abstract

The application discloses a target tracking method based on hierarchical Transform, and steps include: 1) taking the trained PVT_v2 improved network as a feature extraction and feature fusion network, and taking an FCN full convolution network as a frame prediction module; 2) obtaining a tracking video, manually selecting a target region in a first frame, determining a search region of a current frame according to the target size of a previous frame, inputting into the feature extraction and feature fusion network, and obtaining attention features from three stages of the feature extraction and feature fusion network; 3) performing weighted fusion to obtain final attention features; 4) inputting the final attention features into the frame prediction module, performing corner point prediction to obtain coordinate positions, and then obtaining a tracking target frame. The method solves the tracking failure problem caused by less information contained in target features and the tracking drift problem caused by target appearance changes.

Description

Technical Field

[0001] This invention belongs to the field of target tracking technology and relates to a target tracking method based on hierarchical Transformer. Background Technology

[0002] In the field of computer vision, object tracking has always been an important topic and research direction. The main content of object tracking is to predict the position, size, and other key information of a target in subsequent frames, given the target information in the initial frame of a video sequence. Object tracking is widely used in video surveillance, intelligent transportation, human-computer interaction, autonomous navigation, and many other fields, and has significant theoretical research and engineering application value.

[0003] Because the attention module in the Transformer architecture is a highly flexible system with dynamic and global modeling capabilities, it can be used for general relational modeling. Furthermore, the Transformer adapts well to large datasets, making almost no assumptions about data structure; its performance improves with increasing data volume. Therefore, Transformer-related application improvements have been introduced into the field of object tracking, resulting in excellent tracking performance.

[0004] Most Transformer-based trackers currently follow a design paradigm that separates feature extraction and feature fusion, which has the following problems: First, the Transformer structure in some trackers is only used for a portion of the operations, and the entire tracking process still depends on other structures, failing to fully utilize the dynamic and global modeling capabilities of the attention module in the Transformer structure. In particular, some trackers still rely on CNN feature extraction, applying Transformer attention operations only in the latter high-level abstract representation space. Second, typical Transformer trackers only utilize the Transformer's attention mechanism to extract attention features, without considering the different information contained in features at different levels. Summary of the Invention

[0005] The purpose of this invention is to provide a target tracking method based on hierarchical Transformer, which solves the problem that existing Transformer target tracking methods fail to fully utilize the structural capabilities of Transformer and do not consider feature information at different levels, leading to tracking failures.

[0006] The technical solution adopted in this invention is a target tracking method based on hierarchical Transformer, implemented according to the following steps:

[0007] Step 1: Use the trained PVT_v2 improved network as the feature extraction and feature fusion network, denoted as... The FCN fully convolutional network is used as the bounding box prediction module, denoted as .

[0008] Step 2: Acquire the tracking video, manually select the target region in the first frame, determine the search region of the current frame based on the target size in the previous frame, and input them together into the feature extraction and feature fusion network.

[0009] The attention features are obtained from stages 1, 2, and 3 of the feature extraction and feature fusion networks, respectively. and

[0010] Step 3, Utilizing attentional characteristics and The information at different scales contained within is weighted and fused to obtain the final attention features.

[0011] Step 4, final attention features In the input bounding box prediction module, corner point prediction is performed to obtain coordinate positions, and the target bounding box is obtained based on the coordinate positions.

[0012] The beneficial effects of this invention are: 1) By using the Transformer structure, the network has global modeling capabilities, effectively solving the tracking failure problem caused by insufficient information contained in the target features. 2) By weighted fusion of features obtained from different network layers, the advantages of features at different scales are fully utilized, making the network more robust to changes in the target's appearance and effectively solving the tracking drift problem caused by changes in the target's appearance. Attached Figure Description

[0013] Figure 1 This is a network structure block diagram of the method of the present invention;

[0014] Figure 2a This is a flowchart of the Transformer EncoderBlock in the Transformer Encoder used in the method of this invention;

[0015] Figure 2b This is a flowchart of the MLP Block in the Transformer Encoder Block used in this invention;

[0016] Figure 3 This is a schematic diagram of the tracking results of an embodiment of the method of the present invention;

[0017] Figure 4 The image shows the frame-by-frame IOU results in the Bird2 video, which is an embodiment of the method of the present invention. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0019] Reference Figure 1 The method of this invention, based on a hierarchical Transformer framework structure, is implemented according to the following steps:

[0020] Step 1: Use the trained PVT_v2 improved network as the feature extraction and feature fusion network, denoted as... The FCN fully convolutional network is used as the bounding box prediction module, denoted as .

[0021] The improved PVT_v2 network is pre-trained on the ImageNet-1k dataset.

[0022] The improved PVT_v2 network consists of three sequentially connected stages. Stage 1 includes convolutional layers, feature concatenation operations, and a Transformer Encoder. Stages 2 and 3 have the same structure, both including a Transformer Encoder. Each of these three stages also incorporates positional encoding, random deactivation layers, and layer normalization operations. (See...) Figure 2a , Figure 2b As shown;

[0023] The FCN fully convolutional network consists of five convolutional layers connected in sequence, with each convolutional layer followed by a batch normalization layer. The first four convolutional layers are followed by the ReLU non-linear activation function.

[0024] The structure and parameters of the FCN fully convolutional network are shown in Table 1:

[0025] Table 1. Parameter table of the bounding box prediction module

[0026]

[0027] Step 2: Acquire the tracking video. Manually select the target region in the first frame. Determine the search region for the current frame based on the target size in the previous frame. Input both of these regions into the feature extraction and feature fusion network. From the perspectives of feature extraction and feature fusion networks respectively The attention features are obtained in stages 1, 2 and 3 respectively. and

[0028] The specific process is as follows:

[0029] 2.1) Acquire the tracking video. Manually select the target area in the first frame of the video. Let (x1, y1) be the coordinates of the center point of the target area in the first frame, and w1 and h1 be the width and height of the target area in the first frame, respectively. Using the center point of the target area in the first frame as the center, extract a square area with a side length of sz1. The formula for calculating sz1 is:

[0030]

[0031] Where p1 = (w1 + h1) / 4 represents the amount of padding in the current frame. If the size of the square region exceeds the image size, the excess part is filled with the average value of the image.

[0032] 2.2) Scale the square region with side length sz1 to a size of 224×224 to obtain the target region Z1 of the first frame;

[0033] 2.3) Starting from the second frame as the subsequent frame, t>1, the tracking target coordinates (x, y, t) of the previous frame are used. t-1 ,y t-1 ) and width and height (w t-1 ,h t-1 Let's extract a square region with side length sx. The formula for calculating sx is as follows:

[0034]

[0035] Where, p t-1 =(w t-1 +h t-1 ) / 4 represents the padding amount for the current frame; if the size of the square region exceeds the image size, the excess portion is padded with the average of the image.

[0036] 2.4) Scale the square region of sx to 224×224 to obtain the search region S of the current frame. t ;

[0037] 2.5) Combine the template target region Z1 of the first frame with the search region S of the current frame. t They are fed together into the feature extraction and feature fusion network. In the process, attention features with dimensions of 64×64×64, 32×32×128, and 16×16×320 were obtained from the three stages. and

[0038] Step 3, Utilizing attentional characteristics and The information at different scales contained within is weighted and fused to obtain the final attention features. The specific process is as follows:

[0039] 3.1) Attention characteristics and Downsampling was performed separately to obtain attention features. and The dimensions are all 16×16×320. The calculation formula is as follows:

[0040]

[0041]

[0042] Downsampling employs max pooling downsampling.

[0043] 3.2) Characteristics of attention and Perform average weighted fusion to obtain the final attention features. The dimensions are 16×16×320. The calculation formula is as follows:

[0044]

[0045] Where α1 = α2 = α3 = 1 / 3.

[0046] Step 4, final attention features Input bounding box prediction module In the process, corner point prediction is performed to obtain coordinate positions, and the bounding box of the tracking target is obtained based on the coordinate positions.

[0047] The specific process is as follows:

[0048] 4.1) Final attention features Input to bounding box prediction module In the process, the target position (x, y, w, h) is obtained, with a size of 4×1, which is the tracking result. The calculation formula is as follows:

[0049]

[0050] Where (x,y) are the center coordinates of the tracked target, and (w,h) are the width and height of the tracked target;

[0051] 4.2) After obtaining the tracking result, the tracking process of the target in the current frame ends. The next frame is taken as the current frame and the process jumps to step 2 to track subsequent frames.

[0052] Experimental verification:

[0053] The method of the present invention shall be implemented in accordance with steps 1 to 4 as described above, and the following points shall be noted:

[0054] In step 1, a pre-trained PVT_v2 improved network on the ImageNet-1k dataset is used as the feature extraction and feature fusion network. Stage 1 requires stacking 3 encoder structures, Stage 2 requires stacking 4 encoder structures, and Stage 3 requires stacking 6 encoder structures; FCN fully convolutional network is selected as the bounding box prediction module.

[0055] The selected tracking video in step 2 is the Bird2 video sequence from the OTB2015 tracking dataset. The size of the search region Z1 obtained from the template region is 224×224×3; the current frame search region S t The size is 224×224×3; attention features with sizes of 64×64×64, 32×32×128 and 16×16×320 are obtained from different stages of feature extraction and feature fusion network.

[0056] In step 3.2), the parameters are selected as α1 = α2 = α3 = 1 / 3.

[0057] Reference Figure 3 This is a tracking result image of a single frame from the Bird2 video in the embodiment of the present invention. The Bird2 video sequence mainly presents four target challenges: OCC (Occlusion), FM (Fast Movement), DEF (Deformation), and IPR (In-Plane Rotation). The tracking target in this video sequence is a bird. The video resolution is high, the lighting conditions are good, and there is no background clutter interference. The entire video sequence consists of 99 frames. The bird is constantly moving, undergoing DEF (Deformation). From frame 3 to frame 20, the tracking target is in an OCC (Occlusion) state; from frame 45 to frame 50 and from frame 87 to frame 97, the tracking target undergoes IPR (In-Plane Rotation); and from frame 52 to frame 80, the tracking target undergoes FM (Fast Movement). In fact, the target environment in this video sequence is not complex, but the target movement and deformation directly change the appearance of the tracked target. The present invention utilizes the global modeling capabilities of the Transformer and different attention features at different scales to track the target under these challenges of appearance changes. The Intersection over Union (IOU) ratio reflects the degree of overlap between the predicted tracking bounding boxes and the manually labeled tracking bounding boxes. It is defined as the ratio of the area of ​​the intersection of the predicted bounding boxes and the area of ​​the union of the predicted bounding boxes and the actual labeled bounding boxes, and is calculated as follows:

[0058]

[0059] Among them, B p B represents the prediction box. gThe bounding box represents the true bounding box, and Area represents the area. A higher IOU value indicates higher accuracy of the tracking algorithm; its value ranges from [0,1]. A success rate threshold T is typically introduced. o The requirement is that tracking is considered successful when the Interchange of Us (IOU) in a given frame is greater than To. Therefore, the success rate metric is defined as IOU > T in a video sequence. o The proportion of images in a video sequence to the total number of images in the video sequence.

[0060] Reference Figure 4 , is the frame-by-frame IOU result curve in the Bird2 video of this embodiment of the invention, taking T o =0.5, the success rate of Bird2 videos is 67.7%.

[0061] To verify the effectiveness of the method of this invention, performance tests were conducted on the method of this invention and two existing methods, SiamFC and MDNet, on the currently publicly available OTB2015 tracking dataset. The specific results are shown in Table 2. As can be seen from Table 2, the success rate of the method of this invention is improved on the OTB2015 tracking dataset, indicating that the method of this invention has higher accuracy and better reliability.

[0062] Table 2. Experimental results of the target tracking method on the OTB2015 dataset.

[0063]

Claims

1. A target tracking method based on hierarchical Transformer, characterized in that, Follow these steps to implement the procedure: Step 1: Use the trained PVT_v2 improved network as the feature extraction and feature fusion network, denoted as... The FCN fully convolutional network is used as the bounding box prediction module, denoted as... The improved PVT_v2 network is pre-trained on the ImageNet-1k dataset. The structure of the PVT_v2 improved network consists of three stages connected in sequence. The internal structure of stage 1 includes convolutional layers, concatenation feature operation and Transformer Encoder. Stages 2 and 3 have the same structure and both include Transformer Encoder. Stages 1, 2 and 3 also include positional encoding, random deactivation layer and layer normalization operation. The FCN fully convolutional network consists of five convolutional layers connected in sequence, with each convolutional layer followed by a batch normalization layer. The first four convolutional layers are followed by the ReLU non-linear activation function. Step 2: Acquire the tracking video, manually select the target region in the first frame, determine the search region of the current frame based on the target size in the previous frame, and input them together into the feature extraction and feature fusion network. The attention features are obtained from stages 1, 2, and 3 of the feature extraction and feature fusion networks, respectively. , and The specific process is as follows: 2.1) Acquire the tracking video, and manually select the target area in the first frame of the video; let The coordinates of the center point of the target area in the first frame. and These are the width and height of the region containing the target in the first frame, respectively; taking the center point of the region containing the target in the first frame as the center, a side length of [value missing] is extracted. a square area The formula for calculation is: (1) in, This indicates the amount of padding in the current frame. If the size of this square region exceeds the image size, the excess portion is filled with the average value of the image. 2.2) Change the side length Scaling the square area to Size, to obtain the target area of ​​the first frame. ; 2.3) Starting from the second frame, as subsequent frames, t >1, using the target coordinates from the previous frame. and width and height Cut off the side with a length of a square area The calculation formula is as follows: (2) in, This indicates the amount of padding in the current frame; if the size of this square region exceeds the image size, the excess portion is padded with the average of the image. 2.4) Scaling the square area to Size, to obtain the search area of ​​the current frame. ; 2.5) The template target area of ​​the first frame and the current frame search area They are fed together into the feature extraction and feature fusion network. In the middle, the dimensions obtained from the three stages are respectively , and attention characteristics , and ; Step 3, Utilizing attentional characteristics , and The information at different scales contained within is weighted and fused to obtain the final attention features. ; Step 4, final attention features In the input bounding box prediction module, corner point prediction is performed to obtain coordinate positions, and the target bounding box is obtained based on the coordinate positions.

2. The target tracking method based on hierarchical Transformer according to claim 1, characterized in that, Step 3, the specific process is as follows: 3.1) Attention characteristics and Downsampling was performed separately to obtain attention features. and All sizes The calculation formula is as follows: (3) (4) Among them, downsampling Max pooling downsampling is used; 3.2) Characteristics of attention , and Perform average weighted fusion to obtain the final attention features. The size is The calculation formula is as follows: (5) in, .

3. The target tracking method based on hierarchical Transformer according to claim 1, characterized in that, Step 4, the specific process is as follows: 4.1) Final attention features Input to bounding box prediction module In the process, the position of the tracking target is obtained. The size is The result is the tracking result, calculated as follows: (6) in, To track the center position coordinates of the target, To track the width and height of the target; 4.2) After obtaining the tracking result, the tracking process of the target in the current frame ends. The next frame is taken as the current frame and the process jumps to step 2 to track subsequent frames.

Images