A single-target tracking method based on a dual attention network

Abstract

The application discloses a single-target tracking method based on a double-attention network, which decouples template frame features and search frame features from two dimensions of channels and spaces based on channel attention and spatial attention. Channel attention mainly establishes a long-distance dependency relationship for global context information from the perspective of channels. Spatial attention mainly integrates different scale context from the perspective of space. The two independent attention mechanisms indirectly expand the receptive field and enhance the representation ability of the deep learning network for class features. The template frame features and the search frame features enhanced by the attention mechanism are used for a parallel tracking head network to improve robustness and generate a preliminary position of the target to be tracked. Based on the position, the highest confidence vector is obtained by calculating the depth cross-correlation of the template frame and the search frame in the channel dimension and the spatial dimension, and a more accurate binary segmentation mask is obtained based on the highest confidence vector, so that the final positioning is obtained.

Description

Technical Field

[0001] The present invention relates to an image processing method, in particular to a video target tracking method. Background Art

[0002] Video object tracking is one of the classic problems in the field of computer vision. Currently, this technology has been widely researched and applied in many industries and fields, such as intelligent video surveillance and autonomous driving.

[0003] With the improvement of GPU / CPU and other hardware computing power for image and video data processing and the rapid development and widespread application of the new generation of deep learning technology, target tracking has completely broken through the barriers of traditional image processing methods. Theoretical research has continued to mature and has been widely used in complex application scenarios.

[0004] In the field of intelligent video surveillance, target tracking technology is widely used in monitoring scenarios such as crowded scenes and areas where dangerous behavior may occur. By monitoring targets in real time and obtaining characteristic information about dangerous individuals, this technology provides reliable information and data support for subsequent higher-level scene understanding, behavioral analysis, and dangerous event monitoring. This technology helps provide early warning of abnormal behavior during subsequent dangerous event monitoring and enables real-time tracking of dangerous behavior after it occurs. As a key component of intelligent video surveillance systems, target tracking directly impacts system performance.

[0005] In the field of autonomous driving, target tracking technology is primarily used for in-vehicle camera-based path planning and real-time monitoring of the vehicle's surroundings. By monitoring and predicting the vehicle's surroundings in real time, it can track and analyze unusual vehicles and pedestrians, understand road conditions, and plan routes in a timely manner, enabling accurate predictions and avoiding accidents.

[0006] Despite widespread application, object tracking technology still has limitations when used in complex backgrounds. In complex real-world scenarios, trackers are affected by factors such as scale variations. While significant efforts have been made to address these complex scenarios, addressing these factors remains a hot topic in the field of single-object tracking.

[0007] In recent years, research on deep learning has deepened significantly. This groundbreaking advancement in object tracking technology has transcended the constraints of traditional machine learning methods. Compared to traditional machine learning methods, deep learning-based object tracking technology can better adapt to complex environments, exhibiting greater stability, practicality, and robustness, making it more competitive. Summary of the Invention

[0008] Purpose of the invention: In view of the above-mentioned existing technologies, a single target tracking method based on a dual attention network is proposed to enhance the performance of the single target tracker.

[0009] Technical solution: A single target tracking method based on a dual attention network, including:

[0010] Step 1: Use the first frame of the video as the template frame, select the target to be tracked, crop the image based on the target center, and generate a template frame image;

[0011] Step 2: Use the single target tracking model to track the target in the next frame of the video, including:

[0012] Step 2.1: Use the target in the previous frame as the target to be tracked in the current frame, and crop it based on the target center to obtain the search frame image;

[0013] Step 2.2: The template frame image and the search frame image are first subjected to feature extraction by a deep residual network, and then their respective features are output. The template frame feature is represented as Z, and the search frame feature is represented as X.

[0014] Step 2.3: The template frame features and search frame features are respectively subjected to the parallel channel attention branch and spatial attention branch, and feature enhancement is performed from two independent dimensions, channel and space, and the enhanced features are output; the template frame features after channel attention enhancement are denoted as Z1, and the search frame features are denoted as X1; the template frame features after spatial attention enhancement are denoted as Z2, and the search frame features are denoted as X2;

[0015] Step 2.4: Based on the enhanced features, use the RPN-based parallel tracking head network to output the confidence and the target offset estimate compared to the previous frame; the confidence of the channel attention tracking branch is denoted as C1, and the offset estimate is denoted as O1; the confidence of the spatial attention tracking branch is denoted as C2, and the offset estimate is denoted as O2;

[0016] Step 2.5: Output the target position coordinates of the first stage based on C1, O1, C2, and O2;

[0017] Step 3: Perform channel-depth cross-correlation on Z1 and X1, output the channel confidence map P1, perform spatial depth cross-correlation on Z2 and X2, and output the spatial confidence map P2;

[0018] Step 4: Add the confidence maps P1 and P2, extract the highest confidence vector, perform deconvolution, convolution, and multiple upsampling to generate a binary mask image of the tracked target. Based on the target position coordinates in the first stage, use rotation estimation to generate the optimized target position coordinates;

[0019] Step 5: Loop through steps 2 to 4 to track the target in each frame of the video.

[0020] Furthermore, the step 4 specifically includes:

[0021] Step 4.1: Add the confidence maps P1 and P2 and perform deconvolution on the highest confidence vector to produce the feature map F to increase the resolution and reduce the dimensionality to 32 dimensions.

[0022] Step 4.2: Reduce the dimension of the feature map B with the same resolution as the feature map F in the backbone network feature layer of the deep residual network, and enhance the same-dimensional features of the feature map output in step 4.1;

[0023] Step 4.3: Add the two feature maps output from step 4.2 and interpolate upward to increase the resolution;

[0024] Step 4.4: Reduce the dimension of the feature map output in step 4.3;

[0025] Step 4.5: Repeat steps 4.2 to 4.4 until a feature map with a dimension of 1 is generated, i.e., a binary mask image of the tracked target.

[0026] Step 4.6: Based on the target position coordinates in the first stage, use the rotation estimation to generate the optimized target position coordinates.

[0027] Furthermore, the training process of the single target tracking model includes the following steps:

[0028] Step S1: Annotate each frame of the video to be tracked, randomly select two frames, one of which is used as a template frame and the other as a search frame, and crop the template frame and search frame images based on the annotated target center, keeping the resolution of the search frame image twice that of the template frame image;

[0029] Step S2: Perform feature enhancement processing on the image obtained in step S1;

[0030] Step S3: Obtain an anchor frame based on the target center of the image obtained in step S2, and calculate the size of the overlapping area with the target center. The anchor frame is divided into positive samples and negative samples based on the overlapping area as classification labels; wherein, samples with an overlapping area greater than a first threshold are positive samples, and samples less than a second threshold are negative samples. Samples between the first and second thresholds are ignored, and the first threshold is greater than the second threshold, and both thresholds are between 0 and 1;

[0031] Step S4: Calculate the target offset of the corresponding positive and negative samples based on the target center and the anchor box size as the regression label;

[0032] Step S5: Repeat steps S1 and S4 several times to form a training data set.

[0033] Furthermore, the first threshold is 0.6 and the second threshold is 0.3.

[0034] Beneficial effects: The present invention separates feature information based on channel attention and spatial attention, calculates the similarity between features, and establishes long-distance dependencies between features to enhance the expressiveness of features, thereby enabling the single target tracker to better cope with changes in background and target scale.

[0035] Specifically, the channel attention branch primarily establishes dependencies between categories, suppressing interference from similar objects and thereby enhancing the response of specific categories during tracking. The spatial attention branch, based on the contextual relationships between objects, clusters features between identical objects and increases the distance between different objects, thereby enhancing the spatial representation of features. Corresponding to these two different attention mechanisms, the present invention is based on a tracking head network, making fuller use of the separation characteristics of channels and space.

[0036] At the same time, based on the results of the first stage, the present invention also makes more detailed optimizations, namely channel-based depth cross-correlation operations and space-based depth cross-correlation operations, to more fully utilize the features of two independent dimensions. At the same time, with the help of the target position fine-tuning in step 4, the tracking target is binary segmented and the target position is fine-tuned to achieve better positioning accuracy.

[0037] Compared with the existing technology, this tracking method can greatly reduce manual intervention, achieve end-to-end tracking and optimization, and effectively improve the performance of single target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is the overall structure diagram of this method;

[0039] Figure 2 Schematic diagram of the channel attention network;

[0040] Figure 3 Schematic diagram of spatial attention network;

[0041] Figure 4 Fine-tune the network diagram for the target location;

[0042] Figure 5 A visualization diagram of the confidence fusion of the channel attention tracking branch and the spatial attention tracking branch;

[0043] Figure 6 The comparison chart of tracking results with other methods. DETAILED DESCRIPTION

[0044] The present invention will be further explained below with reference to the accompanying drawings.

[0045] like Figure 1 As shown, a single target tracking method based on dual attention network includes:

[0046] Step 1: Use the first frame of the video as the template frame, select the target to be tracked, crop the image based on the target center, and generate a template frame image.

[0047] Step 2: Use the single target tracking model to track the target in the next frame of the video. Specifically include:

[0048] Step 2.1: Use the target in the previous frame as the target to be tracked in the current frame, and crop it based on the target center to obtain the search frame image.

[0049] Step 2.2: The template frame image and the search frame image are first subjected to feature extraction by the deep residual network and then their respective features are output. The template frame feature is represented as Z and the search frame feature is represented as X.

[0050] Step 2.3: The template frame features and search frame features are respectively subjected to the parallel channel attention branch and spatial attention branch, and feature enhancement is performed from two independent dimensions, channel and space, and the enhanced features are output; the template frame features after channel attention enhancement are represented as Z1, and the search frame features are represented as X1; the template frame features after spatial attention enhancement are represented as Z2, and the search frame features are represented as X2.

[0051] The channel attention branch used in this invention is mainly based on the Non-local structure, such as Figure 2 As shown. During calculation, spatial attention mainly calculates the correlation between spaces. With the help of the pyramid architecture, the features of the same space are grid-divided into four scales: 1×1, 2×2, 3×3 and 6×6. Figure 3 The following figure shows the attention calculation method of one level of the pyramid. After completing the calculation of the four scales, the output features of the four scales are concatenated with the original features and the dimensionality is reduced.

[0052] In this embodiment, the feature dimension of the deep residual network output is 256, and the feature dimension of the channel attention branch and the spatial attention branch output is also 256.

[0053] Step 2.4: Based on the enhanced features, use the RPN-based parallel tracking head network to output the confidence and the target offset estimate compared to the previous frame; the confidence of the channel attention tracking branch is denoted as C1, and the offset estimate is denoted as O1; the confidence of the spatial attention tracking branch is denoted as C2, and the offset estimate is denoted as O2.

[0054] Step 2.5: Output the target position coordinates of the first stage based on C1, O1, C2, and O2.

[0055] Step 3: Perform channel-depth cross-correlation on Z1 and X1, output the channel confidence map P1, perform spatial depth cross-correlation on Z2 and X2, and output the spatial confidence map P2.

[0056] Step 4: Add the confidence maps P1 and P2, extract the highest confidence vector, perform deconvolution, convolution and multiple upsampling to generate a binary mask image of the tracked target, and use rotation estimation to generate the optimized target position coordinates based on the target position coordinates in the first stage.

[0057] In this paper, based on the enhanced channel attention features and spatial attention features, the target position fine-tuning network is used to perform channel depth cross-correlation and spatial depth cross-correlation to generate confidence maps of the same resolution and the same dimension, respectively. Figure 4 As shown in Figure 2, the two confidence maps are directly added together to extract the highest confidence vector, which is an embedding vector with a dimension of 256 for processing.

[0058] Step 4 specifically includes:

[0059] Step 4.1: Add the confidence maps P1 and P2 and perform deconvolution on the highest confidence vector to produce the feature map F to increase the resolution and reduce the dimensionality to 32 dimensions.

[0060] Step 4.2: Reduce the dimension of the feature map B with the same resolution as the feature map F in the backbone network feature layer of the deep residual network, and enhance the same-dimensional features of the feature map F output in step 4.1;

[0061] Step 4.3: Add the feature maps F and B output in step 4.2 and interpolate upward to increase the resolution;

[0062] Step 4.4: Reduce the dimension of the feature map output in step 4.3;

[0063] Step 4.5: Repeat steps 4.2 to 4.4 until a feature map with a dimension of 1 is generated, i.e., a binary mask image of the tracked target.

[0064] Step 4.6: Based on the target position coordinates in the first stage, use the rotation estimation to generate the optimized target position coordinates.

[0065] Step 5: Loop through steps 2 to 4 to track the target in each frame of the video.

[0066] In the method of the present invention, the training process of the single target tracking model includes the following steps:

[0067] Step S1: Annotate each frame of the video to be tracked, randomly select two frames, one of which is used as a template frame and the other as a search frame, and crop the template frame image and the search frame image according to the center of their respective annotated targets, keeping the resolution of the search frame image twice that of the template frame image.

[0068] Step S2: Perform feature enhancement on the image obtained in step S1, including translation and rotation processing.

[0069] Step S3: Obtain an anchor frame based on the target center of the image obtained in step S2, and calculate the overlap area with the target center. Based on the overlap area, the anchor frame is divided into positive samples and negative samples, which serve as classification labels. Samples with an overlap area greater than a first threshold are considered positive, while samples with an overlap area less than a second threshold are considered negative. Samples between the first and second thresholds are ignored. The first threshold is greater than the second threshold, and both thresholds are between 0 and 1. In this embodiment, the first threshold is 0.6 and the second threshold is 0.3, which serve as classification labels.

[0070] Step S4: Calculate the target offset of the corresponding positive and negative samples based on the target center and the anchor box size as the regression label.

[0071] Step S5: Repeat steps S1 and S4 several times to form a training data set.

[0072] A neural network was built and the training data set obtained above was used for model training to obtain a single target tracking model. After optimizing the model parameters, a high-accuracy model was obtained.

[0073] The present invention decouples the template frame features and the search frame features from the two dimensions of channel and space based on channel attention and spatial attention. Channel attention mainly establishes long-distance dependency relationships for global context information from the perspective of the channel; spatial attention mainly integrates contexts of different scales from the perspective of space. The two independent attention mechanisms indirectly expand the receptive field and enhance the deep learning network's ability to represent category features. A parallel tracking head network is used to improve the robustness of the template frame features and search frame features enhanced by the attention mechanism, and to generate a preliminary position of the target to be tracked. Based on this position, the highest confidence vector is obtained by calculating the deep cross-correlation of the template frame and the search frame in the channel dimension and the spatial dimension respectively, and based on this, a more accurate binary segmentation mask is obtained to obtain the final positioning.

[0074] like Figure 5 As shown, compared with other methods, the present invention can better suppress the interference of spatial background information, and also better suppress the interference of similar objects. Figure 5The top column is the original image, which contains objects that are similar to the target to be tracked or have large deformation scales. The bottom column is the confidence map. From the perspective of the confidence map, it can better suppress the deformation of similar objects and the target itself, and better locate the object.

[0075] like Figure 6 As shown, compared with other methods, the present invention can better adapt to the impact of scale changes and obtain the target position more accurately. The present invention compares three first-class methods and finds that the present invention method can better locate the position of the object and better fit the shape of the object.

[0076] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A single target tracking method based on dual attention network, characterized in that: include: Step 1: Use the first frame of the video as the template frame, select the target to be tracked, crop the image based on the target center, and generate a template frame image; Step 2: Use the single target tracking model to track the target in the next frame of the video, including: Step 2.1: Use the target in the previous frame as the target to be tracked in the current frame, and crop it based on the target center to obtain the search frame image; Step 2.2: The template frame image and the search frame image are first subjected to feature extraction by a deep residual network, and then their respective features are output. The template frame feature is represented as Z, and the search frame feature is represented as X. Step 2.3: The template frame features and search frame features are respectively subjected to the parallel channel attention branch and spatial attention branch, and feature enhancement is performed from two independent dimensions, channel and space, and the enhanced features are output; the template frame features after channel attention enhancement are denoted as Z1, and the search frame features are denoted as X1; the template frame features after spatial attention enhancement are denoted as Z2, and the search frame features are denoted as X2; Step 2.4: Based on the enhanced features, use the RPN-based parallel tracking head network to output the confidence and the target offset estimate compared to the previous frame; the confidence of the channel attention tracking branch is denoted as C1, and the offset estimate is denoted as O1; the confidence of the spatial attention tracking branch is denoted as C2, and the offset estimate is denoted as O2; Step 2.5: Output the target position coordinates of the first stage based on C1, O1, C2, and O2; Step 3: Perform channel-depth cross-correlation on Z1 and X1, output the channel confidence map P1, perform spatial depth cross-correlation on Z2 and X2, and output the spatial confidence map P2; Step 4: Add the confidence maps P1 and P2, extract the highest confidence vector, perform deconvolution, convolution, and multiple upsampling to generate a binary mask image of the tracked target. Based on the target position coordinates in the first stage, use rotation estimation to generate the optimized target position coordinates; Step 5: Loop through steps 2 to 4 to track the target in each frame of the video.

2. The single target tracking method based on dual twin attention network according to claim 1 is characterized in that The step 4 specifically includes: Step 4.1: Add the confidence maps P1 and P2 and perform deconvolution on the highest confidence vector to produce the feature map F to increase the resolution and reduce the dimensionality to 32 dimensions. Step 4.2: Reduce the dimension of the feature map B with the same resolution as the feature map F in the backbone network feature layer of the deep residual network, and enhance the same-dimensional features of the feature map output in step 4.1; Step 4.3: Add the two feature maps output from step 4.2 and interpolate upward to increase the resolution; Step 4.4: Reduce the dimension of the feature map output in step 4.3; Step 4.5: Repeat steps 4.2 to 4.4 until a feature map with a dimension of 1 is generated, i.e., a binary mask image of the tracked target. Step 4.6: Based on the target position coordinates in the first stage, use the rotation estimation to generate the optimized target position coordinates.

3. The single target tracking method based on dual twin attention network according to claim 1 is characterized in that The training process of the single target tracking model includes the following steps: Step S1: Annotate each frame of the video to be tracked, randomly select two frames, one of which is used as a template frame and the other as a search frame, and crop the template frame and search frame images based on the annotated target center, keeping the resolution of the search frame image twice that of the template frame image; Step S2: Perform feature enhancement processing on the image obtained in step S1; Step S3: Obtain an anchor frame based on the target center of the image obtained in step S2, and calculate the size of the overlapping area with the target center. The anchor frame is divided into positive samples and negative samples based on the overlapping area as classification labels; wherein, samples with an overlapping area greater than a first threshold are positive samples, and samples less than a second threshold are negative samples. Samples between the first and second thresholds are ignored, and the first threshold is greater than the second threshold, and both thresholds are between 0 and 1; Step S4: Calculate the target offset of the corresponding positive and negative samples based on the target center and the anchor box size as the regression label; Step S5: Repeat steps S1 and S4 several times to form a training data set.

4. The single target tracking method based on dual twin attention network according to claim 3 is characterized in that The first threshold is 0.6, and the second threshold is 0.3.

Images