Siamese network video single target tracking method based on feature fusion

Abstract

The application relates to a Siamese network video single-target tracking method based on feature fusion, and particularly relates to a Siamese network monitoring video single-target tracking method based on feature fusion. In order to solve the problems that the Siamese network single-target tracking algorithm has low tracking capability when facing complex environments and obvious background interference near a tracked target, cannot accurately track the target, and the output tracking area is not accurate when tracking some specific targets, a model is trained by using a template region image set and a search region image set, feature maps of the template image and the search image are respectively output, the model sequentially comprises a ResNet-50 network based on a mixed attention mechanism and a twin feature fusion network, the feature map of the template image and the feature map of the search image are input into an RPN network for similarity comparison, a prediction region with the highest similarity to the template image in the search image is output, and single-target tracking is realized. The application belongs to the field of target tracking.

Description

Technical Field

[0001] This invention relates to a target tracking method, specifically a single-target tracking method for Siamese network surveillance video based on feature fusion, belonging to the field of target tracking. Background Technology

[0002] Single-target tracking technology plays a crucial role in military fields such as long-range strikes and enemy detection, as well as in everyday life areas such as finding lost children and tracking illegally parked vehicles. The existing Siamese network single-target tracking algorithm has been widely researched and applied. The Siamese network single-target tracking algorithm uses a convolutional neural network as its backbone for feature extraction, and directly performs target classification and regression processing on the extracted feature maps. However, because the semantic information contained in the image features extracted by each convolutional layer of the convolutional neural network is not the same, the image features learned by the Siamese network single-target tracking algorithm often lack contextual information about the tracked image. This prevents the algorithm from fully utilizing the appearance and semantic information of the tracked target, resulting in slightly insufficient tracking ability and lower tracking accuracy in complex environments such as changing lighting, scene occlusion, and camera shake.

[0003] Meanwhile, in pursuit of higher real-time tracking speed, the Siamese network single-object tracking algorithm typically uses offline-trained models. Offline training often employs more general feature extraction networks, leading to less precise tracking regions when tracking certain specific targets. Furthermore, when significant background interference exists near the tracked target in the image, the features extracted by the single-object tracking algorithm are more likely to concentrate on these background objects, causing the Siamese network single-object tracking algorithm to fail to accurately track the target. Summary of the Invention

[0004] To address the issues of low tracking capability, inaccurate target tracking, and insufficient accuracy in output tracking area when the Siamese network single-target tracking algorithm faces complex environments and significant background interference near the tracked target, as well as the problem of insufficient accuracy in output tracking area when tracking certain specific targets, this invention proposes a Siamese network video single-target tracking method based on feature fusion.

[0005] The technical solution adopted in this invention is:

[0006] It includes the following steps:

[0007] S1. Obtain a surveillance video, extract each frame of the video, label all extracted images, preprocess all labeled images, crop each frame into a pair of fixed-size images, divide each cropped frame into a template image and a search image, assemble all template images into a template region image set, and assemble all search images into a search region image set.

[0008] S2. Model Construction: The model consists of a ResNet-50 network based on a hybrid attention mechanism and a Siamese feature fusion network. Template region image sets and search region image sets are input into the model for training. Feature maps of the template and search images are output separately. This process continues until the upper limit of the number of iterations is met, resulting in a trained model. The specific process is as follows:

[0009] S21. In the template branch, the template region image set is input into the model. Each template image is processed sequentially by the ResNet-50 network based on the hybrid attention mechanism and the Siamese feature fusion network to output a feature map.

[0010] S22. In the search branch, the next frame search image corresponding to each template image is input into the model, and then processed by the ResNet-50 network based on the hybrid attention mechanism and the Siamese feature fusion network in sequence to output the feature map.

[0011] S23. Input the feature maps output by S21 and S22 into the RPN network for similarity comparison, and output the predicted region in the search image that has the highest similarity to the template image.

[0012] S3. Obtain the surveillance video to be tracked. Select a target in the first frame of the surveillance video for tracking. Based on the first frame of the selected target and the second frame of the surveillance video, use the model trained in S2 to output the predicted region of the selected target in the second frame of the first frame, and obtain the position of the target in the second frame of the video.

[0013] Then, based on the second frame image containing the target and the third frame image of the surveillance video, the model trained in S2 is used to output the predicted region of the target in the second frame image in the third frame image, and so on, to achieve single target tracking.

[0014] Furthermore, in S1, the size of the template image is 127×127, and the size of the search image is 255×255.

[0015] Furthermore, the ResNet-50 network based on the hybrid attention mechanism in S2 sequentially includes convolutional block 1, pooling layer, convolutional block 2, convolutional block 3, convolutional block 4, and convolutional block 5. Convolutional block 1 includes one convolutional layer, convolutional block 2 sequentially includes three convolutional layers, convolutional block 3 sequentially includes four convolutional layers, convolutional block 4 sequentially includes six convolutional layers, and convolutional block 5 sequentially includes three convolutional layers.

[0016] Channel attention and spatial attention mechanisms are sequentially set between the last two convolutional layers of convolutional block 2, convolutional block 3, convolutional block 4, and convolutional block 5.

[0017] The outputs of convolutional blocks 3, 4, and 5 are all used as the final outputs of the ResNet-50 network based on the hybrid attention mechanism. The outputs of different convolutional blocks correspond to different scales.

[0018] Furthermore, the twin feature fusion network in S2 sequentially includes convolutional block a, convolutional block b, convolutional block c, and a pyramid feature extraction network. Convolutional block a includes a 1×1 convolutional layer, convolutional block b includes a 1×1 convolutional layer and a bilinear interpolation layer, convolutional block c includes a 1×1 convolutional layer and a bilinear interpolation layer, and the pyramid feature extraction network has three layers, each of which includes a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, and a transposed convolutional layer.

[0019] Furthermore, the RPN network in S23 includes three RPN modules, each of which includes an anchor generator, a convolutional feature extractor, and a candidate box regressor.

[0020] Furthermore, the specific process of S21 is as follows:

[0021] S211. Each template image is input into the ResNet-50 network based on the hybrid attention mechanism. After each template image is processed by convolution block 1 and pooling layer, it is input into convolution block 2. Since convolution block 2 is equipped with channel attention mechanism and spatial attention mechanism, the input of convolution block 2 is processed by the first two convolution layers of convolution block 2 to obtain the corresponding feature vector X. Let the total number of feature channels of feature vector X be C.

[0022] The feature vector X is input into the channel attention mechanism, and the channel-weighted feature M of the output feature vector X is generated. c ;

[0023] Channel weighted feature M c In the input spatial attention mechanism, the output attention map A s ;

[0024] Attention map A sThe result of convolutional block 2 is output from the third convolutional layer that takes convolutional block 2 as input.

[0025] The output of convolution block 2 is sequentially input into convolution block 3, convolution block 4, and convolution block 5. The processing steps of convolution block 2 are repeated in convolution block 3, convolution block 4, and convolution block 5. Since the output of different convolution blocks corresponds to different scales, three features of different scales are obtained.

[0026] S212. Input the three scale features of each template image into convolutional blocks a, b, and c of the Siamese feature fusion network, respectively. Use the 1×1 convolutional layer in each convolutional block to reduce the dimensionality of the input scale features to obtain the three dimensionality-reduced features.

[0027] Each feature is treated as a feature map. The feature map output by convolution block 3 in the ResNet-50 network based on the hybrid attention mechanism is used as the basic feature map. If the size of the dimensionality-reduced feature map obtained by convolution block b or convolution block c of the Siamese feature fusion network is smaller than the size of the basic feature map, the dimensionality-reduced feature map obtained by convolution block b or convolution block c is input into the corresponding bilinear interpolation layer. Bilinear interpolation is used to adjust the size of the feature map to the size of the basic feature map. The adjusted feature map is used as the final output of the corresponding convolution block. The feature maps output by convolution block a, convolution block b and convolution block c are concatenated to obtain a concatenated feature map.

[0028] The spliced ​​feature maps obtained above are input into each layer of the pyramid feature extraction network. In each layer, the spliced ​​feature maps are processed by a 1×1 convolutional layer, a 3×3 convolutional layer and a 1×1 convolutional layer to generate low-resolution feature maps. The tensors at the corresponding positions of the low-resolution feature maps and the spliced ​​feature maps are added to obtain the multi-scale feature maps output by the current layer, thus obtaining three multi-scale feature maps.

[0029] S213. For each template image, the channel size of the three multi-scale feature maps is adjusted using deconvolution to obtain three new feature maps after adjustment.

[0030] Furthermore, the channel weighting feature M c :

[0031] M c =sigmoid(W(BN(X)))

[0032] Where BN represents batch normalization, X represents the feature vector, and W represents the attention weight.

[0033] Furthermore, the attention map A s :

[0034] As =M c ·(σ(f max (Q(f avg (M c ))))))

[0035] Where Q represents the convolution operation, f avg and f max These represent the average and maximum values ​​of the weighted feature maps along the channel dimension, respectively, where σ is the sigmoid function.

[0036] Furthermore, the specific process of S22 is as follows:

[0037] S221. Each search image is input into the ResNet-50 network based on the hybrid attention mechanism. After each search image is processed by convolution block 1 and pooling layer, it is input into convolution block 2. Since convolution block 2 has channel attention mechanism and spatial attention mechanism, the input of convolution block 2 is processed by the first two convolution layers of convolution block 2 to obtain the corresponding feature vector X. Let the total number of feature channels of feature vector X be C.

[0038] The feature vector X is input into the channel attention mechanism, and the channel-weighted feature M of the output feature vector X is generated. c ;

[0039] Channel weighted feature M c In the input spatial attention mechanism, the output attention map A s ;

[0040] Attention map A s The result of convolutional block 2 is output from the third convolutional layer that takes convolutional block 2 as input.

[0041] The output of convolution block 2 is sequentially input into convolution block 3, convolution block 4, and convolution block 5. The processing steps of convolution block 2 are repeated in convolution block 3, convolution block 4, and convolution block 5. Since the output of different convolution blocks corresponds to different scales, three features of different scales are obtained.

[0042] S222. Input the three scale features of each search image into convolutional blocks a, b, and c of the Siamese feature fusion network, respectively. Use the 1×1 convolutional layer in each convolutional block to reduce the dimensionality of the input scale features to obtain the three dimensionality-reduced features.

[0043] Each feature is treated as a feature map. The feature map output by convolution block 3 in the ResNet-50 network based on the hybrid attention mechanism is used as the basic feature map. If the size of the dimensionality-reduced feature map obtained by convolution block b or convolution block c of the Siamese feature fusion network is smaller than the size of the basic feature map, the dimensionality-reduced feature map obtained by convolution block b or convolution block c is input into the corresponding bilinear interpolation layer. Bilinear interpolation is used to adjust the size of the feature map to the size of the basic feature map. The adjusted feature map is used as the final output of the corresponding convolution block. The feature maps output by convolution block a, convolution block b and convolution block c are concatenated to obtain a concatenated feature map.

[0044] The spliced ​​feature maps obtained above are input into each layer of the pyramid feature extraction network. In each layer, the spliced ​​feature maps are processed by a 1×1 convolutional layer, a 3×3 convolutional layer and a 1×1 convolutional layer to generate low-resolution feature maps. The tensors at the corresponding positions of the low-resolution feature maps and the spliced ​​feature maps are added to obtain the multi-scale feature maps output by the current layer, thus obtaining three multi-scale feature maps.

[0045] S223. For each search image, the channel size of the three multi-scale feature maps is adjusted using deconvolution to obtain three new feature maps after adjustment.

[0046] Furthermore, the specific process of S23 is as follows:

[0047] The three adjusted new feature maps obtained in S213 and S222 are input into the RPN network. For each RPN module, one adjusted new feature map obtained in S213 and one adjusted new feature map obtained in S222 are input. The similarity of the two input new feature maps is compared, and feature maps with target candidate boxes are output respectively.

[0048] Since the three RPN modules have the same output size, the outputs of each RPN module are weighted and summed to obtain the predicted region in the search image that has the highest similarity to the template image.

[0049] Beneficial effects:

[0050] This invention uses a ResNet-50 network with a hybrid attention mechanism (channel attention and spatial attention) to extract features from the target image, obtaining image features with different levels and semantics. A Siamese feature fusion module is used to combine and stitch these features together. Then, a pyramid feature extraction network is used to generate deep image features with multi-layered semantic information. This allows for more efficient use of the appearance and semantic information of the tracked target, reducing the impact of the tracking environment on the tracking results. It also exhibits robustness to changes in the target's appearance and scale, enabling more effective differentiation between the tracked target and background information in complex environments, when there is significant background interference near the tracked target, and when tracking specific targets. This not only yields a precise tracking region but also improves the target tracking capability and accuracy. Experiments based on the embodiments show that this invention improves tracking accuracy, anti-interference ability, and robustness compared to traditional single-target tracking algorithms based on Siamese networks. Attached Figure Description

[0051] Figure 1 This is a flowchart of the present invention;

[0052] Figure 2 This is a schematic diagram of the channel attention mechanism;

[0053] Figure 3 This is a schematic diagram of the spatial attention mechanism;

[0054] Figure 4 This is a schematic diagram of the feature fusion module; Detailed Implementation

[0055] Specific implementation method one: Combining Figures 1-4 This embodiment describes a Siamese network-based video single-target tracking method based on feature fusion, which includes the following steps:

[0056] S1. Acquire a surveillance video, extract each frame of the video, label all extracted images, preprocess all labeled images, crop each frame into a pair of fixed-size images, and divide each cropped frame into a template image and a search image. Combine all template images into a template region image set and combine all search images into a search region image set.

[0057] The template image is 127×127 pixels, and the search image is 255×255 pixels. Both the template and search images initially have 3 channels. Since Siamese network single-object tracking calculates the similarity between the tracked target in the first frame and subsequent frames, and then selects the region with the highest similarity in each subsequent frame as the prediction region, this invention optimizes the model's performance during the training process by continuously comparing the similarity between images in the template region image set and the search region image set.

[0058] S2, such as Figure 1 As shown, a model is constructed, which sequentially includes a ResNet-50 network based on a hybrid attention mechanism and a Siamese feature fusion network. Template region image sets and search region image sets are input into the model for training, and feature maps of the template and search images are output respectively. This process continues until the upper limit of the number of iterations is met, resulting in a trained model. The specific process is as follows:

[0059] A ResNet-50 network based on a hybrid attention mechanism is used for feature extraction. The ResNet-50 network based on the hybrid attention mechanism consists of convolutional block 1, pooling layer, convolutional block 2, convolutional block 3, convolutional block 4, and convolutional block 5. Convolutional block 1 includes one convolutional layer. Convolutional block 2 consists of three convolutional layers: Conv2_1, Conv2_2, and Conv2_3. Channel attention and spatial attention mechanisms are applied sequentially between the last two convolutional layers (Conv2_2 and Conv2_3) of convolutional block 2. Convolutional block 3 consists of four convolutional layers: Conv3_1, Conv3_2, Conv3_3, and Conv3_4. Channel attention and spatial attention mechanisms are applied sequentially between the last two convolutional layers (Conv3_3 and Conv3_4) of convolutional block 3. Convolutional block 4 consists of six convolutional layers: Conv4_1, Conv4_2, Conv4_3, Conv4_4, Conv4_5, and Conv4_6. Channel attention and spatial attention mechanisms are sequentially applied between the last two convolutional layers (Conv4_5 and Conv4_6) of convolutional block 4. Convolutional block 5 consists of three convolutional layers: Conv5_1, Conv5_2, and Conv5_3. Channel attention and spatial attention mechanisms are sequentially applied between the last two convolutional layers (Conv5_2 and Conv5_3) of convolutional block 5.

[0060] The outputs of convolutional blocks 3, 4, and 5 are all used as the final outputs of the ResNet-50 network based on the hybrid attention mechanism, with different scales corresponding to the outputs of different convolutional blocks.

[0061] This invention utilizes a ResNet-50 network based on a hybrid attention mechanism for image feature extraction. By reducing the effective spacing between convolutional blocks 4 and 5 to 8 pixels and using dilated convolutions at the output layers of blocks 4 and 5 to expand the receptive field of feature extraction, the receptive fields of each convolutional layer in the ResNet-50 network become distinct. Shallow convolutional layers (blocks 1 and 2) primarily extract low-level information such as texture and shape, playing a crucial role in target location. However, these shallow features lack semantic information and are difficult to use for target recognition. Deep convolutional layers (blocks 3, 4, and 5) extract features with rich semantic information and may be more helpful in complex environments, such as target occlusion, lighting changes, and rapid motion.

[0062] This invention introduces a hybrid attention mechanism in the ResNet-50 network. When extracting features from a target image, the network uses channel attention and spatial attention mechanisms to enable the neural network to focus its limited computational resources on feature channels with richer channel information and feature image regions with richer semantic information during the learning process of feature images, thereby reducing the negative impact of other unimportant feature channels on target tracking.

[0063] The twin feature fusion network consists of convolutional block a, convolutional block b, convolutional block c, and a pyramid feature extraction network. Convolutional block a consists of a 1×1 convolutional layer, convolutional block b consists of a 1×1 convolutional layer and a bilinear interpolation layer, and convolutional block c consists of a 1×1 convolutional layer and a bilinear interpolation layer. The pyramid feature extraction network has three layers, each consisting of a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, and a transposed convolutional layer.

[0064] S21. In the template branch, the template region image set is input into the model. Each template image is processed sequentially by a ResNet-50 network based on a hybrid attention mechanism and a Siamese feature fusion network, outputting a feature map. The specific process is as follows:

[0065] S211, such as Figure 1 As shown, each template image is input into a ResNet-50 network based on a hybrid attention mechanism. After processing by convolutional block 1 and pooling layers, each template image is input into convolutional block 2. Since convolutional block 2 has both channel attention and spatial attention mechanisms, the input to convolutional block 2 is processed by the first two convolutional layers of convolutional block 2 to obtain the corresponding feature vector X. Let the total number of feature channels of feature vector X be C. In this invention, the initial number of channels is set to 3, and the number of channels can be changed subsequently according to the dimensionality adjustment operations of different convolutional layers. Figure 1 and Figure 2As shown, the feature vector X is input into the channel attention mechanism, and a batch normalization (BN) operation is performed on the feature vector X. Then, the scaling factor z for each channel c (c∈C) is calculated. c :

[0066]

[0067]

[0068] Where, γ c x represents the scaling factor of channel c during BN processing. c This represents the value of the current channel c in the feature vector X (a two-dimensional vector). σ is the average value of the current channel c. c This represents the standard deviation of the current channel c. ∈ indicates a non-zero constant to prevent the denominator from being zero. It is usually a minimum value constant of floating-point type.

[0069] The scaling factor z c and Multiply to obtain the channel average s after scaling the current channel c. c :

[0070]

[0071] Channel average s c The input is a fully connected layer, and the output is an attention score vector S from the feature vector X.

[0072] S = W1·s + b1

[0073] Where W1 and b1 represent the weights and biases of the fully connected layer, respectively. In this invention, the weights of the fully connected layer are obtained using the Xavier initialization method, and the biases are initialized to 0. The bias values ​​are continuously updated during model training. s is the scaled channel average vector, s = [s1, s2, ..., s...]. c ].

[0074] Next, the attention score vector S is transformed into the attention weight vector W using a normalization function, as shown in the following equation:

[0075]

[0076] Among them, W i S represents the attention weight of the i-th feature vector. i S represents the attention score of the i-th feature vector. j Let represent the attention score of the j-th feature vector, where j is the total number of feature vectors.

[0077] The input feature vector X and the corresponding attention weight vector W are weighted and summed to obtain the channel attention weighted feature vector A:

[0078] A = ∑ i W i X i

[0079] Finally, the channel-weighted features M of the feature vector X are obtained using the sigmoid activation function. c :

[0080] M c =sigmoid(A)

[0081] The above steps can be simplified to the following formula:

[0082] M c =sigmoid(W(BN(X)))

[0083] Where X is the feature vector and W is the attention weight. Channel-weighted feature M c It can be viewed as a feature map.

[0084] This invention introduces the spatial attention mechanism from the hybrid attention mechanism CBAM into the ResNet-50 network. During the feature extraction process, the input feature map is modeled in the spatial dimension to calculate the importance of different regions in different channels, thereby strengthening the response of important regions and improving the model performance.

[0085] The channel-weighted feature map M output by the channel attention mechanism c ∈R C×H×W (C, H, and W represent the number of channels, height, and width, respectively.) In the input space attention mechanism, global average pooling is performed on each channel of the channel-weighted feature map. For all channels, a vector z containing C elements is obtained, z∈R. C The a-th element z in vector z a This represents the average value of the channel-weighted feature map over the a-th channel. The calculation process is shown below:

[0086]

[0087] To extract useful information from the vector z, the vector z is input into a fully connected layer, resulting in a vector f(z) with r elements, where f(z) ∈ R. r , where r represents the dimension of the vector after channel transformation.

[0088] f(z)=W2δ(W1z)

[0089] Where W1 and W2 represent the weight matrices of the fully connected layer, W1∈Rh×C W2∈R r×h In this invention, the weights of the fully connected layer are generated using the Xavier initialization method, and δ represents the sigmoid activation function.

[0090] The vector f(z) is input into two convolutional layers in sequence. The first convolutional layer maps the vector f(z) into a new feature map M, M∈R, through a 1×1 convolution kernel. d×H×W d represents the number of channels. The second convolutional layer maps the feature map M through a 3×3 convolution kernel, performing convolution, batch normalization (BN) operations, and a ReLU activation function, resulting in a new feature map S′, where S′∈R. 1 ×H×W The feature map S′ is subjected to global max pooling, and then a spatial attention tensor S, S∈R, is obtained by using the sigmoid function. 1×H×W The specific process is as follows:

[0091] M = W3f(b)

[0092] S′=W4ReLU(BN(W5M))

[0093] Where W3, W4, and W5 represent the weight matrices of the first convolutional layer, the second convolutional layer, and the BN layer, respectively, and W3∈R d ×r W4∈R 1×d W5∈R d×d The weights are randomly generated during neural network initialization and are continuously updated as training progresses. This invention uses stochastic gradient descent to minimize the loss function and optimizes the parameters of the weight matrix through backpropagation. After performing global max pooling on the feature map S′, the spatial attention tensor S is obtained using the sigmoid function.

[0094] S = sigmoid(Maxpooling(S′))

[0095] Finally, the spatial attention tensor S is multiplied by the input channel-weighted feature map to obtain the feature map A after feature redirection. s A s ∈R C×H×W :

[0096] A s =S·M c

[0097] Here, "." indicates element-wise multiplication.

[0098] Finally, the process of the spatial attention module described above can be simplified as follows:

[0099] A s =Mc ·(σ(f max (Q(f avg (M c ))))))

[0100] Where Q represents the convolution operation, f avg and f max These represent the average and maximum values ​​of the weighted feature maps along the channel dimension, respectively, where σ is the sigmoid function, and A... s It is the attention map output by the spatial attention mechanism.

[0101] Attention map A s The input is the third convolutional layer (Conv2_3) of convolutional block 2, and the output is the result of convolutional block 2.

[0102] The above explanation uses convolution block 2 as an example. The output of convolution block 2 is input into convolution block 3, and the processing steps of convolution block 2 are repeated to output the result of convolution block 3. The result of convolution block 3 is input into convolution block 4, and the processing steps of convolution block 2 are repeated to output the result of convolution block 4. The result of convolution block 4 is input into convolution block 5, and the processing steps of convolution block 2 are repeated to output the result of convolution block 5. Thus, the features output by each template image after convolution blocks 3, 4, and 5 are obtained. Since the outputs of different convolution blocks correspond to different scales, three features of different scales are obtained, completing the processing of each template image by the ResNet-50 network based on the hybrid attention mechanism.

[0103] S212. Input the three scale features of each template image into convolutional blocks a, b, and c of the Siamese feature fusion network, respectively. Use the 1×1 convolutional layer (Conv1×1) in each convolutional block to reduce the dimensionality of the input scale features, and obtain the three dimensionality-reduced features.

[0104] Each feature is treated as a feature map. The feature map output by convolutional block 3 (Conv 3_4) in the ResNet-50 network based on the hybrid attention mechanism is used as the basic feature map. If the size of the dimensionality-reduced feature map obtained by convolutional block b or convolutional block c of the Siamese feature fusion network is smaller than the size of the basic feature map, the dimensionality-reduced feature map obtained by convolutional block b or convolutional block c is input into the corresponding bilinear interpolation layer. Bilinear interpolation is used to adjust the size of the feature map to the size of the basic feature map, so that all features are the same size in spatial dimension. The adjusted feature map is used as the final output of the corresponding convolutional block. The feature maps output by convolutional blocks a, b, and c are concatenated to obtain a concatenated feature map.

[0105] Assume Yi If there are three scale feature maps that need to be fused, then the twin feature fusion module can be represented by the following formula:

[0106] Y f =θ f {Z i (Y i )}

[0107] Among them, Y f Z represents the spliced ​​feature map. i θ represents the transformation function before fusion of each scale feature map (referring to the convolution, interpolation, etc. operations mentioned above, without an actual formula). f The representative feature fusion function (referring to the above-mentioned concatenation operation).

[0108] After feature fusion, to independently feed the fused feature map (the stitched feature map) into each subsequent RPN module for tracking operations, this invention constructs an improved pyramid feature extraction structure based on the unique network structure of the SiamRPN++ algorithm. Specifically, three additional layers are added after the feature fusion operation, forming the pyramid feature extraction structure. This means the pyramid feature extraction structure has three layers, each consisting of a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, and a transposed convolutional layer. This generates three independent multi-scale fused feature maps. The additional layers are an improvement on the bottleneck residual block. Because the feature fusion module involves operations such as dimensionality adjustment and stitching, the size of the fused feature map is no longer sufficient to meet the input feature map size requirements of subsequent RPN modules. Therefore, to ensure that the size of the fused feature map obtained after the feature fusion operation matches the input size in subsequent RPN modules, this invention adds a transposed convolution operation to the original additional layers, thereby changing the size of the fused feature map.

[0109] First, the concatenated feature maps obtained above are input into each layer of the pyramid feature extraction network. In each layer, the concatenated feature maps are processed sequentially through a 1×1 convolutional layer (Conv 1×1), a 3×3 convolutional layer (Conv 3×3), and a 1×1 convolutional layer (Conv 1×1) to generate low-resolution feature maps. The purpose of the first Conv 1×1 is to reduce the dimensionality of the feature map, thereby reducing the computational cost of the subsequent Conv 3×3. The purpose of the second Conv 1×1 is to increase the dimensionality of the reduced feature map, thereby adjusting it to the same size as the input concatenated feature map, resulting in a low-resolution feature map. The low-resolution feature map is then added to the tensors at corresponding positions of the input concatenated feature map (i.e., a shortcut operation) to obtain the multi-scale feature map output by the current layer. For the three-layer structure of the pyramid feature extraction network, three multi-scale feature maps can be obtained. The specific process is as follows:

[0110] Y′ p =θ P (Y f )

[0111] Among them, Y′ p Represents a multi-scale feature map, θ P Represents a pyramid feature extraction network. This indicates the layer number of the multi-scale feature map in the pyramid feature extraction network.

[0112] S213. For the three multi-scale feature maps of each template image, the channel size of each multi-scale feature map is adjusted by deconvolution operation to increase or decrease its dimensionality. This adjustment yields three new feature maps. The channel size of the new feature maps is determined by the input feature map size set in the RPN module. In this invention, the number of input channels for the three RPN modules are set to 256, 512, and 1024, respectively.

[0113] The ResNet-50 network, incorporating an attention mechanism, effectively distinguishes between the tracked target and background information when extracting features from target images. It extracts more information-rich feature channels and regions from the target image. Simultaneously, the feature fusion module fuses the feature maps from each convolutional layer extracted by ResNet-50, resulting in deep features with multiple semantic information layers. This allows the processed feature image to fully utilize the appearance and semantic information of the tracked target while effectively distinguishing it from background information, ultimately improving the algorithm's robustness to changes in target appearance and scale, as well as background interference. This invention improves the tracking accuracy and anti-interference capability of single-target tracking algorithms based on Siamese networks.

[0114] The following is a partial code snippet of the above process:

[0115]

[0116] S22. In the search branch, the next frame search image corresponding to each template image is input into the model, and then processed sequentially by the ResNet-50 network based on the hybrid attention mechanism and the Siamese feature fusion network to output the feature map. The specific process is as follows:

[0117] The processing of each search image within the model is the same as the process described in S21 above.

[0118] S221. Each search image is input into a ResNet-50 network based on a hybrid attention mechanism. After each search image is processed by convolutional block 1 and pooling layer, it is input into convolutional block 2. Since convolutional block 2 has channel attention mechanism and spatial attention mechanism, the input of convolutional block 2 is processed by the first two convolutional layers of convolutional block 2 to obtain the corresponding feature vector X. Let the total number of feature channels of feature vector X be C.

[0119] The feature vector X is input into the channel attention mechanism, and the channel-weighted feature M of the output feature vector X is generated. c .

[0120] Channel weighted feature M c In the input spatial attention mechanism, the output attention map A s .

[0121] Attention map A s The input to the third convolutional layer of convolutional block 2 is the result of convolutional block 2.

[0122] The output of convolution block 2 is sequentially input into convolution block 3, convolution block 4, and convolution block 5. The processing steps of convolution block 2 are repeated in convolution block 3, convolution block 4, and convolution block 5. Since the output of different convolution blocks corresponds to different scales, three features of different scales are obtained.

[0123] S222. Input the three scale features of each search image into convolutional blocks a, b, and c of the Siamese feature fusion network, respectively. Use the 1×1 convolutional layer in each convolutional block to reduce the dimensionality of the input scale features, and obtain the three dimensionality-reduced features.

[0124] Each feature is treated as a feature map. The feature map output by convolution block 3 in the ResNet-50 network based on the hybrid attention mechanism is used as the basic feature map. If the size of the dimensionality-reduced feature map obtained by convolution block b or convolution block c of the Siamese feature fusion network is smaller than the size of the basic feature map, the dimensionality-reduced feature map obtained by convolution block b or convolution block c is input into the corresponding bilinear interpolation layer. Bilinear interpolation is used to adjust the size of the feature map to the size of the basic feature map. The adjusted feature map is used as the final output of the corresponding convolution block. The feature maps output by convolution block a, convolution block b and convolution block c are concatenated to obtain a concatenated feature map.

[0125] The spliced ​​feature maps obtained above are input into each layer of the pyramid feature extraction network. In each layer, the spliced ​​feature maps are processed sequentially through a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer to generate low-resolution feature maps. The tensors at the corresponding positions of the low-resolution feature maps and the spliced ​​feature maps are added to obtain the multi-scale feature maps output by the current layer, thus obtaining three multi-scale feature maps.

[0126] S223. For each search image, the channel size of the three multi-scale feature maps is adjusted using deconvolution to obtain three new feature maps after adjustment.

[0127] S23. Input the feature maps output from S21 and S22 into the RPN network for similarity comparison, and output the predicted region in the search image with the highest similarity to the template image. The RPN network includes three RPN modules, each of which includes an anchor generator, a convolutional feature extractor, and a box regressor. The specific process is as follows:

[0128] The three adjusted new feature maps obtained in S213 and S222 are input into the RPN network. For each RPN module, one adjusted new feature map obtained in S213 and one adjusted new feature map obtained in S222 are input. The similarity of the two input new feature maps is compared, and feature maps with target candidate boxes are output respectively. Since the three RPN modules have the same output size, the outputs of each RPN module are weighted and summed to obtain the predicted region with the highest similarity to the template image in the search image.

[0129] The Siamese RPN module used in this invention is designed in the same way as the SiamRPN++ algorithm and is a direct reference to existing technology. The role of the RPN module is to provide candidate boxes for subsequent tracking modules and to continuously update the candidate boxes during the tracking process to improve the accuracy and robustness of tracking.

[0130] Example

[0131] This invention uses the datasets COCO, LaSOT, and YOUTUBB to train the algorithm, and preprocesses the datasets in the same way as the SiamRPN++ single-target tracking algorithm.

[0132] During algorithm training, the target template size was set to 127×127, and the search template size was set to 256×256. The algorithm uses a model (a ResNet-50 network based on a hybrid attention mechanism, a Siamese feature fusion network, and an RPN network) for training. During training, the model parameters were continuously optimized using stochastic gradient descent. The model was trained on a single Nvidia GTX 2080TI graphics card with a batch size of 8. Similar to the SiamRPN++ single-object tracking algorithm, this algorithm underwent 20 epochs of training. For the first 10 epochs, the parameters of the Siamese feature fusion network were fixed, and the subsequent parameters of the model were trained and optimized. In the last 10 epochs, the last three blocks of the Siamese feature fusion network were unfixed, and the entire model was trained.

[0133] To verify the effectiveness of the single-target tracking algorithm proposed in this invention, it is compared with several advanced single-target tracking algorithms based on Siamese networks in recent years on the datasets OTB100 and VOT2018.

[0134] The OTB dataset is a classic visual object tracking dataset released by Wu et al. in 2013. It's a video-based tracking evaluation dataset containing 100 video sequences covering various object categories, scenes, and environments, and incorporating multiple challenge factors. Each video sequence in the OTB dataset contains a moving object and provides realistic annotations of its location, size, and shape. The video sequences in the dataset are all at standard VGA resolution, i.e., 640×480 pixels.

[0135] The VOT dataset is a widely used, standardized visual object tracking dataset and evaluation platform designed to provide a unified benchmark for comparing and evaluating tracking algorithms. This dataset and evaluation platform were jointly developed by several European computer vision research teams and are continuously updated and maintained. The VOT dataset mainly consists of two parts: video sequence data and evaluation tools. The video sequence data comes from various sources, such as movies, television, the internet, and self-recorded videos, and includes a wide variety of targets and scenes, such as moving cars, people, animals, natural landscapes, etc.

[0136] Evaluation Methods: To verify the effectiveness of the present invention, ablation experiments were conducted between the improved hybrid attention mechanism and the original SiamRPN++ algorithm. The present invention was tested and compared with various single-target tracking algorithms based on Siamese networks (SiamFC, SiamRPN, DaSiamRPN, SiamRPN++). The experimental results were analyzed quantitatively and qualitatively.

[0137] First, SiamRPN++ was used as the baseline algorithm. Based on this, only an improved channel attention mechanism (CAmodule) was added. The results are shown in Table 1. It can be seen that the SiamRPN++ single-object tracking algorithm with the addition of the channel attention mechanism brings a 1.1% improvement in accuracy on the VOT2018 dataset. The comparison shows that the tracking robustness of the algorithm is also improved to a certain extent. Compared with the SiamRPN++ algorithm, the number of tracking errors is reduced by 3.

[0138] Then, based on the SiamRPN++ single-object tracking algorithm, only the spatial attention mechanism (SAmodule) was added, as shown in Table 1. It can be found that the SiamRPN++ single-object tracking algorithm with the addition of the spatial attention mechanism brings a 0.8% accuracy improvement on the VOT2018 dataset.

[0139] Finally, based on the SiamRPN++ single-target tracking algorithm, channel attention and spatial attention mechanisms were added simultaneously. Experimental results show that the SiamRPN++ algorithm with the improved hybrid attention mechanism proposed in this invention achieves better accuracy than the SiamRPN++ single-target tracking algorithm on the VOT2018 dataset, with improvements in accuracy and robustness.

[0140] Table 1 compares the experimental results on the VOT2018 dataset.

[0141]

[0142] To ensure the objectivity of the experimental results, this invention compares the proposed algorithm with four advanced tracking algorithms developed in recent years: SiamFC, SiamRPN, DaSiamRPN, and SiamRPN++. Tracking was performed on 100 sequences of images from the OTB100 dataset. Evaluation metrics for comparing the performance of these tracking algorithms included tracking success rate and tracking accuracy. For algorithms that allow real-time tracking, a single evaluation can generate success rate and accuracy charts. Figure 1The test results of five different target tracking algorithms (ff-SiamRPN++-a, SiamFC, SiamRPN, DaSiamRPN, SiamRPN++) on the OTB100 dataset are presented.

[0143] Table 2 shows the tracking accuracy and success rate of the proposed single-target tracking algorithm ff-SiamRPN++-a and four other single-target tracking algorithms based on Siamese networks on the OTB100 dataset. This invention introduces an attention mechanism and a Siamese feature fusion module on top of the SiamRPN++ single-target tracking algorithm. The results show that compared to the SiamRPN++ single-target tracking algorithm, the accuracy and success rate of the improved algorithm proposed in this invention are improved by 2.7% and 1.8%, respectively. On the OTB-100 dataset, this invention achieves an accuracy of 0.914 and a success rate of 0.671, respectively. Of course, the attention module also has certain drawbacks, as shown in Table 2 (V...). FPS As shown in the first section, the present invention, which incorporates an attention mechanism and a feature fusion module, suffers from a certain loss in tracking speed. This is because some convolutional blocks in the feature fusion module and attention mechanism increase the computational load of the model. However, the improved SiamRPN++-a algorithm still meets the real-time requirements in terms of real-time inference speed.

[0144] Table 2 shows the accuracy and success rate of the five algorithms on the OTB100 dataset.

[0145]

[0146]

[0147] To more intuitively compare the tracking performance of the algorithm proposed in this invention with other algorithms, this section visualizes representative tracking results of five single-target tracking algorithms on a test dataset. Four image sequences with different interference factors were selected from the OTB100 dataset to demonstrate the tracking results. The figure shows the tracking results of five different single-target tracking algorithms based on the Siamese network (SiamRPN++-a, SiamFC, SiamRPN, DaSiamRPN, SiamRPN++), with different colored rectangles representing the tracking results of different algorithms. Figure 2As shown in the tracking results, the proposed Siamese network single-target tracking algorithm ff-SiamRPN++-a, based on Siamese feature fusion, can still maintain good tracking ability even when the tracked target encounters interference from similar targets, background interference, or rapid movement. Compared with other Siamese network single-target tracking algorithms, the present invention is superior to other target tracking algorithms and has high robustness.

Claims

1. A video single-target tracking method based on feature fusion using Siamese networks, characterized in that: It includes the following steps: S1. Obtain a surveillance video, extract each frame of the video, label all extracted images, preprocess all labeled images, crop each frame into a pair of fixed-size images, divide each cropped frame into a template image and a search image, assemble all template images into a template region image set, and assemble all search images into a search region image set. S2. Model Construction: The model consists of a ResNet-50 network based on a hybrid attention mechanism and a Siamese feature fusion network. Template region image sets and search region image sets are input into the model for training. Feature maps of the template and search images are output separately. This process continues until the upper limit of the number of iterations is met, resulting in a trained model. The specific process is as follows: S21. In the template branch, the template region image set is input into the model. Each template image is processed sequentially by the ResNet-50 network based on the hybrid attention mechanism and the Siamese feature fusion network to output a feature map. S22. In the search branch, the next frame search image corresponding to each template image is input into the model, and then processed by the ResNet-50 network based on the hybrid attention mechanism and the Siamese feature fusion network in sequence to output the feature map. S23. Input the feature map output by S21 and the feature map output by S22 into the RPN network for similarity comparison, and output the predicted region in the search image with the highest similarity to the template image. The twin feature fusion network includes convolutional block a, convolutional block b, convolutional block c, and a pyramid feature extraction network. Convolutional block a includes a 1×1 convolutional layer, convolutional block b includes a 1×1 convolutional layer and a bilinear interpolation layer, and convolutional block c includes a 1×1 convolutional layer and a bilinear interpolation layer. The pyramid feature extraction network has three layers, each including a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, and a transposed convolutional layer. S3. Obtain the surveillance video to be tracked. Select a target in the first frame of the surveillance video for tracking. Based on the first frame of the selected target and the second frame of the surveillance video, use the model trained in S2 to output the predicted region of the selected target in the second frame of the first frame, and obtain the position of the target in the second frame of the video. Then, based on the second frame image containing the target and the third frame image of the surveillance video, the model trained in S2 is used to output the predicted region of the target in the second frame image in the third frame image, and so on, to achieve single target tracking.

2. The Siamese network video single-target tracking method based on feature fusion as described in claim 1, characterized in that: The template image in S1 is 127×127 in size, and the search image is 255×255 in size.

3. The Siamese network video single-target tracking method based on feature fusion as described in claim 2, characterized in that: The ResNet-50 network based on the hybrid attention mechanism in S2 includes, in sequence, convolutional block 1, pooling layer, convolutional block 2, convolutional block 3, convolutional block 4, and convolutional block 5. Convolutional block 1 includes one convolutional layer, convolutional block 2 includes three convolutional layers, convolutional block 3 includes four convolutional layers, convolutional block 4 includes six convolutional layers, and convolutional block 5 includes three convolutional layers. Channel attention and spatial attention mechanisms are sequentially set between the last two convolutional layers of convolutional block 2, convolutional block 3, convolutional block 4, and convolutional block 5. The outputs of convolutional blocks 3, 4, and 5 are all used as the final outputs of the ResNet-50 network based on the hybrid attention mechanism. The outputs of different convolutional blocks correspond to different scales.

4. The Siamese network video single-target tracking method based on feature fusion as described in claim 3, characterized in that: The RPN network in S23 includes three RPN modules, each of which includes an anchor generator, a convolutional feature extractor, and a candidate box regressor.

5. The Siamese network video single-target tracking method based on feature fusion as described in claim 4, characterized in that: The specific process of S21 is as follows: S211. Each template image is input into a ResNet-50 network based on a hybrid attention mechanism. After being processed by convolutional block 1 and pooling layers, each template image is input into convolutional block 2. Since convolutional block 2 has channel attention and spatial attention mechanisms, the input to convolutional block 2 is processed by the first two convolutional layers of convolutional block 2 to obtain the corresponding feature vector. X Let the eigenvectors be... X The total number of feature channels is C; eigenvectors X Within the input channel attention mechanism, the output feature vector X Channel weighting characteristics ; Channel weighting features In the input spatial attention mechanism, the output attention map ; attention map The result of convolutional block 2 is output from the third convolutional layer that takes convolutional block 2 as input. The output of convolution block 2 is sequentially input into convolution block 3, convolution block 4, and convolution block 5. The processing steps of convolution block 2 are repeated in convolution block 3, convolution block 4, and convolution block 5. Since the output of different convolution blocks corresponds to different scales, three features of different scales are obtained. S212. Input the three scale features of each template image into convolutional blocks a, b, and c of the Siamese feature fusion network, respectively. Use the 1×1 convolutional layer in each convolutional block to reduce the dimensionality of the input scale features to obtain the three dimensionality-reduced features. Each feature is treated as a feature map. The feature map output by convolution block 3 in the ResNet-50 network based on the hybrid attention mechanism is used as the basic feature map. If the size of the dimensionality-reduced feature map obtained by convolution block b or convolution block c of the Siamese feature fusion network is smaller than the size of the basic feature map, the dimensionality-reduced feature map obtained by convolution block b or convolution block c is input into the corresponding bilinear interpolation layer. Bilinear interpolation is used to adjust the size of the feature map to the size of the basic feature map. The adjusted feature map is used as the final output of the corresponding convolution block. The feature maps output by convolution block a, convolution block b and convolution block c are concatenated to obtain a concatenated feature map. The spliced ​​feature maps obtained above are input into each layer of the pyramid feature extraction network. In each layer, the spliced ​​feature maps are processed by a 1×1 convolutional layer, a 3×3 convolutional layer and a 1×1 convolutional layer to generate low-resolution feature maps. The tensors at the corresponding positions of the low-resolution feature maps and the spliced ​​feature maps are added to obtain the multi-scale feature maps output by the current layer, thus obtaining three multi-scale feature maps. S213. For each template image, the channel size of the three multi-scale feature maps is adjusted using deconvolution to obtain three new feature maps after adjustment.

6. The Siamese network video single-target tracking method based on feature fusion as described in claim 5, characterized in that: The channel weighting features : in, BN This indicates a batch normalization operation. X Represents the eigenvector. W This represents the attention weight.

7. The Siamese network video single-target tracking method based on feature fusion as described in claim 6, characterized in that: The attention map : in, Q Represents the convolution operation. and These represent the average and maximum values ​​of the weighted feature maps along the channel dimension, respectively. It is the sigmoid function.

8. The Siamese network video single-target tracking method based on feature fusion as described in claim 7, characterized in that: The specific process of S22 is as follows: S221. Each search image is input into a ResNet-50 network based on a hybrid attention mechanism. After being processed by convolutional block 1 and pooling layers, each search image is input into convolutional block 2. Since convolutional block 2 has channel attention and spatial attention mechanisms, the input to convolutional block 2 is processed by the first two convolutional layers of convolutional block 2 to obtain the corresponding feature vector. X Let the eigenvectors be... X The total number of feature channels is C; eigenvectors X Within the input channel attention mechanism, the output feature vector X Channel weighting characteristics ; Channel weighting features In the input spatial attention mechanism, the output attention map ; attention map The result of convolutional block 2 is output from the third convolutional layer that takes convolutional block 2 as input. The output of convolution block 2 is sequentially input into convolution block 3, convolution block 4, and convolution block 5. The processing steps of convolution block 2 are repeated in convolution block 3, convolution block 4, and convolution block 5. Since the output of different convolution blocks corresponds to different scales, three features of different scales are obtained. S222. Input the three scale features of each search image into convolutional blocks a, b, and c of the Siamese feature fusion network, respectively. Use the 1×1 convolutional layer in each convolutional block to reduce the dimensionality of the input scale features to obtain the three dimensionality-reduced features. Each feature is treated as a feature map. The feature map output by convolution block 3 in the ResNet-50 network based on the hybrid attention mechanism is used as the basic feature map. If the size of the dimensionality-reduced feature map obtained by convolution block b or convolution block c of the Siamese feature fusion network is smaller than the size of the basic feature map, the dimensionality-reduced feature map obtained by convolution block b or convolution block c is input into the corresponding bilinear interpolation layer. Bilinear interpolation is used to adjust the size of the feature map to the size of the basic feature map. The adjusted feature map is used as the final output of the corresponding convolution block. The feature maps output by convolution block a, convolution block b and convolution block c are concatenated to obtain a concatenated feature map. The spliced ​​feature maps obtained above are input into each layer of the pyramid feature extraction network. In each layer, the spliced ​​feature maps are processed by a 1×1 convolutional layer, a 3×3 convolutional layer and a 1×1 convolutional layer to generate low-resolution feature maps. The tensors at the corresponding positions of the low-resolution feature maps and the spliced ​​feature maps are added to obtain the multi-scale feature maps output by the current layer, thus obtaining three multi-scale feature maps. S223. For each search image, the channel size of the three multi-scale feature maps is adjusted using deconvolution to obtain three new feature maps after adjustment.

9. The Siamese network video single-target tracking method based on feature fusion as described in claim 8, characterized in that: The specific process of S23 is as follows: The three adjusted new feature maps obtained in S213 and S222 are input into the RPN network. For each RPN module, one adjusted new feature map obtained in S213 and one adjusted new feature map obtained in S222 are input. The similarity of the two input new feature maps is compared, and feature maps with target candidate boxes are output respectively. Since the three RPN modules have the same output size, the outputs of each RPN module are weighted and summed to obtain the predicted region in the search image that has the highest similarity to the template image.

Images