A target tracking method and system based on cross-correlation matching enhanced twin network

Abstract

This invention discloses a target tracking method and system based on a cross-correlation matching enhanced Siamese network, comprising: cropping a video sequence of the target to be tracked to obtain template images and search images for all frames; inputting the template images and search images into a constructed and trained cross-correlation matching enhanced Siamese network to extract template features and search features from the template images and search images; performing cross-correlation matching on the template features and search features to obtain cross-correlation features; encoding bounding box information in the template images to obtain bounding box encoded features; performing classification and regression calculations on the fused features of cross-correlation features and bounding box encoded features to obtain corresponding classification score maps and regression prediction maps; and obtaining the final position of the target on the video sequence frames based on the position with the largest response value in the classification score map and the offset of the regression prediction map. This invention has strong adaptability and high accuracy in tracking complex scene changes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and target tracking technology, and in particular to a target tracking method and system based on cross-correlation matching enhanced Siamese networks. Background Technology

[0002] Target tracking is a fundamental yet challenging task in computer vision, and one of the most active research topics in the field in recent decades. The task of target tracking is defined as: given only the initial frame position of the target in a video sequence, the ability to accurately track the target in every subsequent frame. Target tracking has wide applications in autonomous driving, video surveillance, marine exploration, and medical imaging, thus attracting significant attention from both academia and industry. Currently, traditional target tracking methods based on correlation filtering suffer from low robustness; however, the offline training and online tracking approach based on Siamese networks achieves a good balance between tracking accuracy and inference speed. Siamese networks perform similarity learning to estimate the most likely position of the target in the next frame, overcoming the shortcomings of traditional methods.

[0003] As a representative of Siam network-based target tracking methods, SiamFC introduces a cross-correlation structure, truly achieving a balance between speed and accuracy. SiamRPN improves upon SiamFC's cross-correlation method and introduces a region generation network, making regression more accurate. However, the cross-correlation method used in SiamRPN generates a very large number of parameters, making the network difficult to train and optimize overall. SiamRPN++ introduces deep neural networks into the Siam tracking network, significantly improving tracking performance, while employing a depth-separable cross-correlation method, reducing a large number of parameters and stabilizing the entire training process. However, regardless of the existing cross-correlation method, its essence is still a fixed-size sliding window convolution operation between two feature maps. Therefore, when the object undergoes large deformation or the target area is relatively small, cross-correlation will introduce a lot of background information, thus interfering with the tracking of the target object. SiamBAN solves the problem caused by the anchor boxes of the SiamRPN series of networks. It obtains the predicted box of the maximum response position by directly predicting the classification scores of the foreground and background on the output feature map and the distance offset of the four centers, reducing the burden of parameter adjustment. However, due to the loss of some prior information, the tracker does not have the ability to learn and cope well with changes in the target scale. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a target tracking method and system based on cross-correlation matching enhanced Siamese networks. This invention solves the technical problems of existing mainstream target tracking methods based on Siamese networks not making full use of prior information and feature ambiguity caused by simple cross-correlation matching. This invention can reduce irrelevant background information and interference information, improve the discrimination ability of the tracking network, and thus make the target position more accurate.

[0005] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0006] In a first aspect, the present invention provides a target tracking method based on cross-correlation matching enhanced Siamese networks, the method comprising:

[0007] The video sequence of the target to be tracked is cropped to obtain template images and search images for all frames;

[0008] The template image and the search image are input into a constructed and trained cross-correlation matching enhanced Siamese network. The cross-correlation matching enhanced Siamese network extracts template features and search features from the template image and the search image. Cross-correlation matching is performed on the template features and the search features to obtain cross-correlation features. Bounding box information is encoded into the template image to obtain bounding box encoded features. Classification and regression calculations are performed on the fused features of cross-correlation features and bounding box encoded features to obtain the corresponding classification score map and regression prediction map.

[0009] Based on the position with the largest response value in the classification score graph, and combined with the offset in the regression prediction graph, the final position of the target in the video sequence frame is obtained.

[0010] In conjunction with the first aspect, preferably, the construction and training process of the cross-correlation matching enhanced Siamese network includes:

[0011] Obtain the target video sequence frame dataset, crop each frame image in the dataset according to the target position and size of the image, and obtain the template image and search image of all frame images, which are used as the training sample set;

[0012] A cross-correlation matching enhanced Siamese network is constructed, which includes a feature extraction network, a cross-correlation matching network, a classification and regression network, and a bounding box encoding module. The feature extraction network is an improved ResNet50 deep residual network. The improvement of the ResNet50 deep residual network includes: removing the fifth convolutional layer of the original ResNet50 deep residual network, setting the stride of the third and fourth convolutional layers to 1, and setting the size of the dilated convolutions of the third and fourth layers to 4.

[0013] The constructed cross-correlation matching enhanced Siamese network is trained based on the training sample set to obtain the trained cross-correlation matching enhanced Siamese network.

[0014] In conjunction with the first aspect, preferably, the method for cropping each frame of the image in the dataset includes:

[0015] The first frame of the target video sequence is cropped into a template image of size 127*127*3 with the target as the center. Starting from the second frame, the subsequent frames of the target video sequence are cropped into a template image of size 255*255*3 with the target as the center.

[0016] In conjunction with the first aspect, preferably, the cross-correlation matching network includes a horizontal scale extraction module, a vertical scale extraction module, and a cascaded dual cross-correlation module;

[0017] The cascaded dual cross-correlation module is used to perform pixel-matching cross-correlation on template features and search features, and then perform depth-separable cross-correlation on template features to obtain cascaded dual cross-correlation features. The calculation formula is as follows:

[0018] f pm =PM(f z ,f x )

[0019] F Dw =DW(f pm ,f z (1)

[0020] In the formula, f pm Representing template feature f z and search features f x The pixel-matching cross-correlation feature, where PM represents pixel-matching cross-correlation; F Dw This indicates a cascaded dual cross-correlation feature, and DW indicates a deep separable cross-correlation.

[0021] The lateral scale extraction module is used to extract the lateral scale branch features of the template features and the search features. Its calculation formula is as follows:

[0022] H k =HD 3:1 (N k )

[0023] F H =DW(f z *H k ,f x *H k (2)

[0024] Where: N k H represents a regular N*N convolution kernel. kRepresents the convolution kernel N k via HD 3:1 A horizontal 3:1 convolution is expanded to a 7*3 horizontal convolution kernel; F H Indicates horizontal scale branching features; * indicates a convolution operation;

[0025] The vertical scale extraction module is used to extract the vertical scale branch features of the template features and the search features. Its calculation formula is as follows:

[0026] V k =VD 1:3 (N k )

[0027] F V =DW(f z *V k ,f x *V k (3)

[0028] In the formula, V k Represents the convolution kernel N k via VD 1:3 Vertically expand the convolution by a 1:3 ratio to create a 3x7 vertical convolution kernel; F V This represents the branching features at the vertical scale.

[0029] In conjunction with the first aspect, preferably, based on the cascaded dual cross-correlation feature F Dw Horizontal scale branching feature F H and longitudinal scale branching feature F V The cross-correlation feature F output by the cross-correlation matching network is calculated as follows:

[0030] F=α1F H +α2F K +α3F DW (4)

[0031] In the formula, α1, α2, and α3 represent the cascaded double cross-correlation characteristics F. Dw Horizontal scale branching feature F H and longitudinal scale branching features F V The fusion coefficients, α1, α2 and α3, are obtained by optimizing the values ​​of the fusion coefficients based on network training.

[0032] In conjunction with the first aspect, preferably, the bounding box encoding module includes multiple fully connected layers; the step of the bounding box encoding module encoding bounding box information of the template image includes:

[0033] The coordinates of the target bounding box in the template image are transformed into a one-dimensional feature vector B, where B∈(x,y,w,h), (x,y) represents the corner coordinates of the target bounding box, w represents the width of the target bounding box, and h represents the height of the target bounding box.

[0034] The feature vector B is encoded through multiple fully connected layers to obtain bounding box encoded features B. C Represented as:

[0035] B C =f C (B) (5)

[0036] In the formula: f C B represents a fully connected layer structure. C This represents the output feature of feature vector B after passing through a fully connected layer.

[0037] In conjunction with the first aspect, preferably, the extraction step of the fusion feature of the cross-correlation feature and the bounding box encoding feature includes:

[0038] The cross-correlation feature F is compared with the bounding box encoding feature B. C Performing a broadcast addition operation yields the initial fusion feature F. b :

[0039] F b =F+B C (6)

[0040] Regarding the preliminary fusion feature F b Perform 1x1 convolutional encoding to obtain the fused feature F. BM :

[0041] F BM =f g (F b (7)

[0042] In the formula, f g This indicates a convolutional encoding operation.

[0043] In conjunction with the first aspect, preferably, the step of training the constructed cross-correlation matching enhanced Siamese network based on the training sample set includes:

[0044] Pairs of template images and search images are randomly selected from the training sample set and used as inputs to the two branches of the cross-correlation matching enhanced Siamese network;

[0045] Gradient backpropagation is performed using the SGD stochastic gradient descent method with momentum to optimize network parameters until the joint task loss function converges.

[0046] The formula for calculating the joint task loss function is as follows:

[0047] L=λ1L cls +λ2L reg (8)

[0048] In the formula, L cls L represents the binary cross-entropy loss function. reg Let λ1 and λ2 represent the IOU loss function, respectively. cls and L reg The weight.

[0049] Secondly, the present invention provides a target tracking system based on cross-correlation matching enhanced Siamese network, the system comprising:

[0050] The cropping module is used to crop the video sequence of the target to be tracked, and obtain template images and search images for all frames.

[0051] The classification prediction module is used to input the template image and the search image into a constructed and trained cross-correlation matching enhanced Siamese network, extract template features and search features from the template image and the search image through the cross-correlation matching enhanced Siamese network; perform cross-correlation matching on the template features and the search features to obtain cross-correlation features; encode the bounding box information of the template image to obtain bounding box encoded features; and perform classification and regression calculations on the fused features of cross-correlation features and bounding box encoded features to obtain the corresponding classification score map and regression prediction map.

[0052] The target location acquisition module is used to obtain the final position of the target on the video sequence frames based on the position with the largest response value in the classification score map and the offset of the regression prediction map.

[0053] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the target tracking method based on cross-correlation matching enhanced Siamese network as described in any of the first aspects.

[0054] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0055] I. The cross-correlation matching enhanced Siamese network constructed in this invention makes full use of existing prior information. By using the bounding box encoding module, the rich foreground information of the template image bounding box is encoded into the network, so that the tracking network that integrates foreground information can focus more on the region where the target is located, thereby improving the performance of the tracker.

[0056] Second, the tracking network constructed in this invention adopts an improved ResNet50 deep residual network and introduces unconventional scale convolution kernels to extract multi-scale features from template features and search features. This avoids a series of anchor box parameter adjustments and can obtain more scale information, making it more suitable for tracking tasks.

[0057] Third, the cross-correlation matching network provided by this invention uses a dual matching method of pixel matching cross-correlation and depth separable cross-correlation cascade to further refine the extraction of target local area information based on the capture of features in different directions. This solves the inherent problems of existing cross-correlation matching methods, reduces irrelevant background information and interference information, and improves the discrimination ability of the tracking network.

[0058] Furthermore, the present invention has undergone extensive performance evaluation experiments on various target tracking dataset benchmarks. The experimental results show that the present invention can effectively address issues such as occlusion, disappearance of field of view, motion blur, cluttered background, and scale changes, and has high real-time performance with good tracking performance, further improving the accuracy and robustness of target tracking. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the network structure of the target tracking method based on cross-correlation matching enhanced twin network provided in the embodiments of the present invention;

[0060] Figure 2 This is a schematic diagram illustrating the fusion of the output of the cross-correlation matching network and the output of the bounding box encoding module provided in this embodiment of the invention;

[0061] Figure 3 This is a schematic diagram of cascaded dual cross-correlation provided in an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of pixel matching cross-correlation provided in an embodiment of the present invention;

[0063] Figure 5 This is a schematic diagram of multi-scale feature extraction provided in an embodiment of the present invention;

[0064] Figure 6 This is a comparison chart of the tracking visualization results of the method of the present invention and other methods for OTB100 video sequences provided in this embodiment of the invention;

[0065] Figure 7 This is a block diagram illustrating the structural principle of a target tracking system based on cross-correlation matching enhanced twin network provided in an embodiment of the present invention. Detailed Implementation

[0066] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0067] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0068] Example 1:

[0069] This embodiment introduces a target tracking method based on cross-correlation matching enhanced Siamese networks. The overall framework of cross-correlation matching enhanced Siamese network tracking is described in reference to... Figure 1 As shown, the method comprises the following steps:

[0070] Step 1: Crop the video sequence of the target to be tracked to obtain template images and search images for all frames;

[0071] Step 2: Input the template image and the search image into the constructed and trained cross-correlation matching enhanced Siamese network. Extract template features and search features from the template image and the search image through the cross-correlation matching enhanced Siamese network; perform cross-correlation matching on the template features and the search features to obtain cross-correlation features; encode the bounding box information of the template image to obtain bounding box encoded features; perform classification and regression calculations on the fused features of cross-correlation features and bounding box encoded features to obtain the corresponding classification score map and regression prediction map.

[0072] Step 3: Based on the position with the largest response value in the classification score map, and combined with the offset of the regression prediction map, obtain the final position of the target on the video sequence frame.

[0073] As an embodiment of the present invention, the construction and training process of the cross-correlation matching enhanced Siamese network includes:

[0074] Step a: Obtain the target video sequence frame dataset. Based on the target position and size of the image, crop each frame in the dataset to obtain the template image and search image of all frame images, and use them as the training sample set.

[0075] Step b: Construct a cross-correlation matching enhanced Siamese network, which includes a feature extraction network, a cross-correlation matching network, a classification and regression network, and a bounding box encoding module; the feature extraction network is an improved ResNet50 deep residual network, and the improvement of the ResNet50 deep residual network includes: removing the fifth convolutional layer of the original ResNet50 deep residual network, setting the stride of the third and fourth convolutional layers to 1, and setting the size of the dilated convolutions of the third and fourth layers to 4;

[0076] Step c: Train the constructed cross-correlation matching enhanced Siamese network based on the training sample set to obtain the trained cross-correlation matching enhanced Siamese network.

[0077] The feature extraction network is used to obtain template features and search features of the template image and search image, respectively; the cross-correlation matching network is used to perform cross-correlation matching on the template features and search features to obtain cross-correlation features; the bounding box encoding module is used to encode the bounding box information of the template image to obtain bounding box encoded features; and the classification and regression network is used to perform classification and regression calculations on the fused features of cross-correlation features and bounding box encoded features to obtain the corresponding classification score map and regression prediction map.

[0078] Specifically, in step a, the first frame of the target video sequence is cropped into a template image of size 127*127*3 with the target as the center. Starting from the second frame, the subsequent frames of the target video sequence are cropped into a template image of size 255*255*3 with the target as the center. In addition, as a training sample set, before cropping the template image and the search image, some perturbation is performed on the image to enhance the robustness of training. For example, the contrast is randomly adjusted, probabilistic flipping is performed, a certain angle is rotated, and the center point is shifted to a certain extent. Then, the cropping is performed based on the center point.

[0079] In step b of this invention, the feature extraction network constructed is an improved ResNet50 deep residual network for deep feature extraction. The improvements include setting the stride of the third and fourth convolutional layers to 1 and the size of the dilated convolutions of the third and fourth layers to 4 to increase the size of the receptive field. The fifth convolutional layer of the original ResNet50 deep residual network is removed, which can significantly accelerate the training and inference speed of the Siamese network without significantly affecting the accuracy.

[0080] Specifically, the cross-correlation matching enhanced Siamese network constructed in this embodiment of the invention includes two output branches: a template image branch and a search image branch. After the ResNet50 deep residual network extracts features from the two branch images, it obtains the template features. and search features Indicates a height of H z Width is W z A feature map with C channels. Indicates a height of H x Width is W x A feature map with C channels;

[0081] Reference Figure 1 and Figure 3 As shown, the cross-correlation matching network includes a horizontal scale extraction module, a vertical scale extraction module, and a cascaded dual cross-correlation module; template features and search features When inputting cascaded dual mutual modules, pixel matching cross-correlation is performed first:

[0082] f pm =PM(f z ,f x (1)

[0083] In the formula, f pm Representing template feature f z and search features f x The pixel matching cross-correlation feature, where PM represents pixel matching cross-correlation;

[0084] Next, the pixel matching cross-correlation feature f pm With template feature f z Deeply separable cross-correlation was performed, ultimately yielding cascaded double cross-correlation features:

[0085] F Dw =DW(f pm ,f z (2)

[0086] In the formula, F Dw This indicates a cascaded dual cross-correlation feature, and DW indicates a deep separable cross-correlation. Figure 3 In the diagram, C×H×W represents the height H, width W, and number of channels C of the cascaded dual cross-correlation feature;

[0087] Furthermore, Figure 4 The process of pixel matching and cross-correlation is shown in detail:

[0088] First, the template image branch and the search image branch are respectively encoded by convolutional layers to obtain f. z1 and f x1 ;f z1 Perform dimensional transformation operations to obtain f respectively z11 and f z12 f x1 Perform a dimensional transformation operation to obtain f x2 :

[0089] f z11 =reshape(Conv(f) z ))

[0090] f z12 =reshape(Conv(f) z ))

[0091] f x2 =reshape(Conv(f) x1 (3)

[0092] In the formula, Conv represents a convolutional layer of size 1*1*C, and reshape represents a dimensionality transformation operation;

[0093] Next, f z12 with f x2 Perform matrix multiplication, and then multiply the result by f. z11 Perform matrix multiplication to obtain f. m1 and for f m1 Perform a dimensional transformation operation to obtain f m2 :

[0094] f m1 =MatMul(MatMul(f x2 ,f z12 ),f z11 )

[0095] f m2 =Reshape(f m1 (4)

[0096] In the formula, MatMul represents the matrix multiplication operation;

[0097] Finally, f m2 Compared with the previously encoded features f x1 The stitching operation is performed along the channels, and finally, after dimensionality reduction by 1*1 convolution, the output f of pixel matching cross-correlation is obtained. pm :

[0098] f pm =Conv(Concate(f) x1 ,f m2 (5)

[0099] In the formula, Concate indicates a splicing operation along the channel.

[0100] Specifically, refer to Figure 5As shown, the template features and search features input to the cross-correlation matching network are processed by the lateral scale extraction module to obtain their respective lateral scale features. The two lateral features are then cross-correlated to output the lateral scale branch features.

[0101] H k =HD 3:1 (N k )

[0102] F H =DW(f z *H k ,f x *H k (6)

[0103] Where: N k H represents a regular N*N convolution kernel. k Represents the convolution kernel N k via HD 3:1 A horizontal 3:1 convolution is expanded to a 7*3 horizontal convolution kernel; F H Indicates horizontal scale branching features; * indicates a convolution operation;

[0104] Similarly, the vertical scale extraction module extracts the vertical scale branch features of the template features and the search features, and outputs the vertical scale branch features:

[0105] V k =VD 1:3 (N k )

[0106] F V =DW(f z *V k ,f x *V k (7)

[0107] In the formula, V k Represents the convolution kernel N k via VD 1:3 Vertically expand the convolution by a 1:3 ratio to create a 3x7 vertical convolution kernel; F V This represents the branching features at the vertical scale.

[0108] Based on the cascaded double cross-correlation characteristic F Dw Horizontal scale branching feature F H and longitudinal scale branching feature F V The cross-correlation feature F of the entire cross-correlation matching network output is expressed as:

[0109] F=α1F H +α2F K +α3F DW(8)

[0110] In the formula, α1, α2, and α3 represent the cascaded double cross-correlation characteristics F. Dw Horizontal scale branching feature F H and longitudinal scale branching features F V The fusion coefficients, where the values ​​of fusion coefficients α1, α2 and α3 are obtained by optimizing the network training; Figure 5 In the middle, F H ∈R (C×H×W) F represents the output feature of the lateral scale branch with height H, width W, and number of channels C. V ∈R (C ×H×W) This represents the output feature of the vertical scale branch with height H, width W, and number of channels C.

[0111] Reference Figure 2 As shown, the bounding box encoding module of this embodiment includes multiple fully connected layers. The bounding box encoding module first transforms the target bounding box coordinates of the template image into a one-dimensional feature vector B, where B∈(x,y,w,h), (x,y) represents the corner coordinates of the target bounding box, w represents the width of the target bounding box, and h represents the height of the target bounding box. The feature vector B is encoded through multiple fully connected layers to obtain the bounding box encoded feature B. C Represented as:

[0112] B C =f C (B) (9)

[0113] In the formula: f C B represents a fully connected layer structure. C This represents the output feature of feature vector B after passing through a fully connected layer.

[0114] Then the cross-correlation feature F and the bounding box encoding feature B C Performing a broadcast addition operation yields the initial fusion feature F. b :

[0115] F b =F+B C (10)

[0116] Then, the initial fusion feature F b Perform 1x1 convolutional encoding to obtain the fused feature F. BM :

[0117] F BM =f g (F b (11)

[0118] In the formula, f gThis represents the convolutional encoding operation. Figure 2 F in BM ∈R (C×H×W) This indicates that the fusion features of the classification and regression network should be input.

[0119] Furthermore, such as Figure 1 As shown, the input to the classification and regression network is a fusion feature of size 25*25*1. After passing through a series of different convolutional layers, the final output is a classification score map of size 25*25*1 and a regression prediction map of size 25*25*4. In the classification and regression network, the classification branch is responsible for predicting the score when the target is the foreground, and the regression branch is responsible for predicting four distances (l, t, b, r), where l, t, b, and r represent the offset distances of the target center position from the four sides of the regression box, respectively.

[0120] As an embodiment of the present invention, in step c, the training process of the cross-correlation matching enhanced Siamese network includes:

[0121] Pairs of template images and search images are randomly selected from the training sample set and used as inputs to the two branches of the cross-correlation matching enhanced Siamese network;

[0122] Gradient backpropagation is performed using the SGD stochastic gradient descent method with momentum to optimize network parameters until the joint task loss function converges.

[0123] The formula for calculating the joint task loss function is as follows:

[0124] L=λ1L cls +λ2L reg (12)

[0125] In the formula, L cls L represents the binary cross-entropy loss function. reg Let λ1 and λ2 represent the IOU loss function, respectively. cls and L reg The weights are set as follows, where in this embodiment, λ1 = 1 and λ2 = 1;

[0126] Specifically, during training, this embodiment of the invention sets the batch size to 16 per iteration on a single GPU and uses SGD stochastic gradient descent with momentum for gradient backpropagation optimization. The total number of training rounds is 50. During training, the first five rounds use a learning rate gradually decreasing from 0.001 to 0.005 for warm-up training, and the subsequent forty-five rounds use a learning rate gradually decreasing from 0.005 to 0.00001 for training. After the tenth round of training, the parameters of the feature extraction network are no longer frozen and are optimized end-to-end along with the entire network. The weight decay coefficient and momentum parameter are set to 0.0001 and 0.9, respectively. During the training phase, the cropped template image and the search image are used as inputs to the network, sharing the same network and network parameters. The template image and the search image are simultaneously processed by the feature extraction network, the cross-correlation matching network, and the classification and regression network to obtain the final output classification score map and regression prediction map. Then, the entire network is trained and optimized end-to-end by combining the joint task loss function.

[0127] Furthermore, in the online tracking phase of this embodiment, the first frame of the test video sequence is first cropped and used as a template image to obtain template features in the feature extraction network. The obtained template features are then fixed within the network to avoid repeated feature extraction, thereby accelerating the network's testing speed. Simultaneously, the bounding box encoding module encodes the given bounding box information of the template frame for subsequent fusion with cross-correlation features. Next, each subsequent frame of the test video sequence is cropped with a region four times larger than the template image, with the center of the cropped region being the predicted target center point of the previous frame. This cropped region is then used as the search image and fed into the feature extraction network. After feature extraction via a ResNet50 deep residual network, it is processed with the previously obtained template features through a cross-correlation matching network and a classification regression network to obtain a classification score map and a regression prediction map. Finally, through a series of post-processing operations, at the position with the highest classification score, four offset regressions are performed on the regression prediction map to obtain the final predicted bounding box of the target in this frame. The entire testing process ends after the tracking of the last frame of the video sequence is completed or begins tracking of the next video sequence, updating the corresponding template features.

[0128] Furthermore, this invention was evaluated on the OTB100 dataset. The tracking performance of the method was assessed using two metrics: accuracy and area under the success curve. The mainstream tracking algorithms compared with this invention for performance evaluation included Ocean, ATOM, DaSiamRPN, GradNet, SiamRPN, CFNet, and SiamFC. The experimental results show that, in terms of success rate, this invention outperforms Ocean, ATOM, DaSiamRPN, GradNet, SiamRPN, CFNet, and SiamFC by 0.8, 1.7, 2.6, 4.5, 5.5, 9.7, and 10.2 percentage points, respectively; and in terms of accuracy, this invention outperforms Ocean, ATOM, DaSiamRPN, GradNet, SiamRPN, CFNet, and SiamFC by 0.9, 2.6, 2.7, 4.5, 5.9, 12.8, and 13.4 percentage points, respectively. Furthermore, this invention selected six scenarios under the OTB100 benchmark—occlusion, disappearance of view, motion blur, cluttered background, scale change, and deformation—to further test the tracking performance of the proposed method. Experiments showed that under various challenging factors, this invention still achieved excellent performance compared to other mainstream tracking algorithms. In particular, under deformation scenarios, this invention achieved a significant improvement in both success rate and accuracy. Therefore, it is evident that the design of the cross-correlation matching network is beneficial to tracking performance.

[0129] Reference Figure 6 As shown, a visual comparison of the present invention with ATOM, DaSiamRPN, and SiamRPN is presented in four OTB100 video sequences: Diving, Board, Girl2, and Jump. It can be seen that in the Diving video sequence, the target's pose changes significantly between consecutive video frames. ATOM, DaSiamRPN, and SiamRPN all produce large deviations in their predictions after the target deformation, failing to accurately revert the target bounding box to the target object. However, due to the introduction of horizontal and vertical scale branches in the cross-correlation matching network, the regression prediction of the present invention is more accurate, demonstrating its excellent scale adaptability. In the Board video sequence, in scenes with a cluttered background, the other three tracking algorithms produce incorrect predictions during tracking, demonstrating the good discriminative ability of the present invention. In the Girl2 video sequence, even after the target object is occluded, the present invention can still correctly maintain tracking of the target, demonstrating the tracker's good robustness. In the Jump video sequence, when the target object is moving rapidly, the present invention produces better prediction results than the other three tracking algorithms.

[0130] In summary, the target tracking method based on cross-correlation matching enhanced Siamese network provided by this invention can fully utilize existing prior information. By encoding the given target boundary ground truth box information into the network through a bounding box encoding module, the foreground response of the target region is enhanced. Through analysis and research on the characteristics of current cross-correlation structures, a novel cross-correlation cascade method is adopted, solving the inherent problems of existing cross-correlation matching, reducing irrelevant background and interference information, and improving the discrimination ability of the tracking network, thereby making the target position more accurate. Furthermore, by introducing unconventional scale convolutional kernels to extract and fuse template features and search features from multiple perspectives, a series of anchor box parameter adjustments are avoided, while obtaining more scale information. This invention can effectively address problems such as occlusion, disappearance of view, motion blur, cluttered background, and scale changes, and maintains good robustness during long-term tracking. It has the advantages of real-time performance and high accuracy, and is worthy of widespread application.

[0131] Example 2:

[0132] like Figure 7 As shown, this embodiment of the invention provides a target tracking system based on cross-correlation matching enhanced Siamese networks, which can be used to implement the method described in Embodiment 1, specifically including:

[0133] The cropping module is used to crop the video sequence of the target to be tracked, and obtain template images and search images for all frames.

[0134] The classification prediction module is used to input the template image and the search image into a constructed and trained cross-correlation matching enhanced Siamese network, extract template features and search features from the template image and the search image through the cross-correlation matching enhanced Siamese network; perform cross-correlation matching on the template features and the search features to obtain cross-correlation features; encode the bounding box information of the template image to obtain bounding box encoded features; and perform classification and regression calculations on the fused features of cross-correlation features and bounding box encoded features to obtain the corresponding classification score map and regression prediction map.

[0135] The target location acquisition module is used to obtain the final position of the target on the video sequence frames based on the position with the largest response value in the classification score map and the offset of the regression prediction map.

[0136] The target tracking method based on cross-correlation matching enhanced Siamese network provided in this embodiment of the invention is based on the same technical concept as the target tracking method based on cross-correlation matching enhanced Siamese network provided in Embodiment 1, and can produce the beneficial effects described in Embodiment 1. For the contents not described in detail in this embodiment, please refer to Embodiment 1.

[0137] Example 3:

[0138] This invention provides a computer-readable storage medium storing a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of any of the methods in Embodiment 1.

[0139] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0141] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0143] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A target tracking method based on cross-correlation matching enhanced Siamese network, characterized in that, The method includes: The video sequence of the target to be tracked is cropped to obtain template images and search images for all frames; The template image and the search image are input into a constructed and trained cross-correlation matching enhanced Siamese network. The cross-correlation matching enhanced Siamese network extracts template features and search features from the template image and the search image. Cross-correlation matching is performed on the template features and the search features to obtain cross-correlation features. Bounding box information is encoded into the template image to obtain bounding box encoded features. Classification and regression calculations are performed on the fused features of cross-correlation features and bounding box encoded features to obtain the corresponding classification score map and regression prediction map. Based on the position with the largest response value in the classification score map, combined with the offset in the regression prediction map, the final position of the target in the video sequence frame is obtained. The cross-correlation matching enhanced Siamese network includes a feature extraction network, a cross-correlation matching network, a classification and regression network, and a bounding box encoding module. The cross-correlation matching network includes a horizontal scale extraction module, a vertical scale extraction module, and a cascaded dual cross-correlation module; The cascaded dual cross-correlation module is used to perform pixel-matching cross-correlation on template features and search features, and then perform depth-separable cross-correlation on template features to obtain cascaded dual cross-correlation features. The calculation formula is as follows: ； （1）； In the formula, Representing template features and search features Pixel matching cross-correlation features, Indicates pixel-matching cross-correlation; This indicates the cascaded double cross-correlation characteristic. This indicates a deep separable cross-correlation; The lateral scale extraction module is used to extract the lateral scale branch features of the template features and the search features. Its calculation formula is as follows: ； （2）； In the formula: Indicates regular Convolution kernel, Represents convolution through A horizontal 3:1 convolution is expanded into a 7*3 horizontal convolution kernel; Indicates lateral scale branching features; Indicates the convolution operation; The vertical scale extraction module is used to extract the vertical scale branch features of the template features and the search features. Its calculation formula is as follows: ； （3）； In the formula, Represents the convolution kernel through Vertically expand the convolution by 1:3 to form a 3*7 vertical convolution kernel; This represents the branching features at the vertical scale.

2. The target tracking method based on cross-correlation matching enhanced Siamese network according to claim 1, characterized in that, The construction and training process of the cross-correlation matching enhanced Siamese network includes: Obtain the target video sequence frame dataset, crop each frame image in the dataset according to the target position and size of the image, and obtain the template image and search image of all frame images, which are used as the training sample set; A cross-correlation matching enhanced Siamese network is constructed, wherein the feature extraction network is an improved ResNet50 deep residual network. The improvement of the ResNet50 deep residual network includes: removing the fifth convolutional layer of the original ResNet50 deep residual network, setting the stride of the third and fourth convolutional layers to 1, and setting the size of the dilated convolutions of the third and fourth layers to 4. The constructed cross-correlation matching enhanced Siamese network is trained based on the training sample set to obtain the trained cross-correlation matching enhanced Siamese network.

3. The target tracking method based on cross-correlation matching enhanced Siamese network according to claim 2, characterized in that, The method for cropping each frame of the dataset includes: The first frame of the target video sequence is cropped into a template image of size 127*127*3 with the target as the center. Starting from the second frame, the subsequent frames of the target video sequence are cropped into a template image of size 255*255*3 with the target as the center.

4. The target tracking method based on cross-correlation matching enhanced Siamese network according to claim 1, characterized in that, Based on the cascaded dual cross-correlation feature Horizontal scale branching features and vertical scale branching features Calculate the cross-correlation features output by the cross-correlation matching network. for: （4）； In the formula, , and These represent the cascaded dual cross-correlation characteristics. Horizontal scale branching features and vertical scale branch features The fusion coefficient, the fusion coefficient , and The value is obtained by optimizing the network training.

5. The target tracking method based on cross-correlation matching enhanced Siamese network according to claim 4, characterized in that, The bounding box encoding module includes multiple fully connected layers; the steps of the bounding box encoding module to encode bounding box information of the template image include: Transform the target bounding box coordinates of the template image into a one-dimensional feature vector. ,in, ( , , , ), ( , () represents the coordinates of the corner points of the target bounding box. Indicates the width of the target bounding box. Indicates the height of the target bounding box; The feature vector After encoding the vector dimension through multiple fully connected layers, the bounding box encoded features are obtained. Represented as: （5）； In the formula: This indicates a fully connected layer structure. Representing the eigenvector Output characteristics after a fully connected layer.

6. The target tracking method based on cross-correlation matching enhanced Siamese network according to claim 5, characterized in that, The extraction steps for the fusion features of the cross-correlation features and bounding box encoding features include: The cross-correlation features bounding box encoded features Preliminary fusion features are obtained by performing broadcast addition operations. : （6）； Regarding the preliminary fusion features Perform 1x1 convolutional encoding to obtain fused features. : （7）； In the formula, This indicates a convolutional encoding operation.

7. The target tracking method based on cross-correlation matching enhanced Siamese network according to any one of claims 2 to 6, characterized in that, The steps for training the constructed cross-correlation matching enhanced Siamese network based on the training sample set include: Pairs of template images and search images are randomly selected from the training sample set and used as inputs to the two branches of the cross-correlation matching enhanced Siamese network; Gradient backpropagation is performed using the SGD stochastic gradient descent method with momentum to optimize network parameters until the joint task loss function converges. The formula for calculating the joint task loss function is as follows: （8）； In the formula, This represents the binary cross-entropy loss function. Represents the IOU loss function. and They are respectively and The weight.

8. A target tracking system based on cross-correlation matching enhanced Siamese network, characterized in that, The system includes: The cropping module is used to crop the video sequence of the target to be tracked, and obtain template images and search images for all frames. The classification prediction module is used to input the template image and the search image into a constructed and trained cross-correlation matching enhanced Siamese network, extract template features and search features from the template image and the search image through the cross-correlation matching enhanced Siamese network; perform cross-correlation matching on the template features and the search features to obtain cross-correlation features; encode the bounding box information of the template image to obtain bounding box encoded features; and perform classification and regression calculations on the fused features of cross-correlation features and bounding box encoded features to obtain the corresponding classification score map and regression prediction map. The target location acquisition module is used to obtain the final position of the target on the video sequence frame based on the position with the largest response value in the classification score map and the offset of the regression prediction map. The cross-correlation matching enhanced Siamese network includes a feature extraction network, a cross-correlation matching network, a classification and regression network, and a bounding box encoding module. The cross-correlation matching network includes a horizontal scale extraction module, a vertical scale extraction module, and a cascaded dual cross-correlation module; The cascaded dual cross-correlation module is used to perform pixel-matching cross-correlation on template features and search features, and then perform depth-separable cross-correlation on template features to obtain cascaded dual cross-correlation features. The calculation formula is as follows: ； （1）； In the formula, Representing template features and search features Pixel matching cross-correlation features, Indicates pixel-matching cross-correlation; This indicates the cascaded double cross-correlation characteristic. This indicates a deep separable cross-correlation; The lateral scale extraction module is used to extract the lateral scale branch features of the template features and the search features. Its calculation formula is as follows: ； （2）； In the formula: Indicates regular Convolution kernel, Represents convolution through A horizontal 3:1 convolution is expanded into a 7*3 horizontal convolution kernel; Indicates lateral scale branching features; Indicates the convolution operation; The vertical scale extraction module is used to extract the vertical scale branch features of the template features and the search features. Its calculation formula is as follows: ； （3）； In the formula, Represents the convolution kernel through Vertically expand the convolution by 1:3 to form a 3*7 vertical convolution kernel; This represents the branching features at the vertical scale.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the target tracking method based on cross-correlation matching enhanced Siamese network as described in any one of claims 1 to 7.

Images