A deep spatio-temporal correlation-based video target tracking method and system

Abstract

The application discloses a deep spatio-temporal correlation video target tracking method and system, which realizes target tracking for each video sequence and guarantees to obtain accurate video target tracking results. Firstly, a spatio-temporal feature extractor is designed to extract spatio-temporal features of a template sequence and a search sequence. Secondly, a feature matching module is introduced, which is composed of a classification branch and a regression branch. The extracted template spatio-temporal features and the search spatio-temporal features are subjected to similarity matching through correlation filtering to obtain multi-channel correlation filtering features respectively. Then, a target tracking module is deployed, which includes a classification head and a regression head, and the classification score map and the regression score map are calculated according to the input multi-channel correlation filtering features respectively, which are used to predict the target position and estimate the target scale. Finally, the spatio-temporal correlation visual tracking model is optimized by minimizing the defined joint loss. In the target tracking test, a confidence region estimation strategy is proposed, so that the target tracking is kept robust and accurate in the video sequence.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically, to a video target tracking method and system with deep spatiotemporal correlation. Background Technology

[0002] Video target tracking refers to a technique that uses contextual information from video or image sequences to model the appearance and motion information of a target, thereby predicting the target's motion state and pinpointing its location. Typically, based on a target specified in the first frame of a video, the technique continuously tracks that specific target in subsequent video frames to achieve target localization and scale estimation. Video target tracking has wide-ranging applications, including video surveillance, autonomous driving, and precision guidance.

[0003] In recent years, with the rapid development of deep learning and convolutional networks, an increasing number of video object trackers based on convolutional networks have emerged. Researchers increasingly favor Siam network-based trackers, which not only offer advantages in tracking speed but also achieve good accuracy. These Siam network-based trackers treat visual tracking as a similarity matching problem. In 2016, Bertinetto et al. proposed the SiamFC tracker for visual tracking (Luca Bertinetto, Jack Valmadre, ...). F. Henriques, Andrea Vedaldi, Philip HSTorr: Fully-Convolutional Siamese Networks for Object Tracking. ECCV Workshops(2)2016:850-865.), which uses Siamese networks to extract templates and search features, and uses correlation filtering to calculate the cross-correlation between the target template and the search region. Subsequently, Held et al. proposed the GOTURN tracker (David Held, Sebastian Thrun, Silvio Savarese: Learning to Track at 100FPS with Deep Regression Networks. ECCV(1)2016:749-765.), which regresses the target box predicted in the previous frame to obtain the target box in the current frame.

[0004] To further improve accuracy, in 2018, Li et al. combined Siamese networks and region proposal networks to propose the SiamRPN tracker (Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, Xiaolin Hu: High Performance Visual Tracking With Siamese Region Proposal Network. CVPR 2018:8971-8980.). By introducing a region proposal network, the target box scale is estimated more accurately. However, the introduction of anchor boxes can easily lead to ambiguity in similarity matching, thus affecting tracking accuracy, causing error accumulation, reducing the robustness of the target tracker, and introducing more hyperparameters. In 2020, Chen et al. designed a simple and efficient anchor-box-less tracker, SiamBAN (Zedu Chen, Bineng Zhong, Guorong Li, Shengping Zhang, Rongrong Ji: Siamese Box Adaptive Network for Visual Tracking. CVPR 2020:6667-6676.), which improves the tracker's performance by adding a feature combination module branch and a quality evaluation branch. These trackers demonstrate excellent performance and real-time tracking speed in most video scenarios. However, existing methods often treat video target tracking as a frame-by-frame target detection problem, neglecting the rich spatiotemporal information between video frames. Siamese network-based visual tracking methods should effectively utilize the rich information across time frames and better learn spatiotemporal features to model the target's appearance, thereby improving the accuracy of tracking and localization. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a deep spatiotemporal correlation video target tracking method and system. By utilizing spatiotemporal information, the tracker not only preserves spatiotemporally correlated feature information but also, by storing features of different template frames using template sequences, can better model the appearance of video targets, improving tracker accuracy. Simultaneously, by using the template sequence and search sequence as input, the target prediction results in the search sequence can be obtained. This sequence-by-sequence processing method for video target tracking significantly improves the speed of video target tracking.

[0006] To achieve the above objectives, the present invention provides a video target tracking method based on deep spatiotemporal correlation, comprising the following steps:

[0007] S1. Construct a network architecture consisting of a spatiotemporal feature extractor, a feature matching subnetwork, and a target prediction subnetwork. This network architecture improves the model's detection and localization capabilities, resulting in more accurate video target tracking results, including:

[0008] S11. The spatiotemporal feature extractor based on 3D twin network includes a template branch and a search branch. It uses a 3D fully convolutional neural network as the base network and shares weights. It is used to extract template spatiotemporal features and search spatiotemporal features from the input template sequence block and search sequence block.

[0009] S12. The feature matching subnetwork consists of a classification branch and a regression branch, which take the template spatiotemporal features and the search spatiotemporal features as inputs, respectively. It uses correlation filtering to perform feature similarity matching and obtains multi-channel correlation filtering features.

[0010] S13. The target prediction sub-network includes a classification head and a regression head. It takes multi-channel correlation filter features as input and obtains the classification score map and regression score map through the classification head and regression head, respectively.

[0011] S2. Given template sequence video frames and search sequence video frames respectively, cut them into template sequence blocks and search sequence blocks as input to the entire network architecture;

[0012] S3. Construct a spatiotemporal feature extractor. This sub-network is a 3D twin fully convolutional network, including a template branch and a search branch, using a 3D fully convolutional network as the base network and sharing weights. Taking template sequence blocks and search sequence blocks as input, the spatiotemporal feature extractor extracts template spatiotemporal features and search spatiotemporal features from them.

[0013] S4. Construct a feature matching sub-network, including a classification branch and a regression branch. Use the obtained template spatiotemporal features and search spatiotemporal features as inputs to these two branches, and perform feature similarity matching using correlation filtering operations, so that the classification branch and regression branch output multi-channel correlation-filtered features respectively.

[0014] S5. Construct a target prediction sub-network, mainly composed of a classification head and a regression head. Input the multi-channel correlation filter features output from the classification and regression branches into the corresponding classification and regression heads to obtain the classification score map and regression score map;

[0015] S6. Locate the target position in each video frame of the sequence based on the classification score map; estimate the target scale in each video frame of the sequence based on the regression score map to obtain the target prediction box in each video frame of the search sequence.

[0016] S7. The network model is optimized by minimizing the joint loss, including the cross-entropy loss for classification and the cross-union ratio loss for regression, and finally the video target tracker model is obtained.

[0017] S8. Using the trained network model as a visual tracker, perform target tracking on a video-by-video sequence basis. To ensure stable and accurate tracking, a confidence search region estimation strategy is defined. The search region for the next sequence is cropped based on the different target states in the current video sequence to reduce error accumulation and accurately locate the target in each video frame of the search sequence.

[0018] This invention provides an end-to-end trainable neural network architecture and system for video target tracking, including a video sequence input module for cropping template sequence blocks and search sequence blocks; a model training module for training a video target tracker based on a 3D twin network, training the target tracking by minimizing the combined loss, including cross-entropy and cross-union loss, ultimately achieving target tracking per video sequence; a video target tracking module that uses the classification map and regression map output by the model to estimate the target state and predict the scale in the search sequence video frames, respectively, and calculates the target prediction box in the search sequence; using the target prediction box of the current video sequence, it calculates the confidence search region of the next set of video sequences and inputs it into the search branch for target tracking of subsequent video sequences.

[0019] Compared with existing technologies, it has the following beneficial effects:

[0020] This invention utilizes a 3D twin fully convolutional network to extract template spatiotemporal features and search spatiotemporal features, learning rich spatiotemporal information across multiple consecutive video frames. The extracted template spatiotemporal features and search spatiotemporal features are input into a feature matching sub-network to obtain multi-channel correlation-filtered features. Classification and regression heads are used to process the multi-channel correlation-filtered features, predicting target localization and estimating target scale. In the target tracking stage, to obtain a more accurate search sequence region, a confidence search region estimation strategy is defined. The next search sequence region is estimated based on different states of the target in the current video sequence, ensuring the stability and accuracy of target tracking. This method departs from traditional single-frame processing and directly processes the video sequence, improving tracking speed while maintaining tracking accuracy. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is the overall network structure diagram in this invention patent.

[0023] Figure 2This is a schematic diagram of the template sequence block and the search sequence block in this invention patent.

[0024] Figure 3 This is a schematic diagram of the spatiotemporal feature extractor structure in this invention patent.

[0025] Figure 4 This is the confidence search region estimation map in this invention patent.

[0026] Figure 5 This is a schematic diagram of some video frames in this invention patent.

[0027] Figure 6 This is a schematic diagram of the video target tracking result in this invention patent. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other. The invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0029] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. A video target tracking method with deep spatiotemporal correlation includes steps S1 to S8:

[0030] S1. Construct a network architecture consisting of a spatiotemporal feature extractor, a feature matching subnetwork, and a target prediction subnetwork. This network architecture improves the model's detection and localization capabilities, resulting in more accurate video target tracking results, including:

[0031] S11. The spatiotemporal feature extractor based on 3D twin network includes a template branch and a search branch. It uses a 3D fully convolutional neural network as the base network and shares weights. It is used to extract template spatiotemporal features and search spatiotemporal features from the input template sequence block and search sequence block.

[0032] S12. The feature matching subnetwork consists of a classification branch and a regression branch, which take the template spatiotemporal features and the search spatiotemporal features as inputs, respectively. It uses correlation filtering to perform feature similarity matching and obtains multi-channel correlation filtering features.

[0033] S13. The target prediction sub-network includes a classification head and a regression head. It takes multi-channel correlation filter features as input and obtains the classification score map and regression score map through the classification head and regression head, respectively.

[0034] S2. Given template sequence video frames and search sequence video frames respectively, cut them into template sequence blocks and search sequence blocks as input to the entire network architecture;

[0035] S3. Construct a spatiotemporal feature extractor. This sub-network is a 3D twin fully convolutional network, including a template branch and a search branch, using a 3D fully convolutional network as the base network and sharing weights. Taking template sequence blocks and search sequence blocks as input, the spatiotemporal feature extractor extracts template spatiotemporal features and search spatiotemporal features from them.

[0036] S4. Construct a feature matching sub-network, including a classification branch and a regression branch. Use the obtained template spatiotemporal features and search spatiotemporal features as inputs to these two branches, and perform feature similarity matching using correlation filtering operations, so that the classification branch and regression branch output multi-channel correlation-filtered features respectively.

[0037] S5. Construct a target prediction sub-network, mainly composed of a classification head and a regression head. Input the multi-channel correlation filter features output from the classification and regression branches into the corresponding classification and regression heads to obtain the classification score map and regression score map;

[0038] S6. Locate the target position in each video frame of the sequence based on the classification score map; estimate the target scale in each video frame of the sequence based on the regression score map to obtain the target prediction box in each video frame of the search sequence.

[0039] S7. The network model is optimized by minimizing the joint loss, including the cross-entropy loss for classification and the cross-union ratio loss for regression, and finally the video target tracking model is obtained.

[0040] S8. Using the trained network model as a visual tracker, perform target tracking on a video-by-video sequence basis. To ensure stable and accurate tracking, a confidence search region estimation strategy is defined. The search region for the next sequence is cropped based on the different target states in the current video sequence, reducing error accumulation and accurately locating the target in each video frame of the search sequence. The following is a detailed description of each step.

[0041] In step S1, the network architecture is constructed, such as... Figure 1 As shown, the network consists of a spatiotemporal feature extractor, a feature matching subnetwork, and a target prediction subnetwork.

[0042] In step S2, given template sequence video frames and search sequence video frames respectively, they are cropped into template sequence blocks and search sequence blocks, such as... Figure 2 As shown, this serves as the input to the overall network architecture. The specific steps are as follows:

[0043] S21. Given a template sequence, obtain the center position, width, and height information of the target based on the real value information of the target in each video frame of the template sequence, and represent them as (x, y, w, h).

[0044] S211. Based on the real target box information given in S21, calculate the augmentation value p = (w + h) / 2 for the width and height of the target box, and calculate the scaling factor. This is used to scale the expanded target bounding box region. If the target bounding box region, after adding the expansion value, exceeds the boundary values ​​of the video frame, it is filled with the average RGB value of the current video frame. Ultimately, each video frame in the template sequence is cropped into a 127×127 template block.

[0045] S212. After cropping each video frame in the template sequence, a template block is obtained. Where k represents the total number of video frames in the template sequence.

[0046] S22. Given a search sequence, obtain the center position, width, and height information of the target based on the real value information of the target in the first video frame of the template sequence, and represent it as (X,Y,W,H).

[0047] S221. Based on the real target bounding box information given in S22, calculate the expansion value P = (W + H) / 2 for the width and height of the target bounding box, and calculate the scaling factor. This is used to scale the expanded target bounding box region. If the target bounding box region, after adding the expansion value, exceeds the boundary value of the video frame, it is filled with the average RGB value of the current video frame. Finally, each video frame in the search sequence is cropped into a 255×255 search block.

[0048] S222. After cropping each video frame in the search sequence, a search block is obtained. Where k represents the total number of video frames in the search sequence.

[0049] In step S3, the spatiotemporal feature extractor is a 3D twin fully convolutional network, including a template branch and a search branch, using a 3D fully convolutional network as the base network and sharing weights. Taking the template sequence block and the search sequence block as input, the spatiotemporal feature extractor extracts the template spatiotemporal features and the search spatiotemporal features from them. The specific steps are as follows:

[0050] S31. Construct a feature extraction network, such as Figure 3 As shown, each branch is a Res3D network consisting of five residual blocks.

[0051] S32. Modify the padding attribute of the first residual block of Res3D to 1×4×4 and adjust the stride to 1×1×1. Modify the output channels of the fourth residual block and the input channels of the fifth residual block to 128 respectively. Remove the downsampling and final classification layer of the fifth residual block. As a result, the output spatiotemporal features have the same time length as the input video sequence.

[0052] S34. Input the template block and search block obtained in steps S212 and S222 into the spatiotemporal feature extractor to obtain the template spatiotemporal features respectively. and search spatiotemporal features

[0053] In step S4, the feature matching subnetwork includes a classification branch and a regression branch. The obtained template spatiotemporal features and search spatiotemporal features are used as inputs to these two branches. Correlation filtering is used for feature similarity matching, resulting in the classification and regression branches outputting multi-channel correlation-filtered features. The specific steps are as follows:

[0054] S41. The template features obtained in S3 and search features The data are input into the classification and regression branches respectively, and correlation filtering is performed. The specific calculations are as follows:

[0055]

[0056] in, Indicates a category branch, * indicates the regression branch, and * indicates the correlation filter.

[0057] S42, the classification branch and the regression branch respectively output multi-channel correlation filter features. and

[0058] In step S5, the target prediction module includes a classification head and a regression head. The multi-channel correlation filter features output from the classification and regression branches are input into the classification head and regression head to obtain the classification score map and regression score map, respectively. The specific steps are as follows:

[0059] S51, the classification head consists of a 1×1 convolutional layer, using the multi-channel correlation filter feature F output from the classification branch in S42. cls As input to the classification head, the output is a classification score image:

[0060] S52, the regression head consists of a 1×1 convolutional layer, using the multi-channel correlation filter feature F output from the regression branch in S42. reg As input to the regression head, the output is a regression score plot:

[0061] In step S6, the location of the target in each video frame of the sequence is located based on the classification score map; the target scale in each video frame of the sequence is estimated based on the regression score map, resulting in the target prediction box for each video frame in the search sequence. The specific steps are as follows:

[0062] S61. The size of the classification score map is k×17×17. Find the point with the largest response value in the classification score map. In the original video frame, it is represented as: Where s = 8 is the total step size of the entire network.

[0063] S62. The regression score plot is a four-channel vector with a size of k×4×17×17. Using l... i ,t i ,r i ,b i This represents the offset of the regression target. The target's coordinate information can be represented as:

[0064]

[0065] in Represents target prediction box B i The coordinates of the top left and bottom right corners.

[0066] In step S7, the network model is optimized by minimizing the joint loss, including the cross-entropy loss for classification and the cross-union ratio loss for regression, ultimately resulting in a fast and accurate video object tracker. The specific steps are as follows:

[0067] S71, The total training loss is defined as:

[0068]

[0069] Among them, L i Let be the loss for the i-th search frame. Let k represent the total number of classification score maps (regression score maps). This represents the probability that the position (x, y) in the i-th search block belongs to the target. This represents the distance of position (x, y) in the i-th regression score plot from the edges of the bounding box.

[0070] S72, Training Loss L i The cross-entropy loss for classification and the intersection-union ratio loss for regression are defined as follows:

[0071]

[0072] Here, 1{·} is an indicator function that indicates whether something belongs to the target; if it does, it is assigned a value of 1, otherwise it is assigned a value of 0. L cls L represents the cross-entropy loss for classification.reg This represents the intersection-union ratio loss of the regression. If the current position (x... i ,y i If the current position belongs to the positive sample, that is, the target is located at the current position, then... Assign a value of 1; if it is a negative sample, then... The value is assigned to 0. Indicates the center position (x) of the true target in the i-th search block. i ,y i Offset from the perimeter of the bounding box.

[0073] In step S8, the trained network model is used as a visual tracker to perform target tracking on a video-by-video sequence basis. To ensure stable and accurate tracking, a confidence search region estimation strategy is defined. The search region for the next sequence is cropped based on the different target states in the current video sequence, reducing error accumulation and accurately locating the target in each video frame of the search sequence. The specific steps are as follows:

[0074] S81. Due to the possibility of significant positional changes of the target within the video sequence, based on the predicted bounding box results of the current search sequence {B t-k ,..,B t-1 B t}, where B t It is the predicted bounding box of the target in the t-th frame of the search sequence, based on the coordinates of the top-left corner of each bounding box. and the coordinates of the bottom right corner The minimum bounding box b is calculated. m ,like Figure 4 As shown.

[0075] S82, for the smallest bounding box b m Expand the search region b used to crop the next set of video sequences. s This ensures that the search area covers the target in every video frame of the search sequence. The video target tracking results are as follows: Figure 6 As shown.

[0076] According to another aspect of this application, a deep spatiotemporal correlation video target tracking system is also provided, comprising the following modules:

[0077] Video sequence input module: Given a set of template sequence video frames and search sequence video frames, trim them into template sequence blocks and search sequence blocks of specified sizes according to the form in S2.

[0078] The model training module is used to train a video target tracker based on a 3D twin network. This target tracker includes a spatiotemporal feature extractor module, a feature matching module, and a target prediction module. The spatiotemporal feature extractor takes template sequence blocks and search sequence blocks as input, extracting template spatiotemporal features and search spatiotemporal features from them. These spatiotemporal features are then input into the feature matching module, where similarity matching is performed using correlation filtering to obtain multi-channel correlation-filtered features. These features are then sequentially input into the classification head and regression head of the target prediction module, ultimately obtaining a classification score map and a regression map. The target tracking is trained by minimizing the cross-entropy loss of classification and the intersection-union ratio loss of regression.

[0079] During the testing phase, the video target tracking module uses the classification and regression maps output by the model to estimate the target state and predict the scale in the video frames of the search sequence, thereby obtaining the target prediction boxes in the search sequence. Then, using these target prediction boxes, a set of confidence search regions is obtained and input into the search branch for target tracking in subsequent sequence frames.

[0080] This system is used to implement the functions of the methods in the above embodiments. The specific implementation steps of the methods involved in the system module have been described in the methods and will not be repeated here.

[0081] In this embodiment, a spatiotemporal feature extractor is first designed to extract spatiotemporal features from the input template sequence and search sequence, learning a highly discriminative target appearance representation. Secondly, a feature matching module is designed, consisting of a classification branch and a regression branch. By utilizing correlation filtering, the extracted template spatiotemporal features and search spatiotemporal features are similarly matched in a high-dimensional feature space to obtain multi-channel correlation-filtered features. Then, a target tracking module is deployed, including a classification head and a regression head, which calculate classification and regression score maps based on the input multi-channel correlation-filtered features, respectively, to predict target location and estimate target scale. Finally, the spatiotemporal correlation visual tracking model is optimized by minimizing a defined joint loss and used for video target tracking prediction. In target tracking testing, a confidence region estimation strategy is defined to calculate the search region for the next video sequence based on the target tracking results of the current video sequence, minimizing error accumulation and thus maintaining robust and accurate target tracking in the video sequence.

[0082] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A video target tracking method with deep spatiotemporal correlation, characterized in that, The method is executed by a computer and includes the following steps: S1. Construct a network architecture consisting of a spatiotemporal feature extractor, a feature matching subnetwork, and a target prediction subnetwork. This network architecture improves the model's detection and localization capabilities, resulting in more accurate video target tracking results, including: S11. The spatiotemporal feature extractor based on 3D twin network includes a template branch and a search branch. It uses a 3D fully convolutional neural network as the base network and shares weights. It is used to extract template spatiotemporal features and search spatiotemporal features from the input template sequence block and search sequence block. S12. The feature matching subnetwork consists of a classification branch and a regression branch, which take the template spatiotemporal features and the search spatiotemporal features as inputs, respectively. It uses correlation filtering to perform feature similarity matching and obtains multi-channel correlation filtering features. S13. The target prediction sub-network includes a classification head and a regression head. It takes multi-channel correlation filter features as input and obtains the classification score map and regression score map through the classification head and regression head, respectively. S2. Given template sequence video frames and search sequence video frames respectively, cut them into template sequence blocks and search sequence blocks as input to the entire network architecture; S3. Construct a spatiotemporal feature extractor; the spatiotemporal feature extractor is a 3D twin fully convolutional network, including a template branch and a search branch, using a 3D fully convolutional network as the base network and sharing weights; taking the template sequence block and the search sequence block as input, the spatiotemporal feature extractor extracts the template spatiotemporal features and the search spatiotemporal features from them; S4. Construct a feature matching subnetwork, including a classification branch and a regression branch; use the obtained template spatiotemporal features and search spatiotemporal features as inputs to these two branches, and use correlation filtering to perform feature similarity matching, so that the classification branch and the regression branch output multi-channel correlation filtering features respectively; S5. Construct a target prediction sub-network, mainly composed of a classification head and a regression head; input the multi-channel correlation filter features output by the classification branch and the regression branch into the corresponding classification head and regression head to obtain the classification score map and the regression score map; S6. Locate the target position in each video frame of the sequence based on the classification score map; estimate the target scale in each video frame of the sequence based on the regression score map to obtain the target prediction box in each video frame of the search sequence. S7. The network model is optimized by minimizing the joint loss, including the cross-entropy loss for classification and the cross-union ratio loss for regression, and finally the video target tracker model is obtained. S8. Using the trained network model as a visual tracker, target tracking is performed on a video-by-video sequence basis for a given video. To ensure stable and accurate tracking, a confidence search region estimation strategy is defined. The search region of the next sequence is cropped according to the different target states in the current video sequence to reduce error accumulation and accurately locate the target in each video frame of the search sequence.

2. The video target tracking method with deep spatiotemporal correlation as described in claim 1, characterized in that... The specific implementation process for constructing template sequence blocks and search sequence blocks is as follows: S21. Given a template sequence, based on the target's real value information in each video frame of the template sequence, obtain the target's center position, width, and height information, and represent them as follows: ; S211. Based on the real target box information given in S21, calculate the augmentation values ​​for the width and height of the target box. And calculate the scaling factor. , used to scale the expanded target bounding box area; If the target bounding box region, after adding the augmentation value, exceeds the boundary values ​​of the video frame, then the average RGB value of the current video frame is used for padding; ultimately, each video frame in the template sequence is cropped to... Template blocks of a certain size; S212. After cropping each video frame in the template sequence, a template block is obtained. ,in This indicates the total number of video frames in the template sequence; S22. Given a search sequence, based on the real value information of the target in the first video frame of the template sequence, obtain the center position, width, and height information of the target, and represent them as follows: ; S221. Based on the real target bounding box information given in S22, calculate the augmentation values ​​for the width and height of the target bounding box. And calculate the scaling factor. This is used to scale the expanded target bounding box area; If the target bounding box region, after adding the augmentation value, exceeds the boundary values ​​of the video frame, then the average RGB value of the current video frame is used to fill it. Ultimately, each video frame in the search sequence is cropped to... The size of the search block; S222. After cropping each video frame in the search sequence, a search block is obtained. ,in This represents the total number of video frames in the search sequence.

3. The video target tracking method with deep spatiotemporal correlation as described in claim 2, characterized in that... The spatiotemporal feature extractor is constructed, and its specific implementation process is as follows: S31. Construct a feature extraction network, where each branch is a Res3D network consisting of five residual blocks; S32. Modify the padding property of the first residual block in Res3D to... stride adjusted to The output channels of the fourth residual block and the input channels of the fifth residual block are modified to 128 respectively, and the downsampling and final classification layers of the fifth residual block are removed; thus, the spatiotemporal features of the output have the same time length as the input video sequence. S33. Input the template block and search block obtained in S212 and S222 into the spatiotemporal feature extractor to obtain the template spatiotemporal features respectively. and search spatiotemporal features .

4. The video target tracking method with deep spatiotemporal correlation as described in claim 3, characterized in that... The feature matching subnetwork is constructed, and its specific implementation process is as follows: S41. The template features obtained in S3 and search features The data are input into the classification and regression branches respectively, and correlation filtering is performed. The specific calculations are as follows: ， (1) ， (2) in, Indicates a category branch, Indicates the regression branch, Indicates correlation filtering; S42, the classification branch and the regression branch respectively output multi-channel correlation filter features. and .

5. The video target tracking method with deep spatiotemporal correlation as described in claim 4, characterized in that... The specific implementation process of constructing a target tracking subnetwork for video sequences is as follows: S51, The classification header consists of one The convolutional layers consist of multi-channel correlation-filtered features output from the classification branch in S42. As input to the classification head, the output is a classification score image: ; S52, Return Header is One The convolutional layers consist of multi-channel correlation filtering features output from the regression branch in S42. As input to the regression head, the output is a regression score plot: .

6. The video target tracking method with deep spatiotemporal correlation as described in claim 1, characterized in that... The specific implementation process for predicting the target location and estimating the bounding box scale is as follows: S61. The size of the classification score chart is: Find the point with the largest response value in the classification score map. In the original video frame, it is represented as: ,in This represents the total step size of the entire network. S62. The regression score plot is a four-channel vector with the following size: ,use , , , This represents the offset of the regression target. The target's coordinate information can be represented as: ，(3) in , Represents the target prediction box The coordinates of the top left and bottom right corners.

7. The video target tracking method with deep spatiotemporal correlation as described in claim 1, characterized in that... The specific implementation process for training the visual tracking model is as follows: S71, The total training loss is defined as: ，(4) in, For the first The loss of each search frame; This is represented by the total number of classification score plots (regression score plots); Indicates the first In each search block The probability that the location belongs to the target; Indicates the first Position in the regression score graph Distance from the perimeter of the bounding box; S72, Training Loss The cross-entropy loss for classification and the intersection-union ratio loss for regression are defined as follows: ，(5) in, It is an indicator function that indicates whether something belongs to the target; if it does, it is assigned a value of 1, otherwise it is assigned a value of 0. The cross-entropy loss represents the classification. This represents the intersection-union ratio loss of the regression; if the current position If it belongs to a positive sample, meaning the current position belongs to the target, then... Assign a value of 1; if it is a negative sample, then... The value is assigned to 0; Indicates the first The center position of the real target in each search block Offset from the perimeter of the bounding box.

8. The video target tracking method with deep spatiotemporal correlation as described in claim 1, characterized in that... The specific implementation process for estimating the confidence search region is as follows: S81. Since the target may change position significantly in the video sequence, based on the predicted bounding box results of the current search sequence... ,in It is the first in the search sequence The predicted bounding boxes for each frame are determined based on the coordinates of the top-left corner of each bounding box. and the coordinates of the bottom right corner Calculate the minimum bounding box ; S82, Minimum bounding box Expand the search area for cropping the next set of video sequences. This ensures that the search area can cover the target in every video frame of the search sequence.

9. A video target tracking system with deep spatiotemporal correlation, characterized in that, Performing the steps in claim 1 includes the following modules: Image labeling module: Given an image as input, it randomly labels pixels based on the image's real labels, including foreground and background labels, thereby generating a large amount of interactive information to simulate user interaction; Video sequence input module: Given a set of template sequence video frames and search sequence video frames, trim them into template sequence blocks and search sequence blocks of specified sizes according to the form in S2; Model training module: used to train a video target tracker based on a 3D twin network; the target tracker includes a spatiotemporal feature extractor module, a feature matching module, and a target prediction module; the spatiotemporal feature extractor takes template sequence blocks and search sequence blocks as input, and extracts template spatiotemporal features and search spatiotemporal features from them; these spatiotemporal features are input into the feature matching module, and similarity matching is performed using correlation filtering operations to obtain multi-channel correlation filtered features; The data is then sequentially input into the classification head and regression head of the target prediction module to obtain the classification score map and regression map. The target tracking is trained by minimizing the cross-entropy loss of classification and the intersection-union ratio loss of regression. Video target tracking module: During the testing phase, the classification map and regression map output by the model are used to estimate the target state and predict the scale in the video frames of the search sequence, respectively, so as to obtain the target prediction box in the search sequence. Then, using this set of target prediction boxes, a set of confidence search regions are obtained and input into the search branch for target tracking in subsequent sequence frames.

Images