Target tracking method based on spatiotemporal information fusion
By combining the spatiotemporal information fusion method of MDNet and LSTM networks, the deviation and loss problems of target tracking in the prior art under severely changing scenarios are solved, and accurate tracking and robustness improvement under occlusion and deformation conditions are achieved.
Patent Information
- Application Number
- CN202210258264.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-16
- Publication Date
- 2025-03-25
- Estimated Expiration
- 2042-03-16
AI Technical Summary
When the target and background change drastically, it is difficult to effectively utilize the target motion information in the time dimension, resulting in tracking deviations or loss.
The target tracking method based on spatiotemporal information fusion is adopted, combined with the classic tracking methods MDNet and LSTM network, and the target appearance information in the spatial dimension and the target motion information in the temporal dimension are fused, and the final tracking result is selected through cosine similarity judgment.
Under occlusion, deformation and low resolution conditions, the target can be accurately tracked, which improves the robustness of the tracker, can effectively deal with target appearance changes, and reduces the situation of loss.
Smart Images
Figure CN114612517B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of video target tracking and relates to a target tracking method based on spatiotemporal information fusion. Background Art
[0002] Video target tracking is an important branch of computer vision. The task of video target tracking is to determine the position, shape or area of the target in continuous video image frames, and to determine the target's motion information such as speed, direction and trajectory. Target tracking has important research significance and broad application prospects, and is mainly used in video surveillance, human-computer interaction, intelligent transportation and other aspects.
[0003] Most existing target tracking methods use the target appearance information in the spatial dimension for tracking, and with the rapid development of deep learning, the feature extraction ability of target tracking methods has become stronger and stronger, which enables target tracking methods to better cope with changes in target appearance. However, when the target and background change dramatically, the target appearance information in the spatial dimension becomes less reliable. If the target appearance information is continued to be used for tracking, it is easy to lose the target. Therefore, it is necessary to use the target motion information in the time dimension to predict the target position at the current moment. However, when there are scenes with more drastic camera motion in the video, using the target motion information in the time dimension for prediction will result in deviations, and even lose the target. Therefore, how to reasonably use the target appearance information in the spatial dimension and the target motion information in the time dimension is a major problem to be solved in the field of video target tracking. Summary of the invention
[0004] The purpose of the present invention is to provide a target tracking method based on spatiotemporal information fusion, which solves the problem that the existing tracking methods only use the target appearance information in the spatial dimension but cannot well cope with the feature changes of the target in the temporal dimension.
[0005] The technical solution adopted by the present invention is:
[0006] The target tracking method based on spatiotemporal information fusion has the following specific steps:
[0007] Step 1: Select the classic tracking method MDNet as the base tracker, where MDNet uses the trained VGG-M as the feature extraction network; select the trained LSTM network as the target position predictor;
[0008] Step 2: For the video to be tracked, the target state of the first frame is Input into the feature extraction network to obtain the deep features of the first frame target And the base tracker parameters are initialized using the target state of the first frame;
[0009] Step 3: Enter the subsequent frame and obtain the target position and target width and height (w t-1 ,h t-1 ) constitutes the target state of the previous frame and inputs it into the base tracker for tracking, and obtains the tracking result based on the target appearance features;
[0010] Step 4, using the target state set of the previous L frames obtained by the method in step 3 to input into the target position predictor, and obtain the prediction result based on the target time series information;
[0011] Step 5: Input the tracking results based on the target appearance features and the prediction results based on the target timing information into the feature extraction network to obtain the corresponding deep features. and
[0012] Step 6: Calculate the depth features separately and and the first frame target state The deep features The cosine similarity of is calculated and one of them is selected as the final tracking result.
[0013] The present invention is also characterized in that:
[0014] The VGG-M network in step 1 consists of three convolutional layers and three fully connected layers connected in sequence, where the first convolutional layer and the second convolutional layer are respectively connected to a pooling layer, and each convolutional layer is followed by a random dropout layer and a RELU nonlinear activation function.
[0015] MDNet defines target tracking as a binary classification problem of target and background. It uses the deep features of VGG-M images and finally outputs the probability of the image being classified as the target and the background.
[0016] Both LSTM network and VGG-M are trained using the ILSVRC2015 dataset.
[0017] Step 6 is as follows:
[0018] Step 6.1, deep features and and the first frame target state The deep features The cosine similarity of is:
[0019]
[0020] where g track Depth feature representing the tracking result of the current frame based on the target appearance feature And the first frame target depth feature Cosine similarity;
[0021]
[0022] where g pred Represents the deep features of the prediction result of the current frame based on the target timing information And the first frame target depth feature Cosine similarity;
[0023] Step 6.2, the tracking result is output in which case by deciding the two similarities. The formula is:
[0024]
[0025] in This is the final tracking result.
[0026] Step 6.3, input the tracking result of the current frame into step 3, take the next frame as the current frame and perform image tracking of the next frame until all image frames in the video are tracked.
[0027] The beneficial effects of the present invention are:
[0028] 1. The present invention makes full use of the complementarity of the target's spatial information and temporal information, so that the tracker can accurately track the target under occlusion, deformation and low-resolution conditions.
[0029] 2. The present invention proposes a fusion mechanism based on spatial information and temporal information, which enables the tracker to cope well with changes in target appearance and improves the robustness of the tracker. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 is a flow chart of a target tracking method based on spatiotemporal information fusion of the present invention;
[0031] Figure 2 It is a network structure diagram of the target tracking method based on spatiotemporal information fusion of the present invention;
[0032] Figure 3 is the tracking result of the 19th frame to the 48th frame when the tracking object is a walking pedestrian in Example 1 of the present invention;
[0033] Figure 4 is the tracking result between the 20th frame and the 98th frame when the tracking object is a walking pedestrian in Example 1 of the present invention; DETAILED DESCRIPTION
[0034] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0035] Target tracking methods based on spatiotemporal information fusion, such as Figure 1, the specific steps are as follows:
[0036] Step 1: Select the classic tracking method MDNet as the base tracker, where MDNet uses the trained VGG-M as the feature extraction network f(·), and the feature extraction network f(·) is used to extract appearance features;
[0037] Select the trained LSTM network as the target position predictor;
[0038] Step 2: For the video to be tracked, first obtain the center point coordinates (x1, y1) of the target in the first frame and the width and height (w1, h1) of the target, and use these to form the state variables of the target. Then the target state of the first frame Input into the feature extraction network f(·) to obtain the deep features of the first frame target And using the target state of the first frame Initialize base tracker parameters;
[0039] Step 3: Enter the subsequent frame and obtain the target position (x t-1 ,y t-1 ) and target width and height (w t-1 ,h t-1 ) constitutes the target state of the previous frame And input the base tracker for tracking, and obtain the tracking result based on the target appearance features
[0040] Step 4: Use the method in step 3 to obtain the target state set of the first L frames Input the target position predictor to get the prediction result based on the target time series information
[0041] Step 5: The tracking results based on the target appearance features are And the prediction results based on the target time series information Input into the feature extraction network to obtain the corresponding deep features and
[0042] Step 6: Calculate the depth features separately and and the first frame target state The deep features The cosine similarity of is calculated and one of them is selected as the final tracking result.
[0043] In step 1, VGG-M consists of three convolutional layers and three fully connected layers connected in sequence, where the first convolutional layer and the second convolutional layer are respectively connected to a pooling layer. A random dropout layer and a RELU nonlinear activation function are added after each convolutional layer.
[0044] The network structure and parameters of VGG-M are shown in Table 1:
[0045] Table 1 Feature extraction network parameters
[0046]
[0047] The ILSVRC2015 dataset is used to train VGG-M.
[0048] MDNet defines target tracking as a binary classification problem of target and background. It inputs a 107×107×3 image, uses the VGG-M image depth features, and finally outputs the probability of the image being classified as a target or background.
[0049] The LSTM network in step 1 is trained using the ILSVRC2015 dataset. The input of the LSTM network is the historical position of the target in several consecutive frames, and the output is the position of the target in the current frame.
[0050] Step 6 is as follows:
[0051] Step 6.1, deep features and and the first frame target state The deep features The cosine similarity of is:
[0052]
[0053] where g track Depth feature representing the tracking result of the current frame based on the target appearance feature And the first frame target depth feature Cosine similarity, its value falls in the interval [0,1]. track The closer it is to 1, the and The greater the similarity, the greater the track The closer it is to 0, the and The smaller the similarity. The depth feature of the tracking result based on the target appearance feature of the current frame And the first frame target depth feature The inner product of . and They represent the depth feature of the tracking result based on the target appearance feature of the current frame and the vector modulus of the target depth feature of the first frame respectively.
[0054] where g pred Represents the deep features of the prediction result of the current frame based on the target timing information And the first frame target depth feature Cosine similarity, its value falls in the interval [0,1]. pred The closer it is to 1, the and The greater the similarity, the greater the pred The closer it is to 0, the and The smaller the similarity.
[0055] Step 6.2, the tracking result is output in which case by deciding the two similarities. The formula is:
[0056]
[0057] in This is the final tracking result.
[0058] Step 6.3, input the tracking result of the current frame into step 3, take the next frame as the current frame and perform image tracking of the next frame until all image frames in the video are tracked.
[0059] Example 1
[0060] Follow steps 1 to 4
[0061] In step 1, the ILSVRC2015 dataset is used to train the VGG-M network. The ILSVRC2015 dataset contains 3602 videos, each of which contains 100-120 frames. Ten frames are randomly selected from each video, and 50 positive samples and 200 negative samples are generated for each frame based on the target area. The loss is calculated using the cross entropy function. The initial learning rate of the network is 0.0001, and the training cycle is 100.
[0062] Step 2: First, the target state of the first frame in the video The corresponding area is extracted and scaled to 107×107×3. The scaled target area is then input into the feature extraction network VGG-M to obtain a deep feature with a dimension of 4096.
[0063] The target appearance feature-based tracker MDNet in step 3 also uses the VGG-M network trained in step 1 as a feature extractor. The MDNet input is scaled to a 107×107×3 target area and outputs the probability that the target area is identified as a target and background.
[0064] The target position predictor based on target time series information in step 4 adopts LSTM structure and is also suitable for ILSVRC2015 video dataset for training. During training, the target position of 8 consecutive frames is extracted from a video as output each time, and the label is the target position of the next frame after 8 consecutive frames. The loss is calculated by the mean square error function. The initial learning rate of network training is 0.0001, and the training cycle is 100.
[0065] The final tracking result is Figure 3 and Figure 4 . Figure 3 The tracking object is a walking pedestrian, and it will be blocked by the pedestrian walking on the opposite side between the 38th frame and the 45th frame. The light solid line rectangle in the figure is the result of the tracker MDNet based on appearance features, and the dotted line rectangle is the tracking result of the present invention. It can be seen that when occlusion occurs, MDNet is disturbed by the pedestrian walking on the opposite side and loses the target. The present invention introduces a similarity judgment mechanism, which can use time information when the appearance information is unreliable, so as to track the target well.
[0066] Figure 4 The tracking object is also a pedestrian. Between the 20th frame and the 61st frame, it will be blocked by pedestrians walking on the opposite side and interfered by traffic lights. The light solid rectangle in the figure is the result of the tracker MDNet based on appearance features, and the dashed rectangle is the tracking result of the present invention. It can be seen that when occlusion occurs, the tracking result obtained by the tracker based on appearance features becomes unreliable, and the historical time series information of the target is used to predict the target position in the current frame. Figure 4 It can be seen that the predicted results can follow the target well.
Claims
1. A target tracking method based on spatiotemporal information fusion, characterized in that: The specific steps are as follows: Step 1: Select the classic tracking method MDNet as the base tracker, where MDNet uses the trained VGG-M as the feature extraction network; Select the trained LSTM network as the target position predictor; Step 2: For the video to be tracked, the target state of the first frame is Input into the feature extraction network to obtain the deep features of the first frame target , and initialize the base tracker parameters using the target state of the first frame; Step 3: Enter the subsequent frame and obtain the target position and target width and height of the previous frame Constitutes the target state of the previous frame And input the base tracker for tracking, and obtain the tracking result based on the target appearance features ; Step 4: Use the method in step 3 to obtain the previous L The target state set of the frame Input the target position predictor LSTM to obtain the prediction result based on the target time series information ; Step 5: Input the tracking results based on the target appearance features and the prediction results based on the target timing information into the feature extraction network to obtain the corresponding deep features. and ; Step 6: Calculate the depth features separately and With the first frame target state The deep features The cosine similarity of is calculated and one of them is selected as the final tracking result.
2. The target tracking method based on spatiotemporal information fusion according to claim 1, characterized in that: The network of VGG-M in step 1 includes three convolutional layers and three fully connected layers connected in sequence, wherein the first convolutional layer and the second convolutional layer are respectively connected to a pooling layer, wherein each convolutional layer is followed by a random dropout layer and a RELU nonlinear activation function.
3. The target tracking method based on spatiotemporal information fusion according to claim 1, characterized in that: The MDNet defines target tracking as a binary classification problem of target and background, and uses the VGG-M image depth features to finally output the probability that the image is identified as the target and the background.
4. The target tracking method based on spatiotemporal information fusion according to claim 1, characterized in that: The LSTM network and VGG-M are both trained using the ILSVRC2015 dataset.
5. The target tracking method based on spatiotemporal information fusion according to claim 1, characterized in that: The step 6 is specifically as follows: Step 6.1, calculate the depth features separately and With the first frame target state The deep features The cosine similarity of is: (1) in Depth feature representing the tracking result of the current frame based on the target appearance feature And the first frame target depth feature Cosine similarity; (2) in Represents the deep features of the prediction result of the current frame based on the target timing information And the first frame target depth feature Cosine similarity; Step 6.2, the tracking result is output in which case by deciding the two similarities. The formula is: (3) in This is the final tracking result; Step 6.3, input the tracking result of the current frame into step 3, take the next frame as the current frame and perform image tracking of the next frame until all image frames in the video are tracked.