A high-speed video high-reliability target tracking method combined with time domain motion features

Abstract

The application discloses a high-speed video high-reliability target tracking method combined with time domain motion features. The method is used for the reliable tracking of a high-speed video motion target. A fast global target tracking module of deep learning is used to obtain a high-frame-rate candidate target tracking result according to target appearance features. A target tracking module combined with time domain motion features is used to verify the time domain motion trajectory of the high-frame-rate tracking result according to candidate target motion trajectory features, so that a more reliable target tracking result is obtained. Finally, the target image position is converted into a miss distance and input into a servo system to control the alignment of a sensor to a target, so that a target tracking closed loop is realized. The target appearance features and motion features are fused and processed, so that the reliability of target tracking is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of video target detection and tracking technology, specifically to a high-speed, high-reliability video target tracking method that combines temporal motion features. Background Technology

[0002] Target tracking algorithms use video information acquired by sensors (visible light or infrared cameras) and target information labeled in the initial time period to model the appearance and motion features of the target, enabling robust target tracking in complex backgrounds in subsequent video frames. Currently, with the continuous development of optical sensor technology, high-speed cameras can capture target motion at a frame rate of hundreds of frames per second. This allows tracking algorithms to acquire more target information, thereby improving tracking performance, but it also places higher demands on the processing power and efficiency of the algorithms.

[0003] In target tracking tasks, to prevent moving targets from moving out of the sensor's field of view, it is necessary to obtain the target's image miss distance based on the target coordinates output by the tracking algorithm. This miss distance is then input into the servo system to adjust the camera lens's attitude, ensuring the target remains as centered as possible within the field of view of the captured image, thus completing the target tracking loop. Therefore, achieving highly reliable target tracking is a practical requirement for tracking algorithms in photoelectric tracking systems. Traditional target tracking algorithms rely solely on the target's appearance features in the image. For example, Wu et al. (H.Wu, W.Li, W.Li, and G.Liu, "A real-time robust approach for tracking UAVs in infrared videos," in IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp.4448-4455) used deep learning networks to extract the target's appearance features to track UAVs in infrared video. However, in practical applications, complex backgrounds such as mountains, clouds, and sea surfaces can severely affect target discrimination based on appearance features. Therefore, it is necessary to combine temporal motion features to improve the reliability of target tracking. For example, the SiamYOLO algorithm (H. Fang, X. Wang, Z. Liao, Y. Chang, and L. Yan, "A Real-time Anti-distractor Infrared UAV Tracker with Channel Feature Refinement Module," in IEEE International Conference on Computer Vision Workshops, 2021, pp. 1240-1248) uses the Kalman filter algorithm to estimate the target's trajectory to reduce the impact of background interference on the tracking process. However, when the target undergoes complex and rapid motion, the estimation accuracy of the Kalman filter algorithm will be significantly reduced. To address the above-mentioned problems, this invention fully utilizes the high frame rate temporal information provided by high-speed cameras to improve the accuracy of target motion estimation. Combined with the servo control system of the camera platform, a high-speed video robust target tracking method based on combined temporal motion features is proposed. Summary of the Invention

[0004] This invention proposes a high-speed video robust target tracking method based on joint temporal motion features to improve the reliability of moving target tracking in complex scenes.

[0005] The technical solution of this invention:

[0006] A high-speed, high-reliability target tracking method for video based on joint temporal motion features is proposed. First, a fast global target tracking module based on deep learning is used to generate high-frame-rate candidate target tracking results based on target appearance features. Then, a target tracking module with joint temporal motion features is used to verify the high-frame-rate tracking results based on the motion trajectory features of the candidate targets, outputting more reliable target tracking results. Finally, the target image position is converted into a miss distance and input into the servo system to control the sensor to align with the target, thereby achieving a closed-loop target tracking system. The specific steps are as follows:

[0007] (1) Fast global target tracking based on deep learning:

[0008] (1.1) Feature extraction: The ResNet50 network with shared parameters is used to extract features at four different resolutions from the first frame and the current frame of the video. The resolution is [H / Stride, W / Stride], where h = H / Stride and w = W / Stride, and H and W are the height and width of the video frame, respectively, and Stride ∈ {4, 8, 16, 32}. Then, FPN (Feature Pyramid Network) is used for feature enhancement. The feature maps at resolution level L correspond to the target sizes of [0, 10), [10, 20), [20, 40) and [40, +∞), respectively, where L = 0, 1, 2, 3 corresponds to Stride.

[0009] (1.2) Information Fusion: Information fusion between the first frame and the current frame is achieved through cross-correlation operations, which can also be viewed as a global query of the current frame for a specified target. Specifically, for the first frame, the features of the target are extracted using RoI (Region-Of-Interest) alignment operations based on the ground truth, resulting in a 7×7×C tensor z, where C is the number of channels. Assuming L represents the feature indices of four size levels from largest to smallest, the specific calculation method for the cross-correlation operation is as follows:

[0010]

[0011] Where, x L Features of the current frame A 1×1 convolutional layer is used to transform the number of channels. A 3×3 convolutional layer is used to preprocess feature x. s , For convolution operations, A 7×7 convolutional layer with zero padding is used to convert z into a 1×1×C convolutional kernel.

[0012] The fused features are further enhanced using attention modules, which include channel attention modules and spatial attention modules. The feature x′L After being processed sequentially by the channel attention module and the spatial attention module, the final fused feature x″′ is obtained. L The calculation method for the channel attention module is shown in formula (2), and the calculation method for the spatial attention module is shown in formula (3).

[0013]

[0014]

[0015] Here, ⊙ represents the element-wise dot product operation, and MLP (Multi-layer Perceptron) is a multi-layer perceptron. and These represent max pooling and average pooling, respectively, and [·] denotes a connection operation. The channel attention module generates 1×1×C dimensional tensor weights, while the spatial attention module generates h×w×1 dimensional tensor weights.

[0016] (1.3) Output section: In the output header module, x″′ L After passing through a 3×3 convolutional layer, each layer enters a classification branch to obtain an h×w×1 dimensional score map s. L And a regression branch yields the h×w×4 dimensional target bounding box tensor b. L , where b L The last dimension represents the top-left and bottom-right corners of the target (x0, y0, x1, y1). It's important to note that for the four levels of features x″′... L The parameters of the output header are shared. In the classification branch, the position of the maximum value in the score map of the four levels corresponds to the position O of the candidate target, and the predicted size of the candidate target is obtained according to the corresponding position in the regression branch.

[0017] (2) Target tracking with combined temporal motion features

[0018] Based on the assumption that target motion between a few frames in high frame rate video can be approximated as uniform linear motion, the RANSAC method is used to fit the target motion trajectory according to the candidate target positions in every N-1 frames. This is used to determine the accuracy of the predicted target position in the Nth frame. After further analysis and processing, only the target position in the Nth frame is output as the result. Therefore, N can be regarded as the factor reducing the output frame rate. The specific steps are as follows:

[0019] (2.1) Map the candidate target positions O of frames 1 to N-1 to the same coordinate system. The calculation method is as follows:

[0020]

[0021] Where (x,y) represents the position O of the candidate target in the original coordinate system.i (Δx, Δy) represents the platform motion compensation controlled by the servo system. This indicates the candidate target position mapped to the same coordinate system after motion compensation.

[0022] (2.2) For RANSAC processing is performed to obtain the fitted target motion trajectory, and then the predicted target position based on motion features is obtained. If the candidate target location is predicted based on appearance... Position of the predicted target location based on motion features If the difference is less than the threshold η, the appearance-based predicted target location is considered accurate; otherwise, it is considered inaccurate, and the motion-based predicted target location O is used. N As the target position of the current frame.

[0023] (3) Calculation of miss distance

[0024] The sensor's servo system requires the miss distance as system input. The target image position is converted into the miss distance and input into the servo system to control the sensor to align with the target.

[0025] The calculation method for converting the target position into the miss distance Δ is as follows:

[0026] Δ=(x O -x c ,y O -y c (5)

[0027] Among them, (x O ,y O (x) represents the image coordinates of the target. c ,y c () represents the center point of the image.

[0028] The off-target amount is imported into the servo system to form a stable tracking closed loop.

[0029] The beneficial effects of this invention are as follows: This invention designs a deep global target tracking module to achieve fast and accurate target tracking based on the target's appearance features, and combines the target motion features in high-speed video to improve the reliability of target tracking. Finally, the tracking results are converted and imported into the control system to achieve stable closed-loop tracking. Attached Figure Description

[0030] Figure 1 This is a basic flowchart of the present invention;

[0031] Figure 2 This is a flowchart illustrating the process of fast global target tracking based on deep learning;

[0032] Figure 3 This is a schematic diagram illustrating the principle of mapping target candidate positions to the same coordinate system. Detailed Implementation

[0033] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0034] like Figure 1 As shown, the present invention provides a high-speed video high-reliability target tracking method based on joint temporal motion features. First, a fast global target tracking module based on deep learning is used to generate high frame rate candidate target tracking results based on the target's appearance features. Then, a target tracking module based on joint temporal motion features is used to verify the high frame rate tracking results based on the motion trajectory features of the candidate targets, outputting more reliable target tracking results. Finally, the target image position is converted into a miss distance and input into the servo system to control the sensor to align with the target, thereby realizing a closed loop of target tracking.

[0035] The process of fast global target tracking based on deep learning is as follows: Figure 2 As shown, the specific steps are as follows:

[0036] (1.1) Feature extraction: The ResNet50 network with shared parameters is used to extract features at four different resolutions from the first frame and the current frame of the video. The resolution is [H / Stride, W / Stride], where h = H / Stride and w = W / Stride, and H and W are the height and width of the video frame, respectively, and Stride ∈ {4, 8, 16, 32}. Then, FPN (Feature Pyramid Network) is used for feature enhancement. The feature maps at resolution level L correspond to the target sizes of [0, 10), [10, 20), [20, 40) and [40, +∞), respectively, where L = 0, 1, 2, 3 corresponds to Stride.

[0037] (1.2) Information Fusion: Information fusion between the first frame and the current frame is achieved through cross-correlation operations, which can also be viewed as a global query of the current frame for a specified target. Specifically, for the first frame, the features of the target are extracted using RoI (Region-Of-Interest) alignment operations based on the ground truth, resulting in a 7×7×C tensor z, where C is the number of channels. Assuming L represents the feature indices of four size levels from largest to smallest, the specific calculation method for the cross-correlation operation is as follows:

[0038]

[0039] Where, x L Features of the current frame A 1×1 convolutional layer is used to transform the number of channels. A 3×3 convolutional layer is used to preprocess feature x. s , For convolution operations, A 7×7 convolutional layer with zero padding is used to convert z into a 1×1×C convolutional kernel.

[0040] The fused features are further enhanced using attention modules, which include channel attention modules and spatial attention modules. The feature x′ L After being processed sequentially by the channel attention module and the spatial attention module, the final fused feature x″′ is obtained. L The calculation method for the channel attention module is shown in formula (2), and the calculation method for the spatial attention module is shown in formula (3).

[0041]

[0042]

[0043] Here, ⊙ represents the element-wise dot product operation, and MLP (Multi-layer Perceptron) is a multi-layer perceptron. and These represent max pooling and average pooling, respectively, and [·] denotes a connection operation. The channel attention module generates 1×1×C dimensional tensor weights, while the spatial attention module generates h×w×1 dimensional tensor weights.

[0044] (1.3) Output section: In the output header module, x″′ L After passing through a 3×3 convolutional layer, each part enters a classification branch (composed of a 3×3 convolutional layer) to obtain an h×w×1 dimensional score map s. L A regression branch (consisting of a 3×3 convolutional layer) yields the h×w×4 dimensional target box tensor b. L , where b L The last dimension represents the top-left and bottom-right corners of the target (x0, y0, x1, y1). It's important to note that for the four levels of features x″′... L The parameters of the output header are shared. In the classification branch, the position of the maximum value in the score map of the four levels corresponds to the position O of the candidate target, and the predicted size of the candidate target is obtained according to the corresponding position in the regression branch.

[0045] The specific steps for target tracking based on combined temporal motion features are as follows:

[0046] A fast global target tracking module based on deep learning can predict the position and size of candidate targets in each frame generated by a high-speed camera. However, this result is susceptible to errors due to occlusion and interference from similar objects. To improve the robustness of the tracking results and reduce the output frame rate to meet the input requirements of the servo system, this invention is based on the assumption that the target motion between a few frames in high frame rate video can be approximated as uniform linear motion. Based on the candidate target positions in every N-1 frames, the RANSAC method is used to fit the target motion trajectory to determine the accuracy of the predicted target position in the Nth frame. After further analysis and processing, only the target position in the Nth frame is output as the result. Therefore, N can be considered as a factor reducing the output frame rate. The specific steps are as follows:

[0047] (2.1) The candidate target positions O of frames 1 to N-1 are mapped to the same coordinate system. The principle is as follows: Figure 3 As shown, the calculation method is as follows:

[0048]

[0049] Where (x,y) represents the position O of the candidate target in the original coordinate system. i (Δx, Δy) represents the platform motion compensation controlled by the servo system. This indicates the candidate target position mapped to the same coordinate system after motion compensation.

[0050] (2.2) For RANSAC processing is performed to obtain the fitted target motion trajectory, and then the predicted target position based on motion features is obtained. If the candidate target location is predicted based on appearance... Position of the predicted target location based on motion features If the difference is less than the threshold η, the appearance-based predicted target location is considered accurate; otherwise, it is considered inaccurate, and the motion-based predicted target location O is used. N As the target position of the current frame.

[0051] (3) Calculation of miss distance

[0052] The sensor's servo system requires the miss distance as system input. The target image position is converted into the miss distance and input into the servo system to control the sensor to align with the target.

[0053] The calculation method for converting the target position into the miss distance Δ is as follows:

[0054] Δ=(x O -x c ,y O -y c (5)

[0055] Among them, (xO ,y O (x) represents the image coordinates of the target. c ,y c () represents the center point of the image.

[0056] The off-target amount is imported into the servo system to form a stable tracking closed loop.

[0057] This embodiment uses the AntiUAV dataset to train a fast global target tracking module based on deep learning. Then, it is tested using 300fps, 512×640 infrared video captured by a high-speed camera. The resulting resolutions for the four feature layers are 128×160, 64×80, 32×40, and 16×20. In every six frames, the module outputs the location of the highest-scoring candidate target in the first five frames and the locations of the ten highest-scoring candidate targets in the sixth frame.

[0058] The target tracking module, which combines temporal motion features, fits motion trajectories based on the candidate target positions in the first five frames and uses this to estimate the target position in the sixth frame. It then determines the distances of the ten candidate targets to this estimated position to obtain the tracking result for the sixth frame. Therefore, this module will output a tracking result at 50fps.

[0059] Finally, the tracking results are converted into miss distances and input into the servo control system to keep the camera always pointing at the target. At the same time, the motion parameters of the camera platform are also input into the target tracking module that combines temporal motion features to compensate for the target position offset caused by the platform motion, thus forming a stable tracking closed loop.

Claims

1. A high-speed, high-reliability target tracking method for video using combined temporal motion features, characterized in that, First, a fast global target tracking module based on deep learning is used to generate high-frame-rate candidate target tracking results based on target appearance features. Then, a target tracking module combining temporal motion features is used to verify the high-frame-rate tracking results based on the motion trajectory features of the candidate targets, outputting more reliable target tracking results. Finally, the target image position is converted into a miss distance and input into the servo system to control the sensor to align with the target, thereby achieving a closed-loop target tracking system. The specific steps are as follows: (1) Fast global target tracking based on deep learning: (1.1) Feature extraction: The ResNet50 network with shared parameters is used to extract features at four different resolutions from the first frame and the current frame of the video. The resolution is [H / Stride, W / Stride], and we denote h = H / Stride and w = W / Stride, where H and W are the height and width of the video frame, respectively, and Stride ∈ {4, 8, 16, 32}. Then, FPN is used for feature enhancement. The feature maps at resolution level L correspond to the processing targets with sizes of [0, 10), [10, 20), [20, 40), and [40, +∞), respectively, where L = 0, 1, 2, 3 corresponds to Stride. (1.2) Information Fusion: Information fusion between the first frame and the current frame is achieved through cross-correlation operations, which can also be regarded as a global query of the current frame for a specified target. Specifically, for the first frame, the features of the target are extracted using RoI alignment operations based on the ground truth, resulting in a 7×7×C tensor z, where C is the number of channels. Assuming L represents the feature indexes of four size levels from large to small, the specific calculation method of the cross-correlation operation is as follows: Where, x L Features of the current frame A 1×1 convolutional layer is used to transform the number of channels. A 3×3 convolutional layer is used to preprocess feature x. s , For convolution operations, A 7×7 convolutional layer with zero padding is used to convert z into a 1×1×C convolutional kernel; The fused features are further enhanced using attention modules, which include channel attention modules and spatial attention modules. The feature x′ L After being processed sequentially by the channel attention module and the spatial attention module, the final fused feature x″′ is obtained. L The calculation method for the channel attention module is shown in formula (2), and the calculation method for the spatial attention module is shown in formula (3). Here, ⊙ represents the element-wise dot product operation, and MLP stands for Multilayer Perceptron. and These represent max pooling and average pooling, respectively; [·] denotes a connection operation; the channel attention module generates 1×1×C dimensional tensor weights, while the spatial attention module generates h×w×1 dimensional tensor weights. (1.3) Output section: In the output header module, x″′ L After passing through a 3×3 convolutional layer, each layer enters a classification branch to obtain an h×w×1 dimensional score map s. L And a regression branch yields the h×w×4 dimensional target bounding box tensor b. L , where b L The last dimension represents the top-left and bottom-right corner points of the target (x0, y0, x1, y1); it is important to note that for the four levels of features x″′ L The parameters of the output head are shared; in the classification branch, the position of the maximum value in the score map of the four levels corresponds to the position O of the candidate target, and the predicted size of the candidate target is obtained according to the corresponding position of the regression branch. (2) Target tracking with combined temporal motion features Based on the assumption that target motion between a few frames in high frame rate video can be approximated as uniform linear motion, the RANSAC method is used to fit the target motion trajectory according to the candidate target positions in every N-1 frames to determine the accuracy of the predicted target position in the Nth frame. After further analysis and processing, only the target position in the Nth frame is output as the result. Therefore, N is considered as the factor that reduces the output frame rate. The specific steps are as follows: (2.1) Map the candidate target positions O of frames 1 to N-1 to the same coordinate system. The calculation method is as follows: Where (x,y) represents the position O of the candidate target in the original coordinate system. i (Δx, Δy) represents the platform motion compensation controlled by the servo system. This indicates the candidate target position mapped to the same coordinate system after motion compensation; (2.2) For RANSAC processing is performed to obtain the fitted target motion trajectory, and then the predicted target position based on motion features is obtained. If the candidate target location is predicted based on appearance... Position of the predicted target location based on motion features If the difference is less than the threshold η, the appearance-based predicted target location is considered accurate; otherwise, it is considered inaccurate, and the motion-based predicted target location O is used. N As the target position of the current frame; (3) Calculation of miss distance The sensor's servo system requires the miss distance as system input. The target image position is converted into the miss distance and input into the servo system to control the sensor to align with the target. The calculation method for converting the target position into the miss distance Δ is as follows: Δ=(x O -x c ,and O -and c ) (5) Among them, (x O ,y O (x) represents the image coordinates of the target. c ,y c () represents the center point of the image; The off-target amount is imported into the servo system to form a stable tracking closed loop.

Images