A single target tracking method based on deep learning

Through a single-objective tracking method based on deep learning, the position prediction model is used to form a moving image pair to predict the target position, which solves the problem of poor robustness in complex scenarios in the existing technology, and achieves high efficiency, Robust single-target tracking effect.

CN113870311BActive Publication Date: 2024-11-08ANHUI TSINGLINK INFORMATION TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111138038.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-27
Publication Date
2024-11-08
Estimated Expiration
2041-09-27

AI Technical Summary

Technical Problem

The existing target tracking technology is poorly robust in complex scenarios, difficult to achieve long-term tracking, and deep learning-based methods are costly and slower to calculate.

Method used

Using a single-objective tracking method based on deep learning, the previous frame and the current frame images are composed of moving image pairs through the position prediction model, and the target position is predicted by using feature extraction, detection and tracking modules, and the end is realized without increasing the calculation cost. To-end single-object detection tracking.

Benefits of technology

It realizes efficient and robust single-target tracking without increasing the calculation cost, can track for a long time, has strong tracking effect and fewer errors, and is suitable for complex scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113870311B_ABST
    Figure CN113870311B_ABST
Patent Text Reader

Abstract

The present invention discloses a single target tracking method based on deep learning, which belongs to the field of target tracking technology, and includes step S31, taking an image containing a target to be tracked as a previous frame image, and generating a mask image according to the position of the target to be tracked; S32, forming a motion image pair with the previous frame image, the mask image and the current frame image and inputting it into a position prediction model, predicting all target positions in the current frame image and the motion displacement of the target to be tracked in the previous frame image relative to the current frame image; S33, updating the previous frame image and the corresponding mask image according to all target positions in the current frame image and the motion displacement of the target to be tracked in the previous frame image relative to the current frame image; S34, repeating steps S32 to S33 based on the updated previous frame image and the corresponding mask image to achieve continuous tracking of a single target. The present invention can achieve end-to-end single target detection and tracking without increasing the computational cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a single target tracking method based on deep learning. Background Art

[0002] Target tracking refers to determining the boundary position of the target of interest in the current frame image based on the boundary position of the target in the previous frame image according to the spatiotemporal correlation. It is a core technology in the field of computer vision and has a wide range of applications. It is an essential technology for many downstream applications, such as motion analysis, behavior recognition, monitoring, and human-computer interaction.

[0003] At present, target tracking technology is mainly divided into two categories, as follows:

[0004] (1) Target tracking technology based on traditional technology. Representative technologies include Kalman filter tracking, optical flow tracking, template matching tracking, TLD tracking, CT tracking, KCF tracking, etc. The advantages of this type of technology are simple principles and fast operation speed. They can achieve good results in simpler scenarios and are suitable for short-term tracking. However, their disadvantages are poor robustness. In slightly more complex scenarios, they are prone to losing or mistracking targets and are not suitable for long-term tracking.

[0005] (2) Target tracking technology based on deep learning technology. This type of technology mainly uses the strategy of target detection plus target matching to complete the target tracking process. The process is to use a powerful deep learning-based target detection framework (such as: Faster-RCNN, SSD, YOLO) to first locate the target position in each frame image, and then use the nearest neighbor matching algorithm or feature vector matching algorithm to associate the same target in the previous and next frame images, and then complete the target tracking process. The advantages of this type of technology are strong robustness and the ability to track for a long time. Its disadvantages are over-reliance on the target detection framework, the target cannot run too fast, and the superposition of the two-step algorithm is time-consuming. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies in the above-mentioned background technology and to achieve end-to-end single target detection and tracking without increasing the computational cost.

[0007] To achieve the above objectives, a single target tracking method based on deep learning is adopted, including:

[0008] S31, taking the image containing the target to be tracked as the previous frame image, and generating a mask image according to the position of the target to be tracked;

[0009] S32, the previous frame image, the mask image and the current frame image are combined into a motion image pair and input into a position prediction model to predict the positions of all targets in the current frame image and the motion displacement of the target to be tracked in the previous frame image relative to the current frame image;

[0010] S33, updating the previous frame image and the corresponding mask image according to the positions of all targets in the current frame image and the motion displacement of the target to be tracked in the previous frame image relative to the current frame image;

[0011] S34. Based on the updated previous frame image and the corresponding mask image, steps S32 to S33 are repeatedly executed to achieve continuous tracking of a single target.

[0012] Furthermore, the position prediction model includes a feature extraction module, a detection module and a tracking module, wherein:

[0013] The feature extraction module is used to obtain a high-level feature map of the input motion image pair;

[0014] The detection module is used to predict all the targets and their positions in the current frame image based on the high-level feature map;

[0015] The tracking module is used to predict the new position of the target position to be tracked in the previous frame image in the current frame image based on the high-level feature map.

[0016] Furthermore, the feature extraction module includes a concatenation layer, a backbone network, a feature pyramid network FPN and output feature layers out_feature1 and out_feature2. The motion image pair is used as input of the concatenation layer, the concatenation layer concat concatenates the motion image pair according to the channel dimension and outputs a concatenation image. The output of the concatenation layer is connected to the feature pyramid network FPN via the backbone network, the feature pyramid network FPN outputs a feature map fused with features of different scales, and is output via the output feature layers out_feature1 and out_feature2.

[0017] Furthermore, the detection module includes convolutional layers dconv1_0, dconv2_0, dconv1_1, dconv2_1 and a target information parsing layer yolo, the outputs of the output feature layer out_feature1 and the output feature layer out_feature2 are respectively connected to the convolutional layers dconv1_0 and dconv2_0, the convolutional layers dconv1_0 and dconv2_0 are respectively connected to the target information parsing layer yolo via the convolutional layers dconv1_1 and dconv2_1, and the target information parsing layer yolo is used to extract effective target position information.

[0018] Furthermore, the tracking module includes a maximum pooling layer maxpool1, a concatenation layer concat2, and convolution layers tconv1 and tconv2; the outputs of the output feature layer out_feature1 and the output feature layer out_feature2 are respectively connected to the maximum pooling layer maxpool1 and the concatenation layer concat2, and the output of the concatenation layer concat2 is connected to the convolution layer tconv2 via the convolution layer tconv1.

[0019] Furthermore, the updating of the previous frame image and the corresponding mask image according to all target positions in the current frame image and the motion displacement of the target to be tracked in the previous frame image relative to the current frame image includes:

[0020] Accumulating the position of the target to be tracked in the previous frame image and the motion displacement of the target to be tracked relative to the current frame image to obtain a rough position of the target to be tracked in the current frame image;

[0021] Associating and matching the rough position of the target to be tracked in the current frame image with all target positions in the current frame image, and selecting the best matching target position as the precise position of the target to be tracked;

[0022] According to the precise position of the target to be tracked, the previous frame image and the corresponding mask image are updated.

[0023] Furthermore, the updating of the previous frame image and the corresponding mask image according to the precise position of the target to be tracked includes:

[0024] When the precise position of the target to be tracked is successfully acquired, the current frame image is used as the previous frame image, and a new mask image is generated according to the acquired precise position of the target to be tracked;

[0025] When the precise position of the target to be tracked is not successfully acquired, the previous frame image and the corresponding mask image are not updated.

[0026] Furthermore, before the previous frame image, the mask image and the current frame image are combined into a motion image pair and input into the position prediction model, the method further includes:

[0027] Collect pedestrian videos;

[0028] The pedestrian position in each frame of the pedestrian video and the motion information between different frame image pairs are marked to construct a set of motion image pairs;

[0029] The position prediction model is trained using a set of motion image pairs to learn model parameters.

[0030] Furthermore, the pedestrian position in each frame image in the pedestrian video and the motion information between different frame image pairs are annotated to construct a motion image pair set, including:

[0031] Obtaining the position of pedestrians in each frame of the pedestrian video and marking them;

[0032] Convert the pedestrian video into an image sequence, and use any frame image containing a target in the image sequence as a previous frame image, specify a target to be tracked in the previous frame image and generate a mask image according to the target to be tracked, and form a motion image pair with the previous frame image, the mask image and an arbitrarily selected current frame image;

[0033] The motion information of the motion image pairs is obtained, and based on the motion image pairs and the corresponding motion information, the motion image pair set is constructed.

[0034] Furthermore, the process of acquiring the motion information of the motion image pair includes:

[0035] Determining whether there is an identical target in the preceding and following frame images of the motion image pair;

[0036] If it exists, then the motion offset of the target in the previous and next frame images is calculated as the motion information of the motion image pair;

[0037] If not present, the motion offset -1 is used as the motion information of the motion image pair.

[0038] Compared with the prior art, the present invention has the following technical effects: the position prediction model in the present invention adopts a deep neural network model, and integrates a target matching strategy into a target detection framework based on deep learning, so as to realize an end-to-end single target detection and tracking process with almost no increase in any computational cost. It has strong versatility, high real-time performance, few error sources, can track for a long time, and has a strong robust tracking effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0040] Figure 1 It is the overall structure diagram of the deep neural network model;

[0041] Figure 2 It is a flowchart of a single target tracking method based on deep learning;

[0042] Figure 3 It is the network structure diagram of the feature extraction module;

[0043] Figure 4 It is the network structure diagram of the detection module;

[0044] Figure 5 It is the network structure diagram of the tracking module.

[0045] Among them, the logo on the left side of each neural network structure layer graph indicates the output feature map size of the network structure: feature map width × feature map height × number of feature map channels. DETAILED DESCRIPTION

[0046] In order to further illustrate the features of the present invention, please refer to the following detailed description and drawings of the present invention. The drawings are for reference and illustration only and are not intended to limit the scope of protection of the present invention.

[0047] This embodiment is applicable to all single target tracking scenarios. For the convenience of description, the present invention takes pedestrian single target tracking as an example. This embodiment discloses a single target tracking method based on deep learning, including the following steps:

[0048] S1. Design of deep neural network model:

[0049] The deep neural network model designed in this embodiment is as follows: Figure 1 As shown in the figure, its main function is to use a deep neural network model with a fusion mechanism to directly complete the detection and tracking of pedestrian targets in each frame of the image. Since it no longer deliberately distinguishes between steps such as pedestrian detection and positioning and pedestrian association matching, the entire pedestrian tracking system has a faster operation speed, fewer error sources, and a more robust tracking effect.

[0050] This embodiment adopts a convolutional neural network (CNN). In order to facilitate the description of this solution, some terms are defined: feature map resolution refers to feature map height × feature map width, feature map size refers to feature map width × feature map height × number of feature map channels, kernel size refers to kernel width × kernel height, span refers to width direction span × height direction span, and each convolution layer is followed by a batch normalization layer and a nonlinear activation layer.

[0051] This solution is based on the single-stage target detection framework yolov4-tiny with superior performance and is optimized and improved. The designed deep neural network model contains four modules: feature extraction module backbone module, detection module detectmodule, tracking module track module, and update module update module. Among them, the update module update module does not participate in training and only works during testing. The specific design steps are as follows:

[0052] S11. Design feature extraction module backbone module:

[0053] The feature extraction module is mainly used to obtain high-level features with high abstraction and rich expression capabilities of the input motion image pair. The quality of high-level feature extraction directly affects the performance of subsequent pedestrian target tracking. The present invention uses the same feature extraction module as yolov4-tiny, such as Figure 2 As shown in FIG, the input of the feature extraction network is a motion image pair, which is composed of three 3-channel RGB images with a resolution of 320×320, where one image is the current frame image, one is the previous frame image, and the last image is a mask image of the pedestrian target position in the previous frame image. The mask image is obtained by setting all pixels in the non-pedestrian area to 0.

[0054] concat is the concatenation layer, which is mainly used to concatenate the three 3-channel RGB images into a 9-channel image of the same resolution according to the channel dimension. backbone is the backbone network of yolov4-tiny, FPN is the feature pyramid network, which is mainly used to fuse features of different scales. The specific network structure is the same as yolov4-tiny. out_feature1 and out_feature2 are the output feature layers of the feature extraction module, which are used for subsequent detection and tracking of pedestrian targets. The feature map resolution of out_feature1 is 20x20x384, and the feature map resolution of out_feature2 is 10x10x256.

[0055] S12. Design the detection module:

[0056] The detection module mainly predicts whether there is a pedestrian target and the location of the pedestrian target in the current frame image based on the feature map output by the feature extraction module. The present invention uses the same detection module as yolov4-tiny, such as Figure 3 As shown in the figure, dconv1_0 and dconv2_0 are convolutional layers with a kernel size of 3x3 and a span of 1x1, and dconv1_1 and dconv2_1 are convolutional layers with a kernel size of 1x1 and a span of 1x1. The yolo layer is the same as yolov4-tiny, used to extract valid pedestrian targets, and only works during testing. The feature map resolution of the yolo layer is Nx5, where N represents the number of pedestrian targets detected.

[0057] S13. Design track module:

[0058] The tracking module mainly predicts the new position of the pedestrian target to be tracked in the previous frame image in the current frame based on the feature map output by the feature extraction module. Since the feature map output by the feature extraction module has integrated the motion association information of the previous and next frames, the tracking module predicts the motion displacement of the pedestrian target to be tracked from the previous frame to the current frame based on this motion association information, such as Figure 4 As shown in the figure, maxpool1 is a maximum pooling layer with a kernel size of 2x2 and a span of 2x2, concat2 is a concatenation layer, which mainly concatenates multiple input feature maps into an output feature map according to the channel dimension. tconv1 is a convolutional layer with a kernel size of 3x3 and a span of 2x2, and tconv2 is a convolutional layer with a kernel size of 5x5 and a span of 1x1.

[0059] S14. Design update module:

[0060] The update module mainly calculates the exact position of the pedestrian target to be tracked in the previous frame image in the current frame image based on the output information of the detection module and the tracking module. The specific steps are as follows:

[0061] S141, obtaining a rough new position of the pedestrian target to be tracked, mainly based on the position of the pedestrian target to be tracked in the previous frame image, and accumulating the motion offset obtained by the tracking module, so as to obtain the rough position of the pedestrian target to be tracked in the current frame image.

[0062] S142, obtaining the precise new position of the pedestrian target to be tracked. The main method is to associate and match the rough new position of the pedestrian target to be tracked obtained in step S141 with each pedestrian target position obtained by the detection module, and select the best matching pedestrian target position as the precise new position of the pedestrian target to be tracked.

[0063] It should be noted that the IOU (Intersection over Union) matching degree used in the present invention is used as the association matching function, and any other similarity measurement function may also be used as the association matching function. When the matching degree of the best matching pedestrian target is lower than a certain threshold, it is considered that there is no pedestrian target to be tracked in the current frame image.

[0064] S143, updating the pedestrian target to be tracked, mainly generating a new previous frame image and a mask image according to the new position of the pedestrian target to be tracked in the current frame image. When the precise new position of the pedestrian target to be tracked is successfully obtained in step S142, the current frame image is changed to the previous frame image, and a mask image corresponding to the new previous frame image is generated according to the precise new position of the pedestrian target to be tracked; when the precise new position of the pedestrian target to be tracked is not successfully obtained in step S142, the pedestrian target to be tracked is not updated, and the previous frame image and the mask image are directly used as the new previous frame image and the mask image.

[0065] S2. Training deep neural network model:

[0066] After designing the deep neural network model, the next step is to collect pedestrian video images in various scenarios, feed them into the deep neural network model, and learn the relevant model parameters. The specific steps are as follows:

[0067] S21. Collect pedestrian videos, mainly collecting pedestrian videos in various scenes, various light sources, and various angles;

[0068] S22, marking pedestrian motion information, mainly marking the pedestrian position in each frame image in the video and the motion information between different frame motion image pairs, the specific steps are as follows:

[0069] S221, marking the position of pedestrians, the main method is to use the existing deep learning-based pedestrian detection framework to obtain the position of pedestrians in each frame of the video;

[0070] S222, forming a motion image pair, mainly converting the video into an image sequence, arbitrarily selecting a frame image containing a pedestrian target as the previous frame image, and specifying the pedestrian target to be tracked, generating a tracking mask image, the mask image is generated by keeping the pixels in the position area of ​​the pedestrian target to be tracked unchanged, and setting all the pixels in the non-pedestrian target position area to 0, and then arbitrarily selecting an image in the next 120 frames as the current frame image, and forming a motion image pair together with the previous frame image and the mask image;

[0071] S223, obtaining motion information of the motion image pair, the main method is to manually review each motion image pair, determine whether there is the same pedestrian target in the previous and next frames of the motion image pair, if there is the same pedestrian target, calculate the motion offset of the pedestrian target in the previous and next frames as the motion information of the current image pair, if there is no same pedestrian target, use the motion offset of -1 as the motion information of the current image pair. The motion information is represented by a 2-dimensional vector, which respectively represents the normalized motion offset of the pedestrian target in the x-direction and the normalized motion offset of the pedestrian target in the previous and next images of the motion image pair.

[0072] S23, train the deep neural network model, send the sorted motion image pair set into the defined deep neural network model, and learn the relevant model parameters. The target loss function of the detection module during network model training remains the same as that of yolov4-tiny, while the target loss function of the tracking module adopts the mean square error loss function.

[0073] S3. Using deep neural network model:

[0074] After training the deep neural network model, the next step is to use the model in the actual environment to track pedestrians. For any given frame of pedestrian image, specify the position of the pedestrian target to be tracked, form a motion image pair and send it to the trained deep neural network model, directly output the new position of the pedestrian target in the current frame image, and execute it in a loop to achieve continuous tracking of pedestrian targets. The specific steps are as follows: Figure 5 As shown:

[0075] S31, specifying the initial position of the pedestrian target to be tracked, mainly specifying the position of the pedestrian target to be tracked in any given frame of pedestrian image as the tracking initial position of the pedestrian target to be tracked, and taking the frame image as the previous frame image, and then generating a mask image;

[0076] S32, predicting pedestrian information and motion information of the current frame image, the main method is to form an image pair with the previous frame image, the mask image and the current frame image, and send them to the deep neural network model to directly predict the positions of all pedestrian targets in the current frame image and the motion displacement of the pedestrian targets to be tracked in the previous frame image relative to the current frame image;

[0077] S33, updating the position of the pedestrian target to be tracked, mainly based on the pedestrian target position predicted in step S32 and the motion displacement of the pedestrian target to be tracked, using the update module to obtain a new previous frame image and a mask image, and repeating steps S32 to S33;

[0078] S34. Realize continuous tracking of a single pedestrian target.

[0079] This embodiment adopts a deep neural network model with a fusion mechanism to directly complete the single target tracking process end-to-end without increasing any computational cost.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A single target tracking method based on deep learning, characterized in that: include: S31, taking the image containing the target to be tracked as the previous frame image, and generating a mask image according to the position of the target to be tracked; S32, the previous frame image, the mask image and the current frame image are combined into a motion image pair and input into a position prediction model to predict the positions of all targets in the current frame image and the motion displacement of the target to be tracked in the previous frame image relative to the current frame image; S33, updating the previous frame image and the corresponding mask image according to all target positions in the current frame image and the motion displacement of the target to be tracked in the previous frame image relative to the current frame image, specifically including S141 to S143: S141, accumulating the position of the target to be tracked in the previous frame image and the motion displacement of the target to be tracked relative to the current frame image to obtain a rough position of the target to be tracked in the current frame image; S142, associating and matching the rough position of the target to be tracked in the current frame image with all target positions in the current frame image, and selecting the best matching target position as the precise position of the target to be tracked; S143, updating the previous frame image and the corresponding mask image according to the precise position of the target to be tracked; S34, based on the updated previous frame image and the corresponding mask image, repeat steps S32 to S33 to achieve continuous tracking of a single target; The position prediction model includes a feature extraction module, a detection module, a tracking module and an update module, wherein: The feature extraction module is used to obtain a high-level feature map of the input motion image pair; The detection module is used to predict all the targets and their positions in the current frame image based on the high-level feature map; The tracking module is used to predict the new position of the target position to be tracked in the previous frame image in the current frame image based on the high-level feature map; The updating module is used to calculate the accurate position of the target to be tracked in the previous frame image in the current frame image according to the output information of the detection module and the tracking module.

2. The single target tracking method based on deep learning according to claim 1, characterized in that: The feature extraction module includes a concatenation layer, a backbone network, a feature pyramid network FPN, and output feature layers out_feature1 and out_feature2. The motion image pair is used as the input of the concatenation layer, and the concatenation layer concat concatenates the motion image pair according to the channel dimension and outputs a concatenation image. The output of the concatenation layer is connected to the feature pyramid network FPN through the backbone network, and the feature pyramid network FPN outputs a feature map fused with features of different scales, and is output through the output feature layers out_feature1 and out_feature2.

3. The single target tracking method based on deep learning as claimed in claim 2, characterized in that: The detection module includes convolutional layers dconv1_0, dconv2_0, dconv1_1, dconv2_1 and a target information parsing layer yolo. The outputs of the output feature layer out_feature1 and the output feature layer out_feature2 are respectively connected to the convolutional layers dconv1_0 and dconv2_0. The convolutional layers dconv1_0 and dconv2_0 are respectively connected to the target information parsing layer yolo via the convolutional layers dconv1_1 and dconv2_1. The target information parsing layer yolo is used to extract effective target position information.

4. The single target tracking method based on deep learning as claimed in claim 2, characterized in that: The tracking module includes a maximum pooling layer maxpool1, a concatenation layer concat2, and convolution layers tconv1 and tconv2; the outputs of the output feature layer out_feature1 and the output feature layer out_feature2 are respectively connected to the maximum pooling layer maxpool1 and the concatenation layer concat2, and the output of the concatenation layer concat2 is connected to the convolution layer tconv2 via the convolution layer tconv1.

5. The single target tracking method based on deep learning according to claim 1, characterized in that: The updating of the previous frame image and the corresponding mask image according to the precise position of the target to be tracked includes: When the precise position of the target to be tracked is successfully acquired, the current frame image is used as the previous frame image, and a new mask image is generated according to the acquired precise position of the target to be tracked; When the precise position of the target to be tracked is not successfully acquired, the previous frame image and the corresponding mask image are not updated.

6. The single target tracking method based on deep learning according to any one of claims 1 to 5, characterized in that: Before the previous frame image, the mask image and the current frame image are combined into a motion image pair and input into the position prediction model, the method further includes: Collect pedestrian videos; The pedestrian position in each frame of the pedestrian video and the motion information between different frame image pairs are marked to construct a set of motion image pairs; The position prediction model is trained using a set of motion image pairs to learn model parameters.

7. The single target tracking method based on deep learning according to claim 6, characterized in that: The pedestrian position in each frame image in the pedestrian video and the motion information between different frame image pairs are marked to construct a motion image pair set, including: Obtaining the position of pedestrians in each frame of the pedestrian video and marking them; Convert the pedestrian video into an image sequence, and use any frame image containing a target in the image sequence as a previous frame image, specify a target to be tracked in the previous frame image and generate a mask image according to the target to be tracked, and form a motion image pair with the previous frame image, the mask image and an arbitrarily selected current frame image; The motion information of the motion image pairs is obtained, and based on the motion image pairs and the corresponding motion information, the motion image pair set is constructed.

8. The single target tracking method based on deep learning as claimed in claim 7, characterized in that: The process of acquiring the motion information of the motion image pair includes: Determining whether there is an identical target in the preceding and following frame images of the motion image pair; If it exists, then the motion offset of the target in the previous and next frame images is calculated as the motion information of the motion image pair; If not present, the motion offset -1 is used as the motion information of the motion image pair.