Multi-target tracking method and system based on deep path aggregation network

Abstract

The present invention discloses a multi-target tracking method and system based on a deep path aggregation network. With the deep path aggregation network as the framework, a large amount of external video data is first input to train the network to generate a target center heat map, a target center offset, a prediction box size, and extract the re-ID feature of the target; then a prediction box is generated according to the target position information, and the detected object is marked in the form of a rectangular box; then the cosine distance of the re-ID feature vectors of all detected objects in a certain frame of the video and the previous frame and the IoU of the prediction box are calculated, and all objects in the current frame are connected to the existing trajectory. Finally, the positions of all targets in the current frame are further estimated by the Kalman filter algorithm. The present invention adopts a bottom-up feature fusion layer to extract target spatial feature information, shortens the information path between the bottom and top features, and enables the tracker to have higher tracking accuracy and the advantages of fast real-time tracking speed.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, deep learning technology, and multi-target tracking, and in particular relates to a multi-target tracking method and system based on a deep path aggregation network. Background Art

[0002] Multi-target tracking is a hot research area in computer vision, and has important applications in autonomous driving, smart transportation, intelligent monitoring and other fields. With the rapid development of artificial intelligence, more and more deep learning algorithms are applied to all aspects of life. In single-target tracking, the appearance features of the object are known in advance. In multi-target tracking, the tracker needs to estimate the trajectories of multiple targets in the video and detect targets that leave or enter the video scene. Multiple objects in the video may occlude each other or have similar appearances. External environmental factors such as lighting, weather and video quality may also make it difficult for the tracker to track. In the field of multi-target tracking, trackers based on deep learning have relatively good performance, including JDE, FairMOT, etc., but they usually cannot achieve a good balance between tracking accuracy and tracking speed. And there is no further extraction of target spatial feature information, so the precise position of the target cannot be well inferred. Summary of the invention

[0003] The present invention aims to provide a multi-target tracking method and system based on a deep path aggregation network to solve the technical problems that the existing trackers based on deep learning cannot achieve a good balance between tracking accuracy and tracking speed and cannot infer the precise position of the target.

[0004] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0005] A multi-target tracking method based on a deep path aggregation network comprises the following steps:

[0006] Step 1: Data preprocessing: For each frame of each video in the training set, use rotation, scaling and color jittering to perform data enhancement to obtain the input data set of the network;

[0007] Step 2: Build a deep path aggregation network, including the following sub-steps:

[0008] Step 2.1, design the network structure of the deep path aggregation network;

[0009] Step 2.2: Construct training samples, select images from the input data set, and input them into the deep path aggregation network as the input of the network;

[0010] Step 2.3, design the error function for back propagation and optimize the network parameters until convergence;

[0011] Step 3: Perform multi-target tracking for each detected object in the video: extract the detection area and re-ID features of the target based on the trained deep path aggregation network, calculate the cosine distance of the re-ID features of each target in a certain frame of the video and the previous frame and the IoU of the detection box, use the Kalman filter for tracking prediction, and obtain the position of the target in the current frame.

[0012] Furthermore, the data preprocessing steps in step 1 are as follows:

[0013] For each frame of each video in the training set, we randomly selected an angle between -10 and 10 for rotation, then performed an image scaling operation with a ratio of 4, and finally increased the image color depth and image brightness by 0.5 times to obtain the input data set of the network.

[0014] Furthermore, the network structure of the deep path aggregation network is designed in step 2.1, which specifically includes the following steps:

[0015] Step 201: Based on the DLA network, three feature layers are added from top to bottom in the final stage to downsample and aggregate the output feature map of the DLA network, thereby obtaining three feature maps of different resolutions;

[0016] Step 202: For the three feature maps with different resolutions, two of the feature maps with small and medium resolutions and another feature map with large resolution are multi-scale aggregated to obtain a high-resolution feature map representation. The high-resolution feature map representation is the output feature map of the deep path aggregation network, and the output feature map can be used to output the center, heat map, offset and re-ID feature vector of the target.

[0017] Furthermore, in step 2.2, constructing a training sample specifically includes the following steps:

[0018] Select 4 images from the input dataset and input them into the deep path aggregation network.

[0019] Furthermore, in step 2.3, an error function is designed for back propagation to optimize the network parameters until convergence, specifically:

[0020] The Focal Loss loss function is used to calculate the center heat map of each target in the image, the L1 Loss loss function is used to calculate the center offset and prediction box size of each target in the image, and the ID Loss loss function is used to calculate the re-ID embedding loss value of each target in the image. Then, corresponding weight values ​​are assigned to these three loss values ​​to form a total loss value. After training, the center position, prediction box size and re-ID features of each target in the image can be obtained. The total loss value is:

[0021]

[0022] L detection =L heat +L box

[0023] Where w1 is a learnable parameter for adjusting the target detection in the total loss function; w2 is a learnable parameter for adjusting the re-ID task in the total loss function, which is used to balance the target detection and re-ID tasks; e is a natural constant in mathematics; L identity represents the loss function of the re-ID task; L detection Represents the loss function for the detection task, which includes the heatmap loss, predicted box size and offset loss for each object in the image.

[0024] Furthermore, step 3 of performing multi-target tracking in the video specifically includes the following steps:

[0025] Step 301: In a video, for a certain frame image, input it into the trained deep path aggregation network to obtain its convolution feature map, and then generate a target heat map feature map, a prediction box size feature map, a prediction box offset feature map and a re-ID embedding feature map respectively;

[0026] Step 302: In the subsequent frame, based on all target positions and re-ID features inferred from the previous frame, online tracking is performed for each target. The steps are:

[0027] Step 3021: Calculate the detection boxes and re-ID features of all targets in the current frame using the trained deep path aggregation network;

[0028] Step 3022: Use a Kalman filter to predict the position in the current frame based on the motion trajectory of each target in the previous frame;

[0029] Step 3023: Then calculate the cosine distance of the re-ID features of all targets in the previous frame and all targets in the current frame and the IoU of the detection box. If the cosine distance is greater than 0.4 and the IoU score is greater than 0.5, it is considered that the tracking is successful, and the successfully tracked target is connected to the existing motion trajectory.

[0030] Step 3024: Use the Kalman filter to perform state update for all successfully tracked targets to obtain the optimal estimate in the current frame.

[0031] The present invention also proposes a multi-target tracking system based on a deep path aggregation network, comprising a data preprocessing unit, a deep path aggregation network training unit and a video multi-target tracking unit;

[0032] The data preprocessing unit is used to input an image sequence and perform data enhancement on the image sequence by rotation, scaling and color jittering;

[0033] The deep path aggregation network training unit is used to train the designed deep path aggregation network and is configured to perform the following steps:

[0034] Step A, designing the network structure of the deep path aggregation network;

[0035] Step B: construct training samples as input to the network;

[0036] Step C: Design an error function for back propagation and optimize the network parameters until convergence;

[0037] The video multi-target tracking unit is configured to perform the following actions: extract the detection area and re-ID features of the target based on the trained deep path aggregation network, and use the Kalman filter to perform tracking prediction according to the cosine distance of the re-ID feature of each target and the IoU of the detection box to obtain the position of the target in the current frame.

[0038] The multi-target tracking method and system based on deep path aggregation network of the present invention have the following advantages:

[0039] 1. The multi-target tracking method based on deep path aggregation network proposed in the present invention can be used to track all targets in each frame in any video;

[0040] 2. The present invention utilizes a trained deep path aggregation network and can output features of different levels of the target through multi-scale aggregation across resolutions, making it more robust to changes in the appearance of the target.

[0041] 3. The present invention adopts a bottom-up feature fusion layer to extract the target space feature information, thereby shortening the information path between the bottom and top features, so that the tracker has higher tracking accuracy and can also ensure the real-time tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the multi-target tracking method based on deep path aggregation network of the present invention. DETAILED DESCRIPTION

[0043] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a multi-target tracking method and system based on a deep path aggregation network of the present invention in conjunction with the accompanying drawings.

[0044] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0045] This paper first proposes a multi-target tracking method based on a deep path aggregation network. Figure 1 As shown, the method comprises the following steps:

[0046] Step 1: Data preprocessing step: For the input image sequence, use rotation, scaling and color jitter to perform data enhancement to obtain the input data set of the network;

[0047] For each frame of each video in the training set, we randomly selected an angle between -10 and 10 for rotation, then performed an image scaling operation with a ratio of 4, and finally increased the image color depth and image brightness by 0.5 times to obtain the input data set of the network.

[0048] Step 2: Build a deep path aggregation network, including the following sub-steps:

[0049] Step 2.1: Design the network structure of the deep path aggregation network, which specifically includes the following steps:

[0050] Step 201, based on the DLA network, three feature layers are added from top to bottom in the final stage to downsample and aggregate the output feature map of the DLA network, thereby obtaining three feature maps of different resolutions; the deep layer aggregation network (DLA) is a neural network that can extract features and perform multi-scale aggregation. The deep path aggregation network proposed in this paper is an improvement based on the DLA network.

[0051] Step 202: For the three feature maps with different resolution sizes outputted, two of the feature maps with small and medium resolutions and another feature map with large resolution are multi-scale aggregated to obtain a high-resolution feature map representation; the high-resolution feature map representation is the output feature map of the deep path aggregation network, and the output feature map can be used to output the center, heat map, offset and re-ID feature vector of the target.

[0052] Step 2.2: Construct training samples, select 4 images from the input data set, and input them into the deep path aggregation network as the input of the network;

[0053] Step 2.3: Design an error function for back propagation and optimize the network parameters until convergence. Specifically:

[0054] Use Focal Loss, L1 Loss, and ID Loss to calculate the center heat map, center offset, prediction box size, and re-ID embedding loss values ​​of each target in the image respectively; then, assign corresponding weight values ​​to these three loss values ​​to form a total loss value; after training, the center position, prediction box size, and re-ID features of each target in the image can be obtained; Focal Loss, L1 Loss, and ID Loss are some common loss functions in the field of deep learning, which can be understood as functions that calculate the target center heat map loss, center offset and bounding box size loss, and re-ID embedding loss respectively. Re-ID is the abbreviation of re-identification, which means re-identification. The deep path aggregation network will eventually output the re-ID feature vectors of all targets in each frame of the video. In the tracking stage, the cosine distance of all re-ID feature vectors in two adjacent frames can be calculated to re-identify the position of the target in the previous frame in the next frame. The total loss value is:

[0055]

[0056] L detection =L heat +L box

[0057] Where w1 is the learnable parameter for adjusting the target detection in the total loss function; w2 is the learnable parameter for adjusting the re-ID task in the total loss function, which is used to balance the target detection and re-ID tasks; e is a natural constant, L identity represents the loss function of the re-ID task; L detection Represents the loss function for the detection task, which includes the heatmap loss, predicted box size and offset loss for each object in the image.

[0058] Step 3: Perform multi-target tracking for each detected object in the video: Extract the detection area and re-ID features of the target based on the trained deep path aggregation network, calculate the cosine distance of the re-ID features of each target in a certain frame of the video and the previous frame and the IoU of the detection box. IoU represents the overlap between two boxes. It is often used in the field of target detection to determine the overlap between the predicted box and the real box, so that the size of the predicted box is continuously adjusted during the training process to approximate the position and size of the real box. Use the Kalman filter for tracking prediction to obtain the position of the target in the current frame, which specifically includes the following steps:

[0059] Step 301: In a video, for a certain frame image, input it into the trained deep path aggregation network to obtain its convolution feature map, and then generate a target heat map feature map, a prediction box size feature map, a prediction box offset feature map and a re-ID embedding feature map respectively;

[0060] Step 302: In the subsequent frame, based on all target positions and re-ID features inferred from the previous frame, online tracking is performed for each target. The steps are:

[0061] Step 3021: Calculate the detection boxes and re-ID features of all targets in the current frame using the trained deep path aggregation network;

[0062] Step 3022: Use a Kalman filter to predict the position in the current frame based on the motion trajectory of each target in the previous frame;

[0063] Step 3023: Then calculate the cosine distance of the re-ID features of all targets in the previous frame and all targets in the current frame and the IoU of the detection box. If the cosine distance is greater than 0.4 and the IoU score is greater than 0.5, it is considered that the tracking is successful, and the successfully tracked target is connected to the existing motion trajectory.

[0064] Step 3024: Use the Kalman filter to perform state update for all successfully tracked targets to obtain the optimal estimate in the current frame.

[0065] The present invention also proposes a multi-target tracking system based on a deep path aggregation network, comprising a data preprocessing unit, a deep path aggregation network training unit and a video multi-target tracking unit;

[0066] The data preprocessing unit is used to input an image sequence and perform data enhancement on the image sequence by rotation, scaling and color jittering;

[0067] The deep path aggregation network training unit is used to train the designed deep path aggregation network and is configured to perform the following steps:

[0068] Step A, designing the network structure of the deep path aggregation network;

[0069] Step B: construct training samples as input to the network;

[0070] Step C: Design an error function for back propagation and optimize the network parameters until convergence;

[0071] The video multi-target tracking unit is configured to perform the following actions: extract the detection area and re-ID features of the target based on the trained deep path aggregation network, and use the Kalman filter to perform tracking prediction according to the cosine distance of the re-ID feature of each target and the IoU of the detection box to obtain the position of the target in the current frame.

[0072] It can be understood by those skilled in the art that each block in these structure diagrams and / or block diagrams and / or flow charts and combinations of blocks in these structure diagrams and / or block diagrams and / or flow charts can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a specialized computer, or a processor of other programmable data processing methods to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing method create a method for implementing the block or multiple blocks specified in the structure diagram and / or block diagram and / or flow chart.

[0073] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A multi-target tracking method based on a deep path aggregation network, characterized in that: The following steps are involved: Step 1: Data preprocessing: For each frame of each video in the training set, use rotation, scaling and color jittering to perform data enhancement to obtain the input data set of the network; Step 2: Build a deep path aggregation network, including the following sub-steps: Step 2.1, design the network structure of the deep path aggregation network; The network structure of the deep path aggregation network is designed in step 2.1, specifically including the following steps: Step 201: Based on the DLA network, three feature layers are added from top to bottom in the final stage to downsample and aggregate the output feature map of the DLA network, thereby obtaining three feature maps of different resolutions; Step 202: for the three output feature maps with different resolutions, multi-scale aggregation is performed on two of the feature maps with small and medium resolutions and another feature map with large resolution to obtain a high-resolution feature map representation; Step 2.2: Construct training samples, select images from the input data set, and input them into the deep path aggregation network as the input of the network; Step 2.3, design the error function for back propagation and optimize the network parameters until convergence; Step 3: Perform multi-target tracking for each detected object in the video: extract the detection area and re-ID features of the target based on the trained deep path aggregation network, calculate the cosine distance of the re-ID features of each target in a certain frame of the video and the previous frame and the IoU of the detection box, use the Kalman filter for tracking prediction, and obtain the position of the target in the current frame.

2. The multi-target tracking method based on deep path aggregation network according to claim 1 is characterized in that: The data preprocessing steps in step 1 are as follows: For each frame of each video in the training set, we randomly selected an angle between -10 and 10 for rotation, then performed an image scaling operation with a ratio of 4, and finally increased the image color depth and image brightness by 0.5 times to obtain the input data set of the network.

3. The multi-target tracking method based on deep path aggregation network according to claim 1 is characterized in that: The training samples are constructed in step 2.2, specifically including the following steps: Select 4 images from the input dataset and input them into the deep path aggregation network.

4. The multi-target tracking method based on deep path aggregation network according to claim 3 is characterized in that: In step 2.3, the error function is designed to perform back propagation and optimize the parameters of the network until convergence, specifically: The Focal Loss function is used to calculate the center heat map of each object in the image, the L1 Loss function is used to calculate the center offset and prediction box size of each object in the image, and the ID Loss function is used to calculate the re-ID embedding loss value of each object in the image. Then, corresponding weight values ​​are assigned to these three loss values ​​to form a total loss value. After training, the center position, prediction box size and re-ID features of each object in the image are obtained. The total loss value is: L detection =L heat +L box Where w1 is the learnable parameter for adjusting the target detection in the total loss function; w2 is the learnable parameter for adjusting the re-ID task in the total loss function, which is used to balance the target detection and re-ID tasks; e is a natural constant, L identity represents the loss function of the re-ID task; L detection Represents the loss function for the detection task, which includes the heatmap loss, predicted box size and center offset loss for each object in the image.

5. The multi-target tracking method based on deep path aggregation network according to claim 1 is characterized in that: The step 3 of performing multi-target tracking in the video specifically includes the following steps: Step 301: In a video, for a certain frame image, input it into the trained deep path aggregation network to obtain its convolution feature map, and then generate a target heat map feature map, a prediction box size feature map, a prediction box offset feature map and a re-ID embedding feature map respectively; Step 302: In the subsequent frame, based on all target positions and re-ID features inferred from the previous frame, online tracking is performed for each target. The steps are: Step 3021: Calculate the detection boxes and re-ID features of all targets in the current frame using the trained deep path aggregation network; Step 3022: Use a Kalman filter to predict the position in the current frame based on the motion trajectory of each target in the previous frame; Step 3023: Then calculate the cosine distance of the re-ID features of all targets in the previous frame and all targets in the current frame and the IoU of the detection box. If the cosine distance is greater than 0.4 and the IoU score is greater than 0.5, it is considered that the tracking is successful, and the successfully tracked target is connected to the existing motion trajectory. Step 3024: Use the Kalman filter to perform state update for all successfully tracked targets to obtain the optimal estimate in the current frame.

6. A multi-target tracking system based on a deep path aggregation network according to any one of claims 1 to 5, characterized in that: It includes a data preprocessing unit, a deep path aggregation network training unit and a video multi-target tracking unit; The data preprocessing unit is used to input an image sequence and perform data enhancement on the image sequence by rotation, scaling and color jittering; The deep path aggregation network training unit is used to train the designed deep path aggregation network and is configured to perform the following steps: Step A, designing the network structure of the deep path aggregation network; Step B: construct training samples as input to the network; Step C: Design an error function for back propagation and optimize the network parameters until convergence; The video multi-target tracking unit is configured to perform the following actions: extract the detection area and re-ID features of the target based on the trained deep path aggregation network, and use the Kalman filter to perform tracking prediction according to the cosine distance of the re-ID feature of each target and the IoU of the detection box to obtain the position of the target in the current frame.

Images