A multi-target tracking method based on graph convolutional neural network

Abstract

The present invention provides a multi-target tracking method based on graph convolutional neural network, which uses target appearance features, target motion features and interaction features in the tracking process. The adopted scheme is: first, construct a tracking model consisting of a re-identification (Re-identification, ReID) module, an appearance GCN, a motion GCN, an appearance fusion module, a position fusion module, an edge weight fusion module and a feature similarity fusion module. Then, the target appearance information and position information are used as model inputs, and the multi-level fusion of target appearance information and motion information is realized by using multiple fusion modules in the tracking model. Next, the above-mentioned target tracking model is trained using the Adam optimizer and the binary cross entropy loss function. Then, the detection results obtained by the detector are input into the trained tracking model to obtain the correlation matrix, and the Hungarian algorithm is used to obtain the preliminary matching results. Finally, the linear interpolation algorithm is used to obtain the multi-target tracking results.

Description

Technical Field

[0001] The present invention relates to a tracking technology, in particular to a multi-target tracking technology based on a graph convolutional neural network, and belongs to the field of computer vision. Background Art

[0002] Multi-object tracking is a crucial component of computer vision and is widely used in fields such as video surveillance, scene understanding, and autonomous driving. The primary task of multi-object tracking is to identify individual objects and associate them as multiple continuous tracks within a video sequence. However, in complex scenarios such as those with occlusion and similar appearance, the tracked objects exhibit poor continuity, and the tracking results fall short of practical requirements. Therefore, research on multi-object tracking in complex scenarios holds great value both in theoretical research and practical applications.

[0003] Existing target tracking methods can be divided into two categories: post-detection tracking and tracking-while-detection. The former has gained favor in academia and industry due to its superior performance. After obtaining detection results, post-detection tracking methods primarily use appearance and motion features for tracking. However, under occlusion, the target's appearance changes, and the motion features of the occluded and occluded objects are relatively similar. Relying solely on these two features for tracking results in frequent identity switching, low tracking accuracy, and decreased tracking rate.

[0004] Research has shown that using a graph convolutional neural network (GCN) to introduce interaction information between objects can better distinguish objects. Consequently, several GCN-based tracking methods have been proposed, such as the graph convolutional neural network matching (GCNNMatch) proposed by Papakis et al., the graph matching tracker (GMTracker) proposed by He et al., and the multi-object tracking algorithm via neighbor graph (MOTNG) proposed by Liang et al. Although these methods can improve tracking performance, their performance is still unsatisfactory in complex scenarios. Summary of the Invention

[0005] In order to solve the problem of low target tracking accuracy and tracking rate in occlusion scenes, the present invention designs a multi-target tracking technology that can fully utilize the interactive information between objects.

[0006] The object of the present invention is achieved in that the steps are as follows:

[0007] Step 1: Tracking network construction. The tracking network mainly consists of 7 modules: ReID module, appearance GCN, motion GCN, appearance fusion module, position fusion module, edge weight fusion module and feature similarity fusion module;

[0008] Step 2: Tracking network training: Let the input be a video image and its detection result of MOT17-Trackor. Let the detection result of each target be represented by (t,x,y,w,h), where x,y are the horizontal and vertical coordinates of the upper left corner of the target box, w,h are the width and height of the target box, and t is the timestamp.

[0009] Step 3: Multi-target tracking:

[0010] Step 3.1: Input the video to be tracked and use the MOT17-Trackor detector to obtain the detection results;

[0011] Step 3.2: Use the tracking model trained in step 2 to obtain the correlation matrix S;

[0012] Step 3.3: Input the incidence matrix S into the Hungarian algorithm to obtain the preliminary matching results;

[0013] Step 3.4: Update the trajectory of each target according to the preliminary matching results;

[0014] Step 3.5: Repeat steps 3.1-3.4 until the last frame of the video to obtain preliminary matching results for all video frame targets;

[0015] Step 3.6: Use the linear interpolation algorithm to process the preliminary matching results of all video frame targets to obtain the final multi-target tracking results.

[0016] The present invention also includes such structural features:

[0017] 1. Step 1 specifically includes:

[0018] Step 1.1: Build the appearance GCN: The appearance GCN consists of two graph convolutional layers and one batch normalization layer, and uses the ReLU activation function after the first graph convolutional layer; the appearance GCN is used to encode appearance interaction information;

[0019] Step 1.2: Construct Motion GCN: Motion GCN also consists of 2 graph convolution layers and 1 batch normalization layer, and uses ReLU activation function after the first graph convolution layer; Motion GCN is used to encode motion interaction information

[0020] Step 1.3: Construct the appearance fusion module, position fusion module, edge weight fusion module and feature similarity fusion module; each fusion module consists of a linear layer with ReLU activation function.

[0021] 2. Step 2 specifically includes:

[0022] Step 2.1: Use the object detection results of MOT17-Trackor to crop the image blocks corresponding to each target in the input video. Each image block is input into the ReID module to obtain the 512-dimensional appearance features of each target. The average features of the objects in the track are used as the track appearance features.

[0023] Step 2.2: Calculate edge weights: For any object in frame t-1 and any object in frame t, if the distance between them is less than the distance threshold, then:

[0024] Step 2.2.1: Concatenate the appearance features and input them into the appearance fusion module to obtain the appearance similarity;

[0025] Step 2.2.2: Concatenate the location coordinates and input them into the location fusion module to obtain the location similarity;

[0026] Step 2.2.3: Concatenate the appearance similarity and position similarity and input them into the edge weight fusion module to obtain the edge weight;

[0027] Step 2.3: Construct the input and initial edge weights of the motion GCN: the position coordinates of the object in the current frame and the previous frame are used as the input features of the motion GCN, and the calculation results of step 2.2 are used as the initial edge weights;

[0028] Step 2.4: Construct the input and initial edge weights of the appearance GCN: the appearance features of the current frame and the average features of the objects in the trajectory are used as the input features of the appearance GCN, and the calculation results in step 2.2 are used as the initial edge weights;

[0029] Step 2.5: Calculate the appearance interaction feature cosine similarity, motion interaction feature cosine similarity, appearance feature cosine similarity, and bounding box overlap, and input them into the feature similarity fusion module;

[0030] Step 2.6: Construct (M+1)×(N+1) dimensional incidence matrix S=[s m,n ]: Where M represents the number of objects in the trajectory, N represents the number of detected objects in the current frame, and s m,n is the output result of the feature similarity fusion module in step 2.5, m = 1…M represents the mth trajectory object, n = 1…N represents the nth detection object, and the last row and column of S are all 1;

[0031] Step 2.7: Update the matrix S using the optimal transmission algorithm Sinkhorn shown in formula (1) and formula (2);

[0032]

[0033]

[0034] Step 2.6: Train the above model using the Adam optimizer and weighted binary cross entropy loss function to obtain the trained multi-target tracking model.

[0035] Compared with the prior art, the present invention has the following advantages: the present invention can obtain appearance interaction features and motion interaction features containing global features; can effectively reduce identity changes during tracking; and can effectively reduce false positives in multi-target tracking.

[0036] The present invention designs a multi-target tracking technology based on GCN, which uses target appearance features, target motion features and interaction features in the tracking process. The adopted scheme is: first, construct a tracking model consisting of a re-identification (ReID) module, an appearance GCN, a motion GCN, an appearance fusion module, a position fusion module, an edge weight fusion module and a feature similarity fusion module. Then, the target appearance information and position information are used as model inputs, and the multi-level fusion of target appearance information and motion information is realized by using multiple fusion modules in the tracking model. Next, the above-mentioned target tracking model is trained using the Adam optimizer and the binary cross entropy loss function. Then, the detection results obtained by the detector are input into the trained tracking model to obtain the correlation matrix, and the Hungarian algorithm is used to obtain the preliminary matching results. Finally, the linear interpolation algorithm is used to obtain the multi-target tracking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The appearance feature extraction model structure designed for the present invention.

[0038] Figure 2 The target tracking model structure designed for the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Step 1, tracking network construction: Figure 1 and Figure 2 As shown in Figure 1, the tracking network of the present invention mainly includes 7 modules: ReID module, appearance GCN, motion GCN, appearance fusion module, position fusion module, edge weight fusion module and feature similarity fusion module. Among them, the ReID module uses the ReID model proposed by He et al.

[0041] Step 1.1: Construct an appearance GCN. The appearance GCN consists of two graph convolutional layers and one batch normalization layer, with a ReLU activation function after the first graph convolutional layer. The appearance GCN is used to encode appearance interaction information. The specific structural parameters are shown in Table 1.

[0042] Table 1 Structural parameters of appearance GCN

[0043]

[0044] In step 1.2, construct the Motion GCN. The Motion GCN also consists of two graph convolutional layers and one batch normalization layer, with a ReLU activation function used after the first graph convolutional layer. The Motion GCN is used to encode motion interaction information. The specific structural parameters are shown in Table 2.

[0045] Table 2 Structural parameters of motion GCN

[0046]

[0047] In step 1.3, we construct the appearance fusion module, position fusion module, edge weight fusion module, and feature similarity fusion module. Each fusion module consists of a linear layer with a ReLU activation function. The specific structural parameters are shown in Table 3.

[0048] Table 3 Structural parameters of the fusion module

[0049]

[0050] Step 2: Tracking network training: Let the input be a video image and its detection results from the MOT17-Trackor. Let the detection result of each target be represented by (t, x, y, w, h), where x, y are the horizontal and vertical coordinates of the upper left corner of the target box, w, h are the width and height of the target box, and t is the timestamp.

[0051] Step 2.1, as Figure 1 As shown in Figure 1, the target detection results of MOT17-Trackor are used to crop the image blocks corresponding to each target in the input video. Each image block is input into the ReID module to obtain the 512-dimensional appearance features of each target, and the average features of the objects in the track are used as the track appearance features.

[0052] Step 2.2, calculate edge weights: For any object in frame t-1 and any object in frame t, if the distance between them is less than the distance threshold, then:

[0053] Step 2.2.1: Concatenate the appearance features and input them into the appearance fusion module to obtain the appearance similarity.

[0054] Step 2.2.2: Concatenate the position coordinates and input them into the position fusion module to obtain the position similarity.

[0055] In step 2.2.3, the appearance similarity and position similarity are concatenated and input into the edge weight fusion module to obtain the edge weight.

[0056] Step 2.3, construct the input and initial edge weights of the motion GCN: the position coordinates of the objects in the current frame and the previous frame are used as the input features of the motion GCN, and the calculation results of step 2.2 are used as the initial edge weights.

[0057] Step 2.4, construct the input and initial edge weights of the appearance GCN: the appearance features of the current frame and the average features of the objects in the trajectory are used as the input features of the appearance GCN, and the calculation results in step 2.2 are used as the initial edge weights.

[0058] In step 2.5, the cosine similarity of the appearance interaction feature (output of the appearance GCN), the cosine similarity of the motion interaction feature (output of the motion GCN), the cosine similarity of the appearance feature and the bounding box overlap are calculated and input into the feature similarity fusion module.

[0059] Step 2.6, construct (M+1)×(N+1) dimensional incidence matrix S=[s m,n ]: Where M represents the number of objects in the trajectory, N represents the number of detected objects in the current frame, and s m,n is the output result of the feature similarity fusion module in step 2.5, m=1…M represents the mth trajectory object, n=1…N represents the nth detection object, and the last row and column of S are all 1.

[0060] In step 2.7, the matrix S is updated using the optimal transmission algorithm Sinkhorn shown in formula (1) and formula (2).

[0061]

[0062]

[0063] In step 2.6, the above model is trained using the Adam optimizer and the weighted binary cross entropy loss function to obtain a trained multi-target tracking model.

[0064] Step 3, multi-target tracking:

[0065] Step 3.1: Input the video to be tracked and use the MOT17-Trackor detector to obtain the detection results.

[0066] Step 3.2: Use the tracking model trained in step 2 to obtain the correlation matrix S.

[0067] In step 3.3, the correlation matrix S is input into the Hungarian algorithm to obtain the preliminary matching results.

[0068] Step 3.4: Update the trajectory of each target according to the preliminary matching results.

[0069] In step 3.5, repeat steps 3.1-3.4 until the last frame of the video to obtain the preliminary matching results of the targets in all video frames.

[0070] In step 3.6, the linear interpolation algorithm is used to process the preliminary matching results of all video frame targets to obtain the final multi-target tracking results.

[0071] The MOT17 dataset was selected to test the target tracking method proposed in this paper, and the results are shown in Table 4. The experiments show that on the MOT17 test set, the multi-target tracking accuracy, identity F1 score, and the number of targets with correctly predicted trajectories exceeding 80% are all higher than those of GCNNMatch (the higher the better); the number of targets with correctly predicted trajectories not exceeding 20%, false negatives, and ID changes are all lower than those of GCNNMatch (the lower the better); only the false positive rate is higher than that of GCNNMatch (the lower the better), achieving better tracking results overall.

[0072] Table 4 Experimental results

[0073]

[0074]

Claims

1. A multi-target tracking method based on graph convolutional neural network, characterized in that: Here are the steps: Step 1: Tracking network construction. The tracking network mainly consists of 7 modules: ReID module, appearance GCN, motion GCN, appearance fusion module, position fusion module, edge weight fusion module and feature similarity fusion module; Step 1.1: Build the appearance GCN: The appearance GCN consists of two graph convolutional layers and one batch normalization layer, and uses the ReLU activation function after the first graph convolutional layer; the appearance GCN is used to encode appearance interaction information; Step 1.2: Construct Motion GCN: Motion GCN also consists of 2 graph convolution layers and 1 batch normalization layer, and uses ReLU activation function after the first graph convolution layer; Motion GCN is used to encode motion interaction information Step 1.3: Construct the appearance fusion module, position fusion module, edge weight fusion module, and feature similarity fusion module; each fusion module consists of a linear layer with ReLU activation function; Step 2: Tracking network training: Let the input be a video image and its target detection results of MOT17-Trackor. Let the detection result of each target be represented by (t,x,y,w,h), where x,y are the horizontal and vertical coordinates of the upper left corner of the target box, w,h are the width and height of the target box, and t is the timestamp. Step 2.1: Use the object detection results of MOT17-Trackor to crop the image blocks corresponding to each target in the input video, input each image block into the ReID module to obtain the appearance features of each target, and use the average features of the objects in the track as the track appearance features; Step 2.2: For any object in frame t-1 and any object in frame t, if the distance between them is less than the distance threshold, concatenate the appearance features and input them into the appearance fusion module to obtain the appearance similarity; concatenate the position coordinates and input them into the position fusion module to obtain the position similarity; concatenate the appearance similarity and position similarity and input them into the edge weight fusion module to obtain the edge weight; Step 2.3: Construct the input and initial edge weights of the motion GCN: the position coordinates of the object in the current frame and the previous frame are used as the input features of the motion GCN, and the calculation results of step 2.2 are used as the initial edge weights; Step 2.4: Construct the input and initial edge weights of the appearance GCN: the appearance features of the current frame and the average features of the objects in the trajectory are used as the input features of the appearance GCN, and the calculation results in step 2.2 are used as the initial edge weights; Step 2.5: Calculate the appearance interaction feature cosine similarity, motion interaction feature cosine similarity, appearance feature cosine similarity, and bounding box overlap, and input them into the feature similarity fusion module; Step 2.6: Construct (M+1)×(N+1) dimensional incidence matrix S=[s m,n ]: Where M represents the number of objects in the trajectory, N represents the number of detected objects in the current frame, and s m,n is the output result of the feature similarity fusion module in step 2.5, m = 1…M represents the mth trajectory object, n = 1…N represents the nth detection object, and the last row and column of S are all 1; Step 2.7: Update the matrix S using the optimal transmission algorithm Sinkhorn shown in formula (1) and formula (2); Step 2.8: Train the above model using the Adam optimizer and weighted binary cross entropy loss function to obtain a trained multi-target tracking model; Step 3: Multi-target tracking: Step 3.1: Input the video to be tracked and use the MOT17-Trackor detector to obtain the detection results; Step 3.2: Use the tracking model trained in step 2 to obtain the correlation matrix S; Step 3.3: Input the incidence matrix S into the Hungarian algorithm to obtain the preliminary matching results; Step 3.4: Update the trajectory of each target according to the preliminary matching results; Step 3.5: Repeat steps 3.1-3.4 until the last frame of the video to obtain preliminary matching results for all video frame targets; Step 3.6: Use the linear interpolation algorithm to process the preliminary matching results of all video frame targets to obtain the final multi-target tracking results.

Images