A multi-target pedestrian tracking method based on Attention-JDE network

Abstract

This paper provides a multi-target pedestrian tracking method based on an Attention-JDE network. The method includes inputting a resolution-adjusted image into a DarkNet-53 backbone network to extract feature maps at three different scales. The method also includes the following steps: using the feature maps at the three scales as input to construct a feature pyramid; feeding the outputs of the feature pyramid into a feature enhancement module to refine the features and output new feature maps at three new scales; using the new feature maps to detect pedestrians, generate detection frames, and use the detection frames and corresponding embedded features to match the same target in previous and subsequent frames to form a trajectory. The present invention designs a feature enhancement module based on the attention mechanism, which allows the model to focus more on important features and can combine features from multiple receptive fields. By adding the feature enhancement module and introducing the Mish activation function, the model's tracking performance for small and overlapping targets can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a pedestrian multi-target tracking method based on an Attention-JDE network. Background Art

[0002] Object tracking is a hot research topic in computer vision. It primarily focuses on detecting and locating objects of interest in video sequences, and then using algorithms to predict their positions, states, and other information in subsequent sequences. Multiple object tracking (MOT) involves identifying and tracking multiple objects in a video without prior knowledge of their appearance or number. MOT algorithms typically use object detection algorithms to delimit the objects of interest and then associate their IDs with these detection boxes. Different objects should be assigned different IDs to distinguish them within the class and form a complete tracking trajectory. Pedestrian multi-object tracking, a branch of multi-object tracking, is the underlying foundation for pedestrian behavior analysis. It supports advanced semantic tasks such as scene understanding and crowd behavior analysis and is widely used in intelligent video surveillance, intelligent transportation, and autonomous driving. Pedestrian tracking requires first extracting, processing, and analyzing pedestrian features from each frame of the video data. This information is then matched with the corresponding information in the next frame to complete tracking.

[0003] The difficulty of multi-target tracking primarily stems from various occlusions and intersections between targets, which often have highly similar appearances. Therefore, the design of MOT algorithms is often more complex. With the rapid development of deep learning, an increasing number of algorithms are leveraging its representational capabilities to address multi-target tracking. However, for pedestrians in complex scenes, occlusions are more frequent, and the scale of pedestrians varies greatly, making small-scale targets difficult to track. This still presents challenges for deep learning methods. Furthermore, multi-target tracking processes image sequences, which places high demands on real-time performance. The current trend is to achieve better performance through the design of deeper networks, which increases the demand for computational resources. However, the computational power in practical applications often comes from embedded or mobile devices, which is far from sufficient to meet the growing demand for computing resources. Therefore, deep network models face numerous challenges in practical applications. Therefore, research on how to enhance the capabilities of deep models for pedestrian tracking in complex scenes, reduce the computational resources required, and meet the deployment and real-time requirements of practical applications is of great research significance and practical value.

[0004] Multi-object tracking methods can be categorized into two-step and one-shot approaches. Two-step multi-object tracking methods, also known as tracking-by-detection methods, first obtain a set of bounding boxes containing the objects of interest from a video frame. Bounding boxes containing the same object in previous and subsequent frames are then assigned the same ID. These methods first detect the objects in the video frame, then crop the objects according to the detection bounding boxes. The cropped parts are then fed into an embedding model to extract embedded features. Finally, the embedded features and the detection bounding boxes are used to associate the same objects for tracking. Therefore, the quality of the object detection algorithm significantly impacts the final tracking results. Simple Online and Realtime Tracking (SORT) proposed by Bewley et al. is one of the earliest MOT algorithms to use convolutional neural networks for pedestrian detection. This method replaces detections obtained using aggregated channel features with detections computed by Faster R-CNN. They employ a relatively simple association method: using a Kalman filter to predict the object's motion state. They then employ the Hungarian algorithm to calculate a cost matrix using the Intersection of Union (IoU) distance to associate the detections. The SORT method uses only the intersection-over-union (IoU) distance of the detection bounding boxes as a matching metric, which results in a high number of ID switches during tracking. To address this issue, the DeepSORT method further uses a neural network to extract pedestrian target features as re-ID features. It then uses the cosine distance between all target features to calculate a cost matrix, and combines this with the IoU distance to correlate the matches between targets in the previous and next frames. This method ultimately reduces the number of ID switches by approximately 45% compared to SORT. However, even considering only the time spent extracting embedded features, DeepSORT only achieves a tracking speed of 17.4 FPS. The paper also uses Faster R-CNN for object detection, but differs in that it uses a similarity score-based method to calculate the cost matrix and employs a grouping model for correlation. Yu et al. also achieved promising results using a modified Faster R-CNN. They added skip-pooling and multi-region features to the original Faster R-CNN, achieving good performance on the MOT pedestrian tracking dataset. Similar to DeepSORT, this method only achieves a tracking speed of 10 FPS. In addition, Mask R-CNN, which adds a segmentation branch to Faster R-CNN, was used by Zhou et al. to detect and track pedestrians and achieved good results. The association strategies used in these two methods are Kalman filtering and Hungarian algorithm.

[0005] One-shot methods have recently become a hot topic in the field of multi-object tracking. By integrating surface embedding and object detection into a single network, performing both detection and surface feature extraction in parallel, sharing most of the computational overhead, one-shot methods can achieve inference speeds far exceeding those of Tracking by Detection (TBD), albeit at a cost in accuracy. Track-RCNN adds a fully connected layer on top of Mask-RCNN to extract 128-dimensional embedding features for data association. This method not only tracks objects by detecting bounding boxes, but also tracks objects at the pixel level. JDE, on the other hand, extends the YOLOv3 prediction head to extract embedding features, then uses the Kalman filter and Hungarian algorithm to track and match objects. This transforms the original YOLOv3 method into a multi-task learning network model for classification, localization, and tracking. Compared to Tracking by Detection, this method saves the inference time of one Re-ID module, ultimately achieving near-real-time inference speed while approaching the accuracy of leading Tracking by Detection methods. FairMot builds on JDE's work and discovers that anchor boxes are not suitable for MOT tasks. This is because the anchor boxes and the target region may not align, leading to serious ambiguity during network training. Furthermore, when two targets partially overlap, different anchor boxes may be assigned to the same ID, reducing tracking accuracy. Based on this discovery, FairMot uses an anchor-free object detector as its foundation and extends CenterNet into an object tracking model, achieving superior performance to JDE in both accuracy and inference speed. Liang et al. argue that object classification and localization require class features, while Re-ID requires distinguishing between different objects within a class. In one-shot approaches, learning these two features simultaneously leads to competition between detection and Re-ID. This means that high performance in one task may lead to performance degradation in the other. To address this issue, they proposed a cross-correlation network to learn the commonalities and specificities of features for detection and Re-ID tasks. For specificity learning, self-relations reflecting the correlations between different feature channels are learned to enhance the feature representation of each task. For commonality learning, the shared information between two tasks can be learned through a carefully designed cross-relation mechanism.

[0006] In summary, most two-step methods offer high tracking accuracy, but their computational complexity leads to slow tracking speeds. Consequently, one-shot methods, which can significantly increase tracking speed while maintaining excellent accuracy, are gaining increasing attention. To address this, one-shot methods need to improve accuracy while retaining their speed advantage. This requires addressing issues such as the large scale variation of pedestrian targets in image data, overlapping occlusions, and other issues. Furthermore, further improving tracking speed in practical applications is crucial.

[0007] The invention patent application with publication number CN111767847A discloses a method for multi-target pedestrian tracking that integrates target detection and association, including the following steps: using a training dataset to train a tracking model network to obtain a tracking model; the first frame image in the video stream to be tracked first passes through the detection sub-model to generate a bounding box of the pedestrian target based on the heat map and offset vector; then the appearance feature extraction sub-model extracts feature vectors for each pedestrian target and assigns an ID and trajectory; the other frames in the video stream to be tracked pass through the tracking model in turn, and based on the similarity of the feature vectors between two adjacent frames in the video stream to be tracked, the corresponding trajectory position of each pedestrian in the current frame image is determined, and the trajectory positions corresponding to the same ID in all frames in the video stream to be tracked are connected to obtain the corresponding tracking results. The disadvantage of this method is that it cannot effectively detect and track small-scale pedestrian targets and overlapping targets, which means that the method's usability in certain complex scenarios needs to be improved. Summary of the Invention

[0008] In order to solve the above technical problems, the present invention proposes a pedestrian multi-target tracking method based on the Attention-JDE network, and designs a feature enhancement module based on the attention mechanism, so that the model pays more attention to important features and can combine features under multiple receptive fields. By adding this feature enhancement module and introducing the Mish activation function, the model's tracking effect for small targets and overlapping targets can be improved. Finally, the Attention-JDE algorithm achieved a higher MOTA than the original JDE method in the MOT16 dataset and the speed can reach 18.8FPS, with strong real-time performance.

[0009] The present invention provides a pedestrian multi-target tracking method based on the Attention-JDE network, comprising inputting a resolution-adjusted image into a DarkNet-53 backbone network to extract feature maps of three different scales, and further comprising the following steps:

[0010] Step 1: Take the feature maps of three scales as input and construct a feature pyramid;

[0011] Step 2: The output of the feature pyramid is sent to the feature enhancement module to complete feature refinement and output three new feature maps of new scales.

[0012] Step 3: Use the new feature map to detect pedestrians, generate a detection frame, and use the detection frame and the corresponding embedded features to match the same target in the previous and next frames to form a trajectory.

[0013] Preferably, the DarkNet-53 backbone network uses the Mish activation function, and the Mish activation function image is

[0014] f(x)=x tanh(ln(1+e x ))

[0015] Among them, x is the input feature.

[0016] In any of the above solutions, preferably, the resolution of the image is adjusted to 1088×608.

[0017] In any of the above solutions, preferably, the three scales are 1 / 8, 1 / 16 and 1 / 32 respectively.

[0018] In any of the above solutions, preferably, the feature enhancement module consists of a channel attention module, a spatial attention module and a spatial pyramid pooling module.

[0019] In any of the above solutions, it is preferred that the input of the channel attention module is the output of the previous convolution, expressed as in, Indicates the output dimension with C channels, W width and H height

[0020] In any of the above solutions, preferably, the working method of the channel attention module is:

[0021] Step 201: Use maximum pooling and average pooling to obtain global channel feature distribution information, which are expressed as and in, To represent the output dimension with C channels and 1 width and height, is the channel feature distribution information obtained by maximum pooling, is the channel feature distribution information obtained by average pooling;

[0022] Step 202: Pass the above two feature vectors through a multi-layer perceptron (MLP) with one hidden layer;

[0023] Step 203: Add the obtained feature vectors and perform a sigmoid operation σ to obtain the final channel attention weight M C (F);

[0024] Step 204: Through M C (F) is multiplied with the original input features to assign high weights to channels containing key information and low weights to other channels to achieve channel attention.

[0025] In any of the above solutions, it is preferred that the formula for the channel attention is:

[0026]

[0027] In any of the above solutions, it is preferred that the working method of the spatial attention module is:

[0028] Step 211: Perform maximum pooling and average pooling on the input feature map in the channel dimension to obtain and in, Indicates the output dimension with 1 channel and width W and height H. is the maximum pooling, is average pooling;

[0029] Step 212: Concatenate the two feature maps in the channel dimension and perform a 7×7 convolution operation;

[0030] Step 213: Perform a sigmoid operation (σ) on the feature map of the previous step to obtain the final spatial attention weight

[0031] Step 214: Assign a weight to each pixel of the original input feature by point-by-point multiplication.

[0032] In any of the above schemes, it is preferred that the formula of the spatial attention mechanism is

[0033]

[0034] Among them, f 7×7 It is a convolution operation with a convolution kernel size of 7.

[0035] In any of the above solutions, preferably, the spatial pyramid pooling module includes three maximum pooling operations, which are used to transform the input features into The maximum value pooling of 5×5, 9×9 and 13×13 is performed respectively, and the size of the feature map is maintained by padding 0 around the feature map. Then the feature maps of the three poolings are spliced ​​in the channel dimension to complete the feature fusion.

[0036] In any of the above solutions, preferably, the calculation process of spatial pyramid pooling is:

[0037]

[0038] Among them, Maxpooln×n () represents the maximum pooling operation with a kernel size of n, Indicates the concatenate operation, F in is the input feature, F out is the output feature.

[0039] In any of the above solutions, preferably, step 3 includes using the target detection branch to complete the classification and positioning of the target, and using the embedding branch to complete the extraction of the target's apparent features and use the apparent features for subsequent trajectory association.

[0040] In any of the above schemes, it is preferred that the target detection branch uses predefined anchor boxes for regression on the feature maps of three scales at the end of the network, and four sizes of anchor boxes are assigned to each scale. The predefined sizes of the anchor boxes are obtained by using k-means clustering on the benchmark data set, and a total of 12 categories are clustered. The cost formula of k-means is

[0041]

[0042] Among them, k is the number of cluster centers, c j is the jth cluster, μ j is the jth cluster center, given the sample {x (1) ,...,x (m)}, x (i) is a two-dimensional vector, i.e. width and height, and k cluster centers {μ1,...,μ k}, for each sample point, traverse all k cluster centers and calculate the (i) The center μ with the smallest Euclidean distance j , x (i) The category is set to μ j The corresponding cluster c i ,Right now

[0043] c i :=argmin||x (i) -μ j || 2

[0044] After updating the categories of all samples, calculate each cluster c i The mean of all samples in the cluster is taken as the new cluster center, and the next iteration is performed until the cluster center no longer changes or the maximum number of iterations is reached. The final cluster center is set as the Anchor box.

[0045] According to the formula, it is converted to the position in the actual image. The conversion formula is:

[0046] b x =σ(t x )+c x

[0047] b y =σ(t y )+c y

[0048]

[0049]

[0050] Among them, (b x ,b y ,b w ,b h ) represents the actual center coordinates and width and height of the bounding box, (t x ,t y ,t w ,t h ) are the center coordinates and width and height of the model output, c x and c y Represents the row and column number of the current grid cell, p w and p h Represent the width and height of Anchorbox respectively, and σ represents the sigmoid function.

[0051] In any of the above solutions, preferably, the working method of the embedded branch is:

[0052] Step 31: Add a convolution operation after the feature pyramid of the network to extract a 512-dimensional feature map as the apparent feature required for tracking, with the same size as the corresponding feature scale;

[0053] Step 32: Use the Kalman filter and Hungarian algorithm commonly used in multi-target tracking to complete trajectory association using the previously extracted appearance features.

[0054] This paper proposes a multi-target pedestrian tracking method based on the Attention-JDE network. The proposed feature enhancement module can effectively enhance the model's feature extraction capability, thereby improving the tracking effect for small-scale pedestrians and occluded pedestrians. The introduction of the Mish activation function further enhances the tracking accuracy of the model. By applying these improvements to the JDE method, the proposed Attention-JDE can achieve near real-time multi-target pedestrian tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 The figure is a flowchart of a preferred embodiment of a pedestrian multi-target tracking method based on the Attention-JDE network according to the present invention.

[0056] Figure 2 Schematic diagram of the structure of a preferred embodiment of the Attention-JDE network of the pedestrian multi-target tracking method based on the Attention-JDE network according to the present invention.

[0057] Figure 3 Schematic diagram of a preferred embodiment of the Mish activation function of the pedestrian multi-target tracking method based on the Attention-JDE network according to the present invention.

[0058] Figure 4 Schematic diagram of a preferred embodiment of a feature enhancement module of a pedestrian multi-target tracking method based on an Attention-JDE network according to the present invention.

[0059] Figure 5 Schematic diagram of a preferred embodiment of a channel attention module of a pedestrian multi-target tracking method based on an Attention-JDE network according to the present invention.

[0060] Figure 6 Schematic diagram of a preferred embodiment of the spatial attention module of the pedestrian multi-target tracking method based on the Attention-JDE network according to the present invention.

[0061] Figure 7 Schematic diagram of a preferred embodiment of the Attention-JDE output layer of the pedestrian multi-target tracking method based on the Attention-JDE network according to the present invention. DETAILED DESCRIPTION

[0062] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0063] Example 1

[0064] like Figure 1 As shown, step 100 is executed to input the image with adjusted resolution into the DarkNet-53 backbone network to extract feature maps of three different scales. The resolution of the image is adjusted to 1088×608, and the three scales are 1 / 8, 1 / 16 and 1 / 32 respectively. The DarkNet-53 backbone network uses the Mish activation function, and the Mish activation function image is

[0065] f(x)=x tanh(ln(1+e x ))

[0066] Among them, x is the input feature.

[0067] Step 110 is executed to construct a feature pyramid by taking the feature maps at three scales as input.

[0068] Execute step 120, and send the output of the feature pyramid to the feature enhancement module to complete feature refinement, and also output three new feature maps of new scales. The feature enhancement module consists of a channel attention module, a spatial attention module, and a spatial pyramid pooling module. The input of the channel attention module is the output of the previous convolution, which is expressed as in, The output dimension is C with a width of W and a height of H. The channel attention module works as follows:

[0069] Execute step 1201 and use maximum pooling and average pooling to obtain global channel feature distribution information, which are expressed as and in, Indicates the output dimension with C channels and 1 width and height. is the channel feature distribution information obtained by maximum pooling, is the channel feature distribution information obtained by average pooling;

[0070] Execute step 1202, passing the above two feature vectors through a multi-layer perceptron MLP with one hidden layer;

[0071] Execute step 1203, add the obtained feature vectors and perform a sigmoid operation σ to obtain the final channel attention weight M C (F);

[0072] Execute step 1204, through M C (F) is multiplied with the original input features to assign high weights to channels containing key information and low weights to other channels to achieve channel attention.

[0073] The formula for channel attention is

[0074]

[0075] The working method of the spatial attention module is as follows:

[0076] Execute step 1211 to perform maximum pooling and average pooling on the input feature map in the channel dimension to obtain and in, represents the output dimension with a channel number of 1 and a width of W and a height of H;

[0077] Execute step 1212 to concatenate the two feature maps in the channel dimension and perform a 7×7 convolution operation;

[0078] Execute step 1213 and perform a sigmoid operation (σ) on the feature map of the previous step to obtain the final spatial attention weight

[0079] Execute step 114 to assign a weight to each pixel of the original input feature by point-by-point multiplication.

[0080] The formula of the spatial attention mechanism is

[0081]

[0082] Among them, f 7×7 It is a convolution operation with a convolution kernel size of 7.

[0083] The spatial pyramid pooling module contains three maximum pooling operations, which transform the input features The maximum pooling of 5×5, 9×9 and 13×13 is performed respectively, and the size of the feature map is maintained by filling 0 around the feature map. Then the feature maps of the three poolings are spliced ​​in the channel dimension to complete the feature fusion. The calculation process of spatial pyramid pooling is:

[0084]

[0085] Among them, Maxpool n×n () represents the maximum pooling operation with a kernel size of n, Indicates the concatenate operation, F in is the input feature, F out is the output feature.

[0086] Execute step 130 to detect pedestrians using the new feature map, generate detection boxes, and use the detection boxes and corresponding embedded features to match the same target in previous and subsequent frames to form a track. Use the target detection branch to complete target classification and localization, and use the embedding branch to extract the target's surface features and use them for subsequent track association.

[0087] The target detection branch uses predefined anchor boxes for regression on the feature maps of three scales at the end of the network. Four sizes of anchor boxes are assigned to each scale. The predefined sizes of the anchor boxes are obtained by using k-means clustering on the benchmark dataset and clustered into 12 categories. The cost formula of k-means is

[0088]

[0089] Among them, k is the number of cluster centers, c j is the jth cluster, μ j is the jth cluster center, given the sample {x (1) ,...,x(m)}, x (i) In this paper, it is a two-dimensional vector, namely width and height, and k cluster centers {μ1,...,μ k}, for each sample point, traverse all k cluster centers and calculate the (i) The center μ with the smallest Euclidean distance j , change x (i) The category is set to μ j The corresponding cluster c i ,Right now

[0090] c i :=argmin||x (i) -μ j || 2

[0091] After updating the categories of all samples, calculate each cluster c i The mean of all samples in the cluster is taken as the new cluster center, and the next iteration is performed until the cluster center no longer changes or the maximum number of iterations is reached. The final cluster center is set as the Anchor box.

[0092] According to the formula, it is converted to the position in the actual image. The conversion formula is:

[0093] b x =σ(t x )+c x

[0094] b y =σ(t y )+c y

[0095]

[0096]

[0097] Among them, (b x ,b y ,b w ,b h ) represents the actual center coordinates and width and height of the bounding box, (t x ,t y ,t w ,t h ) are the center coordinates and width and height of the model output, c x and c y Represents the row and column number of the current grid cell, p w and p h Respectively represent the width and height of the Anchorbox, and σ represents the sigmoid function.

[0098] The working method of the embedded branch is:

[0099] Execute step 130 to add a convolution operation after the feature pyramid of the network to extract a 512-dimensional feature map as the surface feature required for tracking, with the same size as the corresponding feature scale;

[0100] Execute step 130 to complete trajectory association using the previously extracted surface features through Kalman filtering and Hungarian algorithm commonly used in multi-target tracking.

[0101] Example 2

[0102] Existing multi-target pedestrian tracking suffers from the drawbacks of large model computational complexity and slow tracking speed. Furthermore, tracking accuracy still needs to be improved, especially for small-scale and occluded pedestrians. The feature enhancement module proposed in this paper can effectively enhance the model's feature extraction capabilities, thereby improving the tracking effect for small-scale and occluded pedestrians. The introduction of the Mish activation function further enhances the model's tracking accuracy. By applying these improvements to the JDE method, the proposed Attention-JDE can achieve near-real-time multi-target pedestrian tracking.

[0103] The Attention-JDE network uses the modified DarkNet-53 as the backbone network, with a total of 5 downsampling operations. The feature maps obtained from the last three downsampling operations are used to construct a feature pyramid. The feature enhancement module proposed in this paper is respectively connected to the feature maps of the three scales of the feature pyramid. The feature enhancement module can further obtain multi-scale features, enhance the tracking ability of small-scale targets, and enable the network to pay more attention to important features and enhance the detection ability of overlapping targets. Finally, the feature maps at three scales are used for prediction and the final result is output. The overall network structure is as follows Figure 2 shown.

[0104] Backbone network: The backbone network of the present invention is modified and designed based on DarkNet-53. Based on DarkNet-19, the network adds a residual module and uses a Bottleneck structure composed of a large number of 3×3 and 1×1 convolutions, which expands the network to 53 layers. As shown in Table 1, DarkNet-53 contains a total of 5 downsampling convolution operations. Each downsampling convolution operation reduces the size of the feature map to half, while doubling the number of channels. The feature map at the end of the network is 32 times smaller than the initial input. Therefore, the size of the initial input image should be an integer multiple of 32. Based on the resolution of the video dataset, this paper modifies the input size of the original network to 1088×608, and the feature map size of the output at the end of the network is 34×19. The feature maps of the last three scales are used to construct a feature pyramid, and the feature pyramid is used to complete multi-scale feature fusion, which is conducive to extracting features of small-scale targets. In addition, this paper replaces the Leaky Relu activation function used in the original DarkNet-53 with the Mish activation function. The Mish activation function is a smooth, continuous, non-monotonic function, and the Mish activation function has no upper bound but a lower bound. Using the Mish activation function can make the gradient conduction of the model smoother, retain more effective information, and enhance the generalization ability of the model. Using the Mish activation function in Attention-JDE can improve the model's ability to detect targets, which is more conducive to the detection and tracking of overlapping targets and effectively reduces the occurrence of ID switching. The mathematical form of the Mish activation function is shown in formula (1), and the function image is shown in Figure 3 As shown,

[0105] f(x)=xtanh(ln(1+e x ))#(1)

[0106] Where ln represents the natural logarithm, and

[0107]

[0108] Feature enhancement module: This paper adds a feature enhancement module at the end of the backbone network, such as Figure 4 As shown in Figure 2, the feature enhancement module consists of a channel attention module, a spatial attention module, and a spatial pyramid pooling module. Using this feature enhancement module can effectively refine key features, expand the model's receptive field, and improve feature extraction efficiency.

[0109] (a) Channel Attention Module

[0110] The input of the channel attention module is the output of the previous convolution, expressed as like Figure 5 As shown, the maximum pooling and average pooling are used to obtain the global channel feature distribution information, which are expressed as Next, the two feature vectors are passed through a multi-layer perceptron (MLP) with one hidden layer, and the resulting feature vectors are added together and subjected to a sigmoid operation (σ) to obtain the final channel attention weight M. C (F), through M C (F) is multiplied with the original input features to assign high weights to the channels containing key information and low weights to other channels, thereby achieving channel attention. The mathematical formula of channel attention is shown in formula (3):

[0111]

[0112] (b) Spatial Attention Module

[0113] After completing the channel attention weight distribution, continue to use the spatial attention module to distribute the weights of spatial features, such as Figure 6 As shown, first perform maximum pooling and average pooling on the input feature map in the channel dimension to obtain Next, the two feature maps are concatenated in the channel dimension and a 7×7 convolution operation is performed without changing the size of the feature map. The sigmoid operation (σ) is then performed on the feature map in the previous step to obtain the final spatial attention weight. Finally, the spatial attention weight is assigned to each pixel of the original input feature by point-by-point multiplication. The spatial attention mechanism is expressed as shown in formula (4):

[0114]

[0115] where f 7×7 Represents a convolution operation with a kernel size of 7.

[0116] (c) Spatial pyramid pooling module

[0117] The spatial pyramid pooling module (SPP) is a type of multi-scale fusion. Using SPP can expand the model's receptive field, allowing features to contain more contextual information, thereby increasing the model's target detection accuracy and improving target tracking performance. The SPP module contains three maximum pooling operations, which convert the input features into The maximum value pooling of 5×5, 9×9, and 13×13 is performed respectively. The size of the feature map is maintained by filling 0 around the feature map. Then, the feature maps of the three poolings are spliced ​​in the channel dimension to complete the feature fusion. The calculation process of SPP is shown in formula (5):

[0118]

[0119] Maxpool n×n () represents the maximum pooling operation with a kernel size of n, Represents a concatenate operation.

[0120]

[0121] Table 1 DarkNet-53 network structure

[0122] Result prediction: Figure 7 As shown in the figure, the output layer of Attention-JDE can be divided into two branches: the target detection branch and the embedding branch. The target detection branch completes the classification and positioning of the target, and the embedding branch completes the extraction of the target's appearance features, which are then used for subsequent trajectory association.

[0123] (a) Object detection branch

[0124] In the same way as YOLOv3, this paper treats the classification and localization of the target as a regression task. Regression is performed using predefined anchor boxes on the feature maps of three scales at the end of the network. Anchor boxes of four sizes are assigned to each scale. The predefined sizes of the anchor boxes are obtained by using k-means clustering on the benchmark dataset, and a total of 12 categories are clustered. Formula (6) represents the k-means cost function.

[0125]

[0126] Given a sample {x (1) ,...,x (m)}, x (i) In this paper, it is a two-dimensional vector, namely width and height. First, k cluster centers {μ1,...,μ k}, for each sample point, traverse all k cluster centers and calculate the (i) The center μ with the smallest Euclidean distance j , x (i) The category is set to μ j The corresponding cluster c i , that is, c i :=argmin||x (i) -μ j || 2 #(7)

[0127] Then, after updating the categories of all samples, calculate each cluster c i The mean of all samples in the cluster is taken as the new cluster center, and the next iteration is performed until the cluster center no longer changes or the maximum number of iterations is reached. The final cluster center is set as the anchor box of this article.

[0128] According to the size of the input image 1088×608, the three scale feature maps corresponding to the end of the network are divided into 136×76, 68×38, and 34×19 grid cells on the image. When the coordinates of the target in the ground truth fall into a grid cell, then this grid cell is responsible for predicting the target. Using logistic regression, we get n a ×(1+4+len(classes)) dimensional vector is output as the detection result, where n a The number of anchor boxes assigned to each grid cell is set to 4 here, 1 corresponds to the target confidence, 4 corresponds to the center coordinates and width and height of the bounding box, and len(classes) corresponds to the number of target types. In addition, the coordinates of the bounding box output by the network are actually the offset relative to the grid cell, so they are converted to the position in the actual image according to formula (8), (t x ,t y ,t w ,t h ) are the center coordinates and width and height of the model output, σ represents the sigmoid function, c x ,c y Represents the row and column number of the current grid cell, p w ,p h Represents the width and height of the Anchor box respectively, (b x ,b y ,b w ,b h ) represents the actual center coordinates and width and height of the bounding box.

[0129] b x =σ(t x )+c x

[0130] b y =σ(t y )+c y

[0131]

[0132]

[0133] (b) Embedding branch

[0134] The embedding branch generates a feature that can be used to distinguish different targets within the same class. Therefore, this paper adds a convolution operation after the feature pyramid of the network to extract a 512-dimensional feature map as the surface feature required for tracking. The size is the same as the corresponding feature scale: 136×76, 68×38, and 34×19. For a single target, the surface feature is a single 512-dimensional feature vector in the feature map. Subsequently, the extracted surface features are used to complete track association through the Kalman filter and Hungarian algorithm, which are commonly used in multi-target tracking. Specifically, some tracks are initialized based on the detection results of the first frame of the video. For targets detected in subsequent frames, the cost matrix is ​​calculated based on the distance and intersection-over-union ratio between their surface features and existing tracks. The Hungarian algorithm is used to complete the matching. In addition, the Kalman filter is used to predict the target position in the current frame. When the distance between the target position and its assigned track exceeds a set threshold, the target is abandoned. If a track is not assigned a new target, it is marked as lost. When the loss time of a track exceeds a given threshold, the track is abandoned.

[0135] Example 3

[0136] This paper first explores the impact of different types of attention mechanisms on tracking performance, including spatial attention mechanism (SAM), channel attention mechanism (CAM), and hybrid attention mechanism (CBAM). Table 2 shows the impact of these attention mechanisms on Attention-JDE, with data verified on the MOT16 training set. It can be seen that SAM improves the model more than CAM. The reason for this is that the target categories in the MOT16 dataset only include pedestrians, so the classification capabilities of the network model are relatively low, while the localization capabilities are relatively high. Therefore, the spatial attention mechanism, which allows the network to pay more attention to the "location" of the target, can bring greater improvement to the model. In addition, this paper explores the impact of two arrangements in the hybrid domain on the results. It can be found that adding the channel domain after the spatial domain is not as effective as adding the spatial domain alone.

[0137]

[0138] Table 2 Comparison results between different attention mechanisms

[0139] Table 3 compares Attention-JDE with other mainstream methods on the MOT16 test set. Attention-JDE achieves a mean of 62.1% and an FPS of 19.5. It is worth noting that the other methods in Table 3 are two-step methods, so the FPS data in the table only relates to the association step. In actual applications, the detection step consumes more time. Attention-JDE is a one-shot method, and its inference speed is related to all steps in the entire system, from detection to association.

[0140]

[0141] Table 3 Comparison of results of different methods on the MOT16 test set

[0142] We also conducted corresponding ablation experiments. Table 4 shows the impact of adding various modules and modifying the backbone network activation function on the MOTA on the MOT16 training set based on the JDE method. The results show that adding CBAM improves MOTA by 2.2, and adding the SPP module improves MOTA by 1.8, but also leads to more IDSw (ID switching). Using the Mish activation function can improve MOTA by 0.7 and reduce IDSw by approximately 10%. The results show that adding the feature enhancement module can effectively improve model performance, and using the Mish activation function can effectively improve the ID switching problem during tracking. Ultimately, the MOTA is improved by 2.6, while only reducing 1.8 FPS, while also reducing IDSw.

[0143]

[0144] Table 4 Ablation experiments of each module

[0145] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A pedestrian multi-target tracking method based on the Attention-JDE network, comprising inputting an image with adjusted resolution into the DarkNet-53 backbone network to extract feature maps of three different scales, characterized in that: The following steps are also included: Step 1: Take the feature maps of three scales as input and construct a feature pyramid; Step 2: The output of the feature pyramid is sent to the feature enhancement module to complete feature refinement and also output three new feature maps of new scales. The feature enhancement module consists of a channel attention module, a spatial attention module and a spatial pyramid pooling module. The working method of the spatial attention module is Step 211: Perform maximum pooling and average pooling on the input feature map in the channel dimension to obtain and in, Indicates the output dimension with 1 channel and width W and height H; Step 212: Combine the two feature maps and Splicing is performed in the channel dimension and a 7×7 convolution operation is performed; Step 213: Perform a sigmoid operation (σ) on the feature map of the previous step to obtain the final spatial attention weight; Step 214: Assign a weight to each pixel of the original input feature by point-by-point multiplication; the formula of the spatial attention mechanism is Among them, f 7×7 It is a convolution operation with a convolution kernel size of 7; Step 3: Use the new feature map to detect pedestrians, generate a detection frame, and use the detection frame and the corresponding embedded features to match the same target in the previous and next frames to form a trajectory.

2. The method for multi-target pedestrian tracking based on the Attention-JDE network according to claim 1, characterized in that: The DarkNet-53 backbone network uses the Mish activation function, and the Mish activation function image is f(x)=xtanh(ln(1+e x )) Among them, x is the input feature.

3. The method for multi-target pedestrian tracking based on the Attention-JDE network according to claim 2, characterized in that: The input of the channel attention module is the output of the previous convolution, expressed as in, Represents the output dimension with C channels, W width and H height.

4. The method for multi-target pedestrian tracking based on the Attention-JDE network according to claim 3 is characterized in that: The working method of the channel attention module is: Step 201: Use maximum pooling and average pooling to obtain global channel feature distribution information, which are expressed as and in, Indicates the output dimension with C channels and 1 width and height. is the channel feature distribution information obtained by maximum pooling, is the channel feature distribution information obtained by average pooling; Step 202: Channel feature distribution information and After a multi-layer perceptron MLP with one hidden layer; Step 203: Add the obtained feature vectors and perform a sigmoid operation σ to obtain the final channel attention weight M C (F); Step 204: Through M C (F) is multiplied with the original input features to assign high weights to channels containing key information and low weights to other channels to achieve channel attention.

5. The method for multi-target pedestrian tracking based on the Attention-JDE network according to claim 4, characterized in that: The spatial pyramid pooling module contains three maximum pooling operations, which The maximum value pooling of 5×5, 9×9 and 13×13 is performed respectively, and the size of the feature map is maintained by padding 0 around the feature map. Then the feature maps of the three poolings are spliced ​​in the channel dimension to complete the feature fusion.

6. The method for multi-target pedestrian tracking based on the Attention-JDE network according to claim 5, characterized in that: The calculation process of spatial pyramid pooling is Among them, Maxpool n×n () represents the maximum pooling operation with a kernel size of n, Indicates the concatenate operation, F in is the input feature, F out is the output feature.

7. The method for multi-target pedestrian tracking based on the Attention-JDE network according to claim 6, characterized in that: The step 3 includes using the target detection branch to complete the classification and positioning of the target, using the embedding branch to complete the extraction of the target's appearance features and using the appearance features for subsequent trajectory association.

Images