A multi-target detection and tracking algorithm in complex urban road environments

Abstract

The present invention discloses a method for multi-target detection and tracking in a complex urban road environment: Step 1: Construct a training set and a test set; Step 2: Add feature fusion modules layer by layer on the basis of the existing DLA34 backbone network to achieve deep and shallow network feature fusion of the input image; Step 3: Use the Transformer encoding module to extract long-range feature dependencies in the feature map; Step 4: Perform further feature fusion and logistic regression processing; Step 5: Use the multi-target tracking module to perform target association processing and tracking to obtain a tracking feature map with a target detection frame; Step 6: Obtain a trained multi-target detection and tracking model; Step 7: Input the video data to be detected into the trained multi-target detection and tracking model to obtain a tracking feature map with a target detection frame. The present invention can accurately detect and track multiple targets in a complex urban road environment and can stably identify targets with large changes in appearance scale.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving and relates to a method for detecting and tracking traffic targets. Background Art

[0002] Smart transportation has become a key direction for future transportation development. Autonomous driving, a prominent example, is a multidisciplinary and multi-faceted technology. The development of autonomous driving technology requires not only autonomous driving capabilities for participating vehicles but also precise perception of complex traffic environments, high-precision mapping, vehicle navigation and positioning, and vehicle dynamics control technologies to build a comprehensive vehicle-infrastructure collaborative transportation system. In recent years, 5G networks combined with cloud computing have enabled traditional infrastructure to gain road perception capabilities through advanced artificial intelligence (AI), further enhancing the environmental awareness of individual vehicles through the Internet of Things (IoT) and cloud computing. Whether for individual vehicle intelligence or vehicle-infrastructure collaboration, sensors are required to collect environmental information. Commonly used sensors include lidar, millimeter-wave radar, and cameras. Compared to other sensors, cameras have become the preferred visual sensor for environmental perception due to their unique cost-effectiveness. Camera-based AI technology has become an essential and key technology for the development of smart transportation. Therefore, multi-target detection and tracking are crucial for the perception of complex traffic environments.

[0003] First, traffic scenes are often captured by cameras mounted at high altitudes. This leads to issues such as smaller objects in the distance, less feature information, and a high number of objects in the same scene, with significantly varying sizes. The convolutional neural networks commonly used in current research downsample the image during the forward propagation process, causing the model to miss smaller objects and making it more difficult to capture them. Second, with the development of deep learning, significant research has been achieved in multi-object tracking. However, due to factors such as target size variations, occlusions, and blurring caused by rapid movement during tracking, existing tracking algorithms fall short of optimal performance. To address the problem of multi-object detection and tracking in traffic scenes, the industry currently uses target detection algorithms and two-stage tracking networks based on Kalman filters and Hungarian algorithms. These models have several challenges: the target detection and tracking modules are independent and cannot be trained simultaneously. Furthermore, target detection accuracy determines target tracking performance, leading to bottlenecks in network training and optimization. Furthermore, stable tracking of objects with large inter-frame displacement is difficult. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-target detection and tracking algorithm in a complex urban road environment to solve the problems in the prior art of low target detection accuracy and inability to stably track targets with large inter-frame displacement.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A multi-target detection and tracking method in a complex urban road environment specifically includes the following steps:

[0007] Step 1: Select a public dataset for data augmentation, obtain the dataset, and construct the training set and test set;

[0008] Step 2: Based on the existing DLA34 backbone network, feature fusion modules are added layer by layer to achieve deep and shallow network feature fusion of the input image, and a two-dimensional feature map is obtained after the three features are fused;

[0009] Step 3: Based on the two-dimensional feature map after feature fusion, the Transformer encoding module is used to extract the long-range feature dependencies in the feature map to obtain the feature map after dependency extraction;

[0010] Step 4: Generate heatmap and target bounding box through further feature fusion and logistic regression processing;

[0011] Step 5: Use the multi-target tracking module to perform target association processing and tracking to obtain a tracking feature map with target detection boxes;

[0012] Step 6: Use the training set of step 1 to train the multi-target detection and tracking model composed of steps 2, 3, 4, and 5, and use the test set to test it, and finally obtain the trained multi-target detection and tracking model;

[0013] Step 7: Input the video data to be detected into the trained multi-target detection and tracking model to obtain a tracking feature map with a target detection frame.

[0014] Furthermore, in step 1, VisDrone_mot in the mainstream traffic target detection dataset VisDrone is selected as the dataset of the present invention.

[0015] Furthermore, the step 2 specifically includes the following sub-steps:

[0016] Step 21: Input the image in the training set into the DLA34 network, and perform two convolution operations on the original image with a convolution kernel of 3×3 size through the BatchNorm layer and the ReLU layer to obtain two feature maps. The two feature maps after convolution are input into the aggregation node for feature fusion to obtain a feature map with a resolution of the original input. Figure 1 / 4 size feature map;

[0017] Step 22: Downsample the 1 / 4 size feature map obtained in step 21 by 2 times to obtain a new feature map. Repeat the convolution operation and aggregation operation in step 21 twice to obtain two feature maps, and perform the aggregation operation again with the aggregation node obtained in step 21 as the common input to obtain a feature map with a resolution of the original input. Figure 1 / 8 size feature map;

[0018] Step 23: Get the original input features from the 1 / 4 size feature map in step 22 Figure 1 In the same way as the feature map of size 1 / 8, the feature map of size 1 / 16 is obtained from the feature map of size 1 / 8, and then the feature map of size 1 / 16 is obtained from the feature map of size 1 / 32;

[0019] Step 24, such as Figure 2 As shown, the obtained feature maps of 1 / 4 size, 1 / 8 size, 1 / 16 size, and 1 / 32 size are sequentially fused using a feature fusion module to perform feature fusion of adjacent feature maps to obtain new feature maps of 1 / 4 size, 1 / 8 size, and 1 / 16 size, respectively.

[0020] Furthermore, in step 24, the feature fusion module is used to implement the following operations:

[0021] Step 241: Perform deformable convolution processing with a convolution kernel of 3×3 on the feature map F1, and pass the processed result through a BatchNorm layer and a ReLU layer to obtain a mapped feature map;

[0022] Step 242: Use direct interpolation upsampling plus convolution processing to replace the transposed convolution in the DLA34 backbone network, and perform a 2-fold upsampling on the feature map obtained in step 241 to obtain a feature map F1';

[0023] Step 243, adding the corresponding channel values ​​of the feature map F1' and the feature map F2 obtained in step 242 to obtain a merged feature map;

[0024] Step 244: The merged feature map obtained in step 243 is processed by a 3×3 deformable convolution, and then passes through a BatchNorm layer and a ReLU layer in sequence to obtain a two-dimensional feature map F2';

[0025] When the feature map F1 and the feature map F2 are respectively a feature map of 1 / 4 size and a feature map of 1 / 8 size, the obtained two-dimensional feature map F2' is a feature map of 1 / 4 size;

[0026] When the feature map F1 and the feature map F2 are respectively a feature map of 1 / 8 size and a feature map of 1 / 16 size, the obtained two-dimensional feature map F2' is a feature map of 1 / 8 size;

[0027] When the feature map F1 and the feature map F2 are respectively a feature map of 1 / 16 size and a feature map of 1 / 32 size, the obtained two-dimensional feature map F2' is a feature map of 1 / 16 size.

[0028] Furthermore, the step 3 specifically includes the following sub-steps:

[0029] Step 31: collapse the 1 / 16 size two-dimensional feature map obtained in step 2 into a one-dimensional sequence and convolute it to form K, V, and Q feature maps;

[0030] Step 32: Add the position code to the feature map K and feature map Q obtained in step 31 pixel by pixel to obtain two feature maps with position information. These two feature maps and feature map V are input into the multi-head attention module as a common input, and a new feature map is obtained after processing.

[0031] Step 33: perform a fusion operation and a LayerNorm operation on the new feature map obtained in step 32 and the V, K, and Q feature maps obtained in step 31 by adding corresponding values ​​between the feature maps.

[0032] Step 34: The result obtained in step 33 is processed in a feedforward neural network and output through a residual connection to obtain a new feature map.

[0033] Furthermore, the position code in step 32 is obtained by the following formula:

[0034] PE (pos,2i) =sin(pos / 10000 2i / d )

[0035] pE (pos,2i+1) =cos(pos / 10000 2i / d )

[0036] Among them, PE (·) It is a position-encoded matrix with the same resolution as the input feature map. Pos represents the position of the vector in the sequence, while i is the index of the channel and d represents the number of channels of the input feature map.

[0037] Furthermore, the step 4 specifically includes the following sub-steps:

[0038] In step 41, the feature map finally obtained in step 3 is upsampled by a factor of 2 to obtain a new feature map.

[0039] Step 42: The feature maps of 1 / 4 size and 1 / 8 size obtained in step 24 are fused using the same feature fusion module as in step 24 to obtain a new feature map of 1 / 4 size.

[0040] Step 43: The feature maps of 1 / 8 size and 1 / 16 size obtained in step 24 are fused using a feature fusion module, and are added pixel by pixel to the feature map obtained in step 41 to obtain a new feature map of 1 / 8 size.

[0041] Step 44: The feature map of 1 / 4 size obtained in step 42 and the feature map of 1 / 8 size obtained in step 43 are fused using the feature fusion module to generate a feature map with the original resolution. Figure 1 / 4 size heatmap;

[0042] Step 45: Perform logistic regression on the heat map obtained in step 44 and the heat map label containing the target center point in the data set obtained in step 1 to obtain the center point of the predicted target.

[0043] In step 46, the coordinates of the upper left and lower right points of the bounding box corresponding to each target are obtained by formula (3) to generate the target bounding box:

[0044]

[0045] in, That is, step 45 obtains the center point of the predicted target, Indicates the offset between the center point and the target center point, Indicates the size of the bounding box corresponding to the target.

[0046] Furthermore, the step 5 specifically includes the following sub-steps:

[0047] Step 51: input the same image in step 2 as the T-1 frame image, and select the next frame image, that is, the T frame image. The T frame and the T-1 frame images are used as input and processed by the CenterTrack backbone network to generate feature maps f T and f T-1 ;

[0048] Step 52: transform the feature map f T and f T-1 Send separately as Figure 5 The cost space module shown in the figure performs target association processing to obtain the output feature map f′ T ;

[0049] Step 53: Compare the heat map obtained in step 4 with the feature map f obtained in step 51 T-1 Perform Hadamard product to generate feature map Will and the feature map f′ obtained in step 52 T Perform deformable convolution together to generate feature maps

[0050] Step 54: Use three 1×1 convolution operations and downsampling operations in sequence to generate the T-1 frame feature map; the feature map f obtained in step 51 is T Use three 1×1 convolution operations to generate the T-th frame feature map;

[0051] Step 55: Input the T-th frame feature map obtained in step 54 and the T-1-th frame feature map into the attention propagation module for feature propagation to obtain the tracking feature map V′ with the target detection frame. T .

[0052] Furthermore, the step 52 specifically includes the following operations:

[0053] Step 521: transform the feature map f T and f T-1 The three-layer weight-sharing convolution structure in the cost space module is fed into the feature map e T and e T-1 , that is, the appearance encoding vector of the target;

[0054] Step 522, the feature map e T and e T-1 Perform the maximum pooling operation to obtain e′ T and e′ T-1 , to reduce the complexity of the model, use e′ T and e′ T-1 The cost space matrix C is obtained by transposing the product. The position of the target in the current frame on the cost space matrix C is (i, j). The two-dimensional cost matrix C containing the position information of the target in the current frame in the previous frame image is extracted from the cost space matrix C. i,j , for C i,j The horizontal and vertical directions are respectively maximized to obtain the feature map of the corresponding direction

[0055] Step 523, define two offset templates using equations (4) and (5)

[0056] G i,j,l =(lj)×s1≤l≤W C (4)

[0057] M i,j,k =(ki)×s1≤k≤H C (5)

[0058] Among them, s is the downsampling multiple of the feature map relative to the original image, W C 、H C is the width and height size of the feature map, G i,j,lis the offset of the target (i, j) in the T frame image at the horizontal position l in the T-1 frame image, M i,j,k is the offset of the target (i, j) in frame T at the vertical position k in the image of frame T-1;

[0059] Step 524, the step 522 obtained After multiplying the offset template G and M defined in step 523, the channel is superimposed to obtain the feature map O T , represents the target's offset template in both horizontal and vertical directions; then O T Perform 2x upsampling to restore to H F ×W F Size, at the same time, will O T The horizontal and vertical channels of the feature map are respectively compared with the f obtained in step 51. T 、f T-1 The channels are superimposed, and then convolution is performed to form two feature maps with the same size of feature maps in the horizontal and vertical directions and 9 channels. These two feature maps are superimposed on the channels to obtain the output feature map f′ T .

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] ① In the present invention, the input image resolution of the adopted dataset is appropriately increased to ensure the size of the final feature map to retain more detailed information;

[0062] ② In the multi-target detection module, the present invention fuses the deep feature map containing more semantic information with the shallow feature map containing more detailed information through a feature fusion module, thereby improving the model's ability to detect small targets;

[0063] ③ In the multi-target detection module, the present invention introduces the self-attention mechanism of the Transformer encoding module to capture long-distance dependencies and explore the potential connections between features in the feature map, which can stably identify targets with large appearance scale changes;

[0064] ④ A multi-target tracking algorithm based on cost space and inter-frame information fusion is proposed. The cost space matrix is ​​used to predict the position of the target in the current frame in the previous frame. The targets between the two frames can be associated to achieve the tracking effect.

[0065] ⑤ In the multi-target tracking module, the attention propagation module is introduced to fuse the features of targets in multiple frames to compensate for the target spatial misalignment problem caused by target motion between frames, so that the model can still accurately track the target even when it is occluded. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a schematic diagram of a multi-target detection module of the present invention;

[0067] Figure 2 It is a schematic diagram of the feature fusion module in the multi-target detection module;

[0068] Figure 3 This is a schematic diagram of the Transformer encoding module in the multi-target detection module;

[0069] Figure 4 is a schematic diagram of a multi-target tracking module of the present invention;

[0070] Figure 5 This is a schematic diagram of the cost space module in the multi-target tracking module;

[0071] Figure 6 Schematic diagrams of the experimental results of the multi-target detection module of the present invention; they are schematic diagrams of the target center point and target bounding box results obtained by the module for detecting small targets and large targets respectively.

[0072] Figure 7 The following is a schematic diagram of the experimental results of the multi-target tracking module of the present invention. The four images are respectively four images of two test cases, and the four images of each test case are the 0th frame, the 5th frame, the 10th frame and the 15th frame respectively. DETAILED DESCRIPTION

[0073] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0074] The multi-target detection and tracking model of the present invention is divided into two parts. Figure 1 The figure shows the framework of the multi-target detection module, which is mainly based on the improved DLA34 as the backbone network. By adding a feature fusion module, a deep and shallow network fusion feature map is obtained. The Transformer encoding module is introduced to perform self-attention encoding on the fused feature map to solve the problem that the network's ability to extract the semantics of large targets is limited due to the large difference in target feature scales. Finally, a target heat map is generated and regressed to obtain the bounding box of the corresponding target to achieve traffic target detection. Figure 4 Figure 2 shows the target tracking module framework. The CenterTrack backbone network generates feature maps and uses the cost space matrix to achieve target association and tracking between two frames. By using the attention propagation module, the target information of the previous and next frames is fused and complemented, achieving accurate tracking of the target even when it is blurred or occluded.

[0075] The multi-target detection and tracking method in a complex urban road environment of the present invention specifically comprises the following steps:

[0076] Step 1: Select a public dataset for data enhancement, obtain the dataset, and construct the training set and test set.

[0077] Specifically, VisDrone_mot in the mainstream traffic target detection dataset VisDrone is selected as the dataset of the present invention. The VisDrone_mot dataset collects aerial bird's-eye views of the streets of multiple cities in China by drones, and provides 96 video sequences, including 56 training video sequences containing 24,201 frames of images, 7 verification video sequences containing 2,819 frames of images, and 33 test sequences containing 12,968 frames of images. The bounding boxes of the identified objects are manually marked in each video frame. The resolution of the input images in the VisDrone_mot dataset is increased to 1024×1024 to ensure that the size of the final feature map output by the multi-target detection module is 256×256 and retain more detailed information. At the same time, a data enhancement method combining random flipping, random scaling with a resolution size between 0.6 and 1.3 times, random cropping and color jittering is used as an extended training sample.

[0078] Step 2: Based on the existing DLA34 backbone network, feature fusion modules are added layer by layer to achieve deep and shallow network feature fusion of the input image, and a two-dimensional feature map is obtained after the three features are fused. Figure 1 As shown, it specifically includes the following sub-steps:

[0079] Step 21: Input the image in the training set into the DLA34 network, and perform two convolution operations on the original image with a convolution kernel of 3×3 size through the BatchNorm layer and the ReLU layer to obtain two feature maps. The two feature maps after convolution are input into the aggregation node for feature fusion to obtain a feature map with a resolution of the original input. Figure 1 / 4 size feature map. The feature fusion of the aggregation node is as follows:

[0080] N(X1,...,X n )=σ(BN(∑w i x i +b),...,BN(∑w i x i +b)) (1)

[0081] Among them, N(·) represents the aggregation node, σ(·) represents the feature aggregation, and w i x i +b indicates convolution operation, BN indicates BatchNorm operation, X i＝1...N Corresponding to the output of the convolution module.

[0082] Step 22: Downsample the 1 / 4 size feature map obtained in step 21 by 2 times to obtain a new feature map. Repeat the convolution operation and aggregation operation in step 21 twice to obtain two feature maps, and perform the aggregation operation again with the aggregation node obtained in step 21 as the common input to obtain a feature map with a resolution of the original input. Figure 1 The purpose of this step is to transfer the feature information of the shallow layer of the network to the deep layer of the network.

[0083] Step 23: Get the original input features from the 1 / 4 size feature map in step 22 Figure 1 In the same way as the feature map of size 1 / 8, the feature map of size 1 / 16 is obtained from the feature map of size 1 / 8, and then the feature map of size 1 / 16 is obtained from the feature map of size 1 / 32;

[0084] Step 24, such as Figure 2 As shown in FIG, the obtained feature maps of 1 / 4 size, 1 / 8 size, 1 / 16 size, and 1 / 32 size are sequentially fused using a feature fusion module to perform feature fusion on adjacent feature maps, and new feature maps of 1 / 4 size, 1 / 8 size, and 1 / 16 size are obtained respectively;

[0085] The feature fusion module is used to implement the following operations:

[0086] Step 241: Perform deformable convolution processing with a convolution kernel of 3×3 on the feature map F1, and pass the processed result through a BatchNorm layer and a ReLU layer to obtain a mapped feature map;

[0087] In step 242, direct interpolation upsampling plus convolution processing is used to replace the transposed convolution in the DLA34 backbone network, and the feature map obtained in step 241 is upsampled by a factor of 2 to obtain a feature map F1′, so as to obtain more target position information and reduce the number of model parameters;

[0088] Step 243, adding the corresponding channel values ​​of the feature map F1′ and the feature map F2 obtained in step 242 to obtain a merged feature map;

[0089] Step 244: The merged feature map obtained in step 243 is processed by a 3×3 deformable convolution, and then passes through a BatchNorm layer and a ReLU layer in sequence to obtain a two-dimensional feature map F2′;

[0090] When the feature map F1 and the feature map F2 are 1 / 4 size and 1 / 8 size respectively, the obtained two-dimensional feature map F2′ is a 1 / 4 size feature map;

[0091] When the feature map F1 and the feature map F2 are respectively a feature map of 1 / 8 size and a feature map of 1 / 16 size, the obtained two-dimensional feature map F2′ is a feature map of 1 / 8 size;

[0092] When the feature map F1 and the feature map F2 are respectively a feature map of 1 / 16 size and a feature map of 1 / 32 size, the obtained two-dimensional feature map F2′ is a feature map of 1 / 16 size.

[0093] Step 3: Based on the feature map obtained after feature fusion in step 2, the Transformer encoding module is used to extract the long-range feature dependencies in the feature map to obtain the feature map after dependency extraction. Figure 3 As shown, it specifically includes the following sub-steps:

[0094] Step 31: collapse the 1 / 16 size two-dimensional feature map obtained in step 2 into a one-dimensional sequence and convolute it to form three K (Key), V (Value), and Q (Query) feature maps;

[0095] In step 32, the position code is added pixel by pixel to the feature map K and feature map Q obtained in step 31 to obtain two feature maps with position information. These two feature maps and feature map V are input into the multi-head attention module and processed to obtain a new feature map to capture the long-range dependency in the image. The position code is obtained by formula (1) (2):

[0096] PE (pos，2i) =sin(pos / 10000 2i / d ) (1)

[0097] PE (pos，2i+1) =cos(pos / 10000 2i / d ) (2)

[0098] Among them, PE (·) It is a position-encoded matrix with the same resolution as the input feature map. Pos represents the position of the vector in the sequence, while i is the index of the channel and d represents the number of channels of the input feature map.

[0099] In step 33, the new feature map obtained in step 32 is fused with the V, K, and Q feature maps obtained in step 31 by adding corresponding values ​​between the feature maps and performing a LayerNorm (LN) operation to avoid information loss.

[0100] Step 34: The result obtained in step 33 is processed in a feedforward neural network and output through a residual connection to obtain a new feature map.

[0101] Step 4: Based on the feature maps obtained in steps 2 and 3, generate a heat map and target bounding box through further feature fusion and logistic regression processing. The specific steps include the following:

[0102] In step 41, the feature map finally obtained in step 3 is upsampled by a factor of 2 to obtain a new feature map.

[0103] Step 42: The feature maps of 1 / 4 size and 1 / 8 size obtained in step 24 are fused using the same feature fusion module as in step 24 to obtain a new feature map of 1 / 4 size.

[0104] Step 43: The feature maps of 1 / 8 size and 1 / 16 size obtained in step 24 are fused using a feature fusion module, and are added pixel by pixel to the feature map obtained in step 41 to obtain a new feature map of 1 / 8 size.

[0105] Step 44: The feature map of 1 / 4 size obtained in step 42 and the feature map of 1 / 8 size obtained in step 43 are fused using the feature fusion module to generate a feature map with the original resolution. Figure 1 / 4 size heatmap;

[0106] Step 45: Perform logistic regression on the heat map obtained in step 44 and the heat map label containing the target center point in the data set obtained in step 1 to obtain the center point of the predicted target.

[0107] In step 46, the coordinates of the upper left and lower right points of the bounding box corresponding to each target are obtained by formula (3) to generate the target bounding box:

[0108]

[0109] in, That is, step 45 obtains the center point of the predicted target, Indicates the offset between the center point and the target center point, Indicates the size of the bounding box corresponding to the target.

[0110] Step 5: Based on the input image of step 2 and the heat map obtained in step 4, use the multi-target tracking module to perform target association processing and tracking, and obtain a tracking feature map with a target detection frame. Figure 4 As shown, it specifically includes the following sub-steps:

[0111] Step 51: input the same image in step 2 as the T-1 frame image, and select the next frame image, that is, the T frame image. The T frame and the T-1 frame images are used as input and processed by the CenterTrack backbone network to generate feature maps f T and f T-1 ;

[0112] Step 52: transform the feature map f T and f T-1 Send separately as Figure 5 The cost space module shown in the figure performs target association processing to obtain the output feature map f′ T The specific operations include the following:

[0113] Step 521: transform the feature map f T and f T-1 The three-layer weight-sharing convolution structure in the cost space module is fed into the feature map e T and e T-1 , that is, the appearance encoding vector of the target;

[0114] Step 522, the feature map e T and e T-1 Perform the maximum pooling operation to obtain e′ T and e′ T-1 , to reduce the complexity of the model, use e′ T and e′ T-1 The cost space matrix C is obtained by transposing the product to save the similarity of the corresponding points between the feature maps of the two frames. The position of the target in the current frame on the cost space matrix C is (i, j). The two-dimensional cost matrix C containing the position information of the target in the current frame in the previous frame image is extracted from the cost space matrix C. i，j , for C i，j The horizontal and vertical directions are respectively maximized to obtain the feature map of the corresponding direction

[0115] Step 523, define two offset templates using equations (4) and (5)

[0116] G i，j，l =(lj)×s1≤l≤W C (4)

[0117] M i，j，k =(ki)×s1≤k≤H C (5)

[0118] Among them, s is the downsampling multiple of the feature map relative to the original image, W C 、H C is the width and height size of the feature map, G i，j，l is the offset of the target (i, j) in the T frame image at the horizontal position l in the T-1 frame image, M i，j，k is the offset of the T-frame object (i, j) appearing at the vertical position k in the T-1 frame image.

[0119] Step 524, the step 522 obtained After multiplying the offset template G and M defined in step 523, the channel is superimposed to obtain the feature map O T , represents the target's offset template in both horizontal and vertical directions; then O T Perform 2x upsampling to restore to H F ×W F Size, at the same time, will O T The horizontal and vertical channels of the feature map are respectively compared with the f obtained in step 51 T 、f T-1 The channels are superimposed, and then convolution is performed to form two feature maps with the same size of feature maps in the horizontal and vertical directions and 9 channels. These two feature maps are superimposed on the channels to obtain the output feature map f′ T .

[0120] Step 53: Compare the heat map obtained in step 4 with the feature map f obtained in step 51 T-1 Perform Hadamard product to generate feature map Will and the feature map f′ obtained in step 52 T Perform deformable convolution together to generate feature maps

[0121] Step 54: Use three 1×1 convolution operations and downsampling operations in sequence to generate the T-1 frame feature map (q t-1 、 k t-1 、v t-1 );The feature map f obtained in step 51 T Use three 1×1 convolution operations to generate the T-th frame feature map (Q t , K t 、V t );

[0122] Step 55: Input the T-th frame feature map obtained in step 54 and the T-1-th frame feature map into the attention propagation module for feature propagation to obtain the tracking feature map V′ with the target detection frame. T Among them, the calculation process of the attention propagation module is shown in formula (6):

[0123]

[0124] in, is a 1×1 convolution, d k is the dimension of feature map Q and K, Q t 、k t-1 、v t-1 、V t is the feature map obtained in step 54.

[0125] In step 6, the multi-target detection and tracking model composed of steps 2, 3, 4, and 5 is trained using the training set of step 1, and tested using the test set to finally obtain a trained multi-target detection and tracking model.

[0126] Step 7: Input the video data to be detected into the trained multi-target detection and tracking model to obtain a tracking feature map with a target detection frame.

[0127] In order to verify the feasibility and effectiveness of the present invention, the present invention conducted the following experiments:

[0128] First, for the multi-target detection module (i.e., steps 2 to 4), the average precision and recall rate are used to evaluate the model. The average precision rate is obtained from the precision rate. The formulas for the precision rate P and recall rate R are shown in Equations (7) and (8).

[0129]

[0130]

[0131] Where P is the percentage of the target that should be retrieved (TP) to all the targets that were retrieved (TP+FP). R is the percentage of the target that should be retrieved (TP) to all the targets that should be retrieved (TP+FN).

[0132] In detection tasks, precision reflects the model's accuracy, while recall reflects its recall. These two metrics constrain each other. The relative balance between precision and recall is found by calculating the average precision (AP) at different confidence thresholds. A two-dimensional PR curve is constructed with precision and recall as the horizontal and vertical coordinates. The average precision (AP) is the area enclosed by the PR curve, equivalent to the average precision.

[0133] This paper first quantitatively analyzes the multi-target detection module, comparing it to a baseline model on the VisDrone_mot dataset. The experiment then adds performance comparisons of each category with the proposed method. The results show that, compared to other models, the proposed method is able to accurately identify smaller targets while also maintaining good performance for larger targets. Compared to conventional models with superior performance, the proposed method achieves the best recognition performance for larger targets, with accuracies of 42.16 and 33.10, demonstrating excellent detection capabilities.

[0134] At the same time, in order to intuitively reflect the performance of the overall multi-target detection module, the module is qualitatively analyzed, and the results are as follows: Figure 6As shown in the figure, it can be seen that the model of the present invention has good detection performance for targets of different scales. After adding the Transformer module, the model captures long-distance dependencies more stably, and has good recognition ability for small targets while the recognition effect for large targets is still relatively robust.

[0135] Secondly, for the multi-target tracking module (i.e., step 5), the MOTA (↑), MOTP (↑), IDF1 (↑), MT (↑), ML (↓), FP (↓), FN (↓), Frag (↓), and IDSW (↓) indicators are used for evaluation. ↑ indicates that the larger the value of the indicator, the better the model performance, and ↓ indicates that the smaller the value of the indicator, the better the model performance.

[0136] Among them, MOTA stands for multi-target tracking accuracy, which measures the ability of the algorithm to continuously track the target and is used to count the error accumulation during tracking. Its formula is shown in (9).

[0137]

[0138] Among them, m t Corresponding to FP, it represents the false positive (false detection number) in the prediction result, that is, there is no corresponding tracking target matching the predicted position in the tth frame. t Corresponding to FN, it represents the false negative (missed detection number), that is, the target in the tth frame has no corresponding predicted position matching it. mme t Corresponding to IDSW, represents the number of mismatches, that is, the number of ID switches of the tracking target in the tth frame, g t Refers to the total number of true targets in a frame. MOTA comprehensively considers false detection, missed detection, and ID swapping in target trajectories.

[0139] The MOTP representation also directly reflects the effect of model tracking and the distance between the tracking result and the label trajectory. The formula is shown in (10).

[0140]

[0141] Among them, c t Indicates the number of matches in the t-th frame, and calculates the trajectory error for each pair of matches Then sum up to get the final value. The larger the index is, the better the model performance is and the smaller the trajectory error is.

[0142] MT stands for Mostly Tracked, which refers to the number of hit trajectories exceeding 80% of the labeled trajectories. A higher value is better. ML stands for Mostly Lost, which refers to the number of lost trajectories exceeding 80% of the labeled trajectories. A lower value is better. Frag stands for Fragmentation, which refers to the number of transitions from the "tracking" state to the "not tracking" state.

[0143] For a multi-target tracking detector, ID-related indicators are equally important. Specifically, there are three important indicators: IDP, IDR, and IDF1. IDP stands for Identification Precision, which refers to the ID recognition accuracy of each target box. Its formula is shown in (11).

[0144]

[0145] Where IDTP and IDFP are the number of true positive and false positive cases of ID prediction, respectively. IDR stands for Identification Recall, which refers to the ID recognition recall rate of each target box, and its formula is shown in (12).

[0146]

[0147] Where IDFN is the false negative case of ID prediction. IDF1 represents the F-score of ID prediction, which refers to the ID recognition F-score of each target box. The larger the value, the better. Its calculation formula is shown in (13).

[0148]

[0149] IDF1 is the first default indicator used to evaluate the quality of the tracker. Any two of the above three indicators can be used to infer the other one.

[0150] First, quantitative experiments compared the multi-target tracking module with mainstream baseline models from recent years. On the VisDrone_mot dataset, the proposed tracking method outperformed the second-best model by 3.2 and 1.8 in the MOTA and MOTP metrics, respectively, and achieved relatively good results on other metrics. However, the low false detection rate of the proposed model resulted in normal perturbations in the ML and MT metrics. Compared to the TBD model, the JDT model, due to the mutual promotion of detection and tracking tasks, enables end-to-end optimization during training and achieves better results on tracking tasks.

[0151] Secondly, the model is qualitatively analyzed in the above dataset, such as Figure 7Figure 1 shows two test examples, each of which displays four images from frames 0, 5, 10, and 15 in time. The figure demonstrates that the model is able to stably track multiple targets in traffic scenarios, and is particularly capable of detecting and tracking small targets in traffic.

Claims

1. A multi-target detection and tracking method in a complex urban road environment, characterized by: The specific steps include: Step 1: Select a public dataset for data augmentation, obtain the dataset, and construct the training set and test set; Step 2: Based on the existing DLA34 backbone network, feature fusion modules are added layer by layer to achieve deep and shallow network feature fusion of the input image, and a two-dimensional feature map is obtained after the three features are fused; Step 3: Based on the two-dimensional feature map after feature fusion, the Transformer encoding module is used to extract the long-range feature dependencies in the feature map to obtain the feature map after dependency extraction. This specifically includes the following sub-steps: Step 31: collapse the 1 / 16 size two-dimensional feature map obtained in step 2 into a one-dimensional sequence and convolute it to form K, V, and Q feature maps; Step 32: Add the position code to the feature map K and feature map Q obtained in step 31 pixel by pixel to obtain two feature maps with position information. These two feature maps and feature map V are input into the multi-head attention module as a common input, and a new feature map is obtained after processing. Step 33: perform a fusion operation and a LayerNorm operation on the new feature map obtained in step 32 and the V, K, and Q feature maps obtained in step 31 by adding corresponding values ​​between the feature maps. Step 34: The result obtained in step 33 is processed in a feedforward neural network and output through a residual connection to obtain a new feature map; Step 4: Generate heatmap and target bounding box through further feature fusion and logistic regression processing; specifically includes the following sub-steps: Step 41, upsampling the feature map finally obtained in step 3 by a factor of 2 to obtain a new feature map; Step 42: The feature maps of 1 / 4 size and 1 / 8 size obtained in step 24 are fused using the same feature fusion module as in step 24 to obtain a new feature map of 1 / 4 size. Step 43: The feature maps of 1 / 8 size and 1 / 16 size obtained in step 24 are fused using a feature fusion module, and are added pixel by pixel to the feature map obtained in step 41 to obtain a new feature map of 1 / 8 size. Step 44: The feature map of 1 / 4 size obtained in step 42 and the feature map of 1 / 8 size obtained in step 43 are fused using the feature fusion module to generate a heat map with a resolution of 1 / 4 size of the original image. Step 45: Perform logistic regression on the heat map obtained in step 44 and the heat map label containing the target center point in the data set obtained in step 1 to obtain the center point of the predicted target. ; In step 46, the coordinates of the upper left and lower right points of the bounding box corresponding to each target are obtained by equation (3) to generate the target bounding box: (3) in, That is, step 45 obtains the center point of the predicted target, Indicates the offset between the center point and the target center point, Indicates the size of the bounding box corresponding to the target; Step 5: Use the multi-target tracking module to perform target association processing and tracking to obtain a tracking feature map with target detection frames. This specifically includes the following sub-steps: Step 51, input the same image of step 2 as the first Frame image, and select the next frame image Frame image, the first Frame and The frame image is used as input and processed by the CenterTrack backbone network to generate feature maps. and ; Step 52: The feature map and They are sent to the cost space module for target association processing to obtain the output feature map ; Specifically include the following operations: Step 521: transform the feature map and The three-layer weight-sharing convolution structure in the cost space module is fed into the feature map and , that is, the appearance encoding vector of the target; Step 522: feature map and Perform the maximum pooling operation to obtain and To reduce the complexity of the model, use and The cost space matrix is ​​obtained by transposing the product , the cost space matrix The position of the target on the current frame is , from the cost space matrix Extract the two-dimensional cost matrix containing the position information of the target in the current frame in the previous frame image ,right The horizontal and vertical directions are respectively maximized to obtain the feature map of the corresponding direction ; Step 523, define two offset templates using equations (4) and (5) 、 ; (4)， (5)， in, is the downsampling multiple of the feature map relative to the original image, is the width and height size of the feature map, for Target in the frame image exist Appears in the horizontal position in the frame image The offset, for Frame Target exist Appears in vertical position in the frame image The offset of Step 524, the step 522 obtained 、 The offset template defined in step 523 and After multiplication, the channels are superimposed to obtain the feature map , represents the target's offset template in both horizontal and vertical directions; then Perform 2x upsampling to restore size, and at the same time, The horizontal and vertical channels of the feature map are respectively obtained in step 51 The channels are superimposed, and then convolution is performed to form two feature maps with the same size of feature maps in the horizontal and vertical directions and 9 channels. These two feature maps are superimposed on the channels to obtain the output feature map ; Step 53: Compare the heat map obtained in step 4 with the feature map obtained in step 51 Perform Hadamard product to generate feature map ,Will The feature map obtained in step 52 Perform deformable convolution together to generate feature maps ; Step 54: Use 3 in sequence Convolution operation and downsampling operation generate the first Frame feature map; the feature map obtained in step 51 Use 3 Convolution is performed to generate the Frame feature map; Step 55, the first The frame feature map is input into the attention propagation module for feature propagation to obtain the tracking feature map with the target detection frame. ; Step 6: Use the training set of step 1 to train the multi-target detection and tracking model composed of steps 2, 3, 4, and 5, and use the test set to test it, and finally obtain the trained multi-target detection and tracking model; Step 7: Input the video data to be detected into the trained multi-target detection and tracking model to obtain a tracking feature map with a target detection frame.

2. The multi-target detection and tracking method in a complex urban road environment according to claim 1, characterized in that: In step 1, VisDrone_mot in the mainstream traffic target detection dataset VisDrone is selected as the dataset.

3. The multi-target detection and tracking method in a complex urban road environment as claimed in claim 1, characterized in that: The step 2 specifically includes the following sub-steps: Step 21: Input the image in the training set into the DLA34 network. After the BatchNorm layer and the ReLU layer, two convolution operations with a kernel size of 3×3 are performed on the original image to obtain two feature maps. The two convolved feature maps are input into the aggregation node for feature fusion to obtain a feature map with a resolution of 1 / 4 of the original input feature map. Step 22: Downsample the 1 / 4 size feature map obtained in step 21 by a factor of 2 to obtain a new feature map. Repeat the convolution operation and aggregation operation in step 21 twice on the feature map to obtain two feature maps. The two feature maps are aggregated again with the aggregation node obtained in step 21 as the common input to obtain a feature map with a resolution of 1 / 8 of the original input feature map. Step 23: In the same manner as in step 22, a feature map of 1 / 8 size is obtained from a feature map of 1 / 4 size, a feature map of 1 / 16 size is obtained from a feature map of 1 / 8 size, and then a feature map of 1 / 32 size is obtained from a feature map of 1 / 16 size; In step 24, the obtained feature maps of 1 / 4 size, 1 / 8 size, 1 / 16 size, and 1 / 32 size are sequentially fused using a feature fusion module to perform feature fusion on adjacent feature maps to obtain new feature maps of 1 / 4 size, 1 / 8 size, and 1 / 16 size, respectively.

4. The multi-target detection and tracking method in a complex urban road environment as claimed in claim 3, characterized in that: In step 24, the feature fusion module is used to implement the following operations: Step 241: Perform deformable convolution processing with a convolution kernel of 3×3 on the feature map F1, and pass the processed result through a BatchNorm layer and a ReLU layer to obtain a mapped feature map; Step 242: Use direct interpolation upsampling plus convolution processing to replace the transposed convolution in the DLA34 backbone network, and perform a 2-fold upsampling on the feature map obtained in step 241 to obtain a feature map F1'; Step 243, adding the corresponding channel values ​​of the feature map F1' and the feature map F2 obtained in step 242 to obtain a merged feature map; Step 244: The merged feature map obtained in step 243 is processed by a 3×3 deformable convolution, and then passes through a BatchNorm layer and a ReLU layer in sequence to obtain a two-dimensional feature map F2'; When the feature map F1 and the feature map F2 are respectively a feature map of 1 / 4 size and a feature map of 1 / 8 size, the obtained two-dimensional feature map F2' is a feature map of 1 / 4 size; When the feature map F1 and the feature map F2 are respectively a feature map of 1 / 8 size and a feature map of 1 / 16 size, the obtained two-dimensional feature map F2' is a feature map of 1 / 8 size; When the feature map F1 and the feature map F2 are respectively a feature map of 1 / 16 size and a feature map of 1 / 32 size, the obtained two-dimensional feature map F2' is a feature map of 1 / 16 size.

5. The multi-target detection and tracking method in a complex urban road environment as claimed in claim 1, characterized in that: The position code in step 32 is obtained by the following formula: ， ， in, The matrix representing the positional encoding has the same resolution as the input feature map. represents the position of the vector in the sequence, Indicates the index of the channel, Indicates the number of channels of the input feature map.

Images