A pedestrian multi-target tracking method combined with instance segmentation

Abstract

The application discloses a kind of pedestrian multi-target tracking methods of joint instance segmentation.The method uses ResNet-18 and FPN as feature extraction network, introduces CBAM attention mechanism module for feature enhancement;In target detection branch, the quality of training is improved using positive and negative sample allocation strategy based on Pseudo-IoU;In segmentation branch, propose data-related global attention upsampling module, supplement the details features lost in upsampling process;Introduce tracking branch to extract the appearance feature vector of segmentation instance, construct score matrix by calculating the inner product of the appearance feature vector of the target instance of the current frame and the previous frame and the IoU value of the target detection frame, and use Hungarian algorithm to maximum match the target instance and tracking instance of the current frame, so that instance segmentation network can simultaneously realize the functions of segmentation and tracking.The application can be deployed in embedded device Jetson NX, to realize real-time pedestrian segmentation and tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and deep learning, and particularly relates to a pedestrian multi-target tracking method combined with instance segmentation. BACKGROUND

[0002] Instance segmentation is a hot issue in computer vision. Image target detection is performed to obtain target regions of different categories, a specific region target candidate region is obtained by subdividing target regions of the same category, and a segmentation result of a target image is obtained by segmenting each candidate region. Instance segmentation is widely used in automatic driving, medical diagnosis, public security management and other fields.

[0003] At present, most instance segmentation methods have good accuracy, such as Mask R-CNN, BlendMask and the like. However, most instance segmentation methods have the characteristics of large amount of calculation and difficulty in deployment on embedded devices, and in some scenes with high population flow and mutual occlusion of crowds, the effect of instance segmentation is greatly reduced, and it is difficult to deploy in application scenarios with more pedestrians.

[0004] Existing multi-target tracking methods are tracking methods in the TBD (Tracking by Detection) paradigm, that is, a target detection algorithm (such as YOLOv5, RetinaNet, etc.) is used to predict the target category in the image, and a rectangular frame is used for positioning, and then a multi-target tracking algorithm (such as SORT, DeepSORT, etc.) is used to track the target according to the obtained positioning frame. In recent years, multi-target tracking methods in the JDE paradigm have gradually developed. Such methods extract the appearance feature vector of the target object in the positioning frame while performing target detection, and match the appearance feature vectors of the targets in the front and rear frames according to the distance between them. However, in some places with serious occlusion, it is difficult to achieve accurate tracking only by the positioning of the target, so it is necessary to segment the edge of the target to provide more accurate positioning information.

[0005] In the TBD paradigm target tracking method, the tracking algorithm matches the front and rear frame targets according to the detection information. Although the structure is simple, some features are wasted, and some extracted features in the target detection method can also be used as the basis for target association. The fusion of instance segmentation and multi-target tracking can solve the problems of target detection, segmentation and tracking at the same time, and can make full use of the features extracted by deep learning to achieve higher precision tracking.

[0006] Currently, there are few joint instance segmentation multi-target tracking methods. The patent application CN202210565509 "Off-site construction deep learning instance segmentation tracking method based on vision" uses a mask R-CNN based on ResNet-101 to perform instance segmentation on the video target, and uses Kalman filtering to predict the position of the detection frame in the next frame, while extracting the appearance feature vector of the target segmentation mask, realizing instance segmentation and tracking of the target. In the industry, although the mask R-CNN has high accuracy, it is difficult to deploy to edge devices due to large calculation, and the method only adds tracking function to the original mask R-CNN without improving the original segmentation network. Based on this, the present application proposes a lightweight joint instance segmentation multi-target tracking method. SUMMARY

[0007] Therefore, the present application proposes a real-time joint instance segmentation pedestrian multi-target tracking method. The method is based on deep learning, and a tracking branch is embedded in the instance segmentation network to realize an efficient multi-target segmentation and tracking method, improving the efficiency of pedestrian multi-target tracking.

[0008] The object of the present application is achieved at least by one of the following technical solutions.

[0009] A joint instance segmentation pedestrian multi-target tracking method, which is a deep learning based method, takes a video frame sequence image as input during forward inference, and finally outputs target detection information of all possible target instances in the video frame, including target class score, target detection frame, target instance segmentation mask and target tracking trajectory. The method includes the following steps:

[0010] S1, after normalizing the video frame data, send it into the feature extraction network to extract multi-scale feature maps;

[0011] S2, send the multi-scale feature maps extracted in step S1 into the target detection branch to generate pedestrian detection frames, classes and segmentation mask coefficients;

[0012] S3, send the largest resolution feature map extracted in step S1 into the segmentation branch to generate a segmentation prototype mask, linearly multiply the segmentation mask coefficients generated in step S2, and normalize through the Sigmoid function to obtain the final target segmentation mask;

[0013] S4, send the segmentation mask of each target instance into the tracking branch to generate an appearance vector, calculate the IoU value between the detection box of the current frame and the tracking box, and the inner product between the target appearance vector of the current frame and the target appearance vector of the previous frame, and the sum of the IoU value and the inner product value constitutes a score matrix, and the same target of the previous frame and the current frame is matched by using the Hungarian algorithm to form a tracking trajectory.

[0014] Further, in step S1, the feature extraction network comprises a lightweight ResNet-18 network and an FPN feature pyramid network, so that the method can consume less computing resources in the feature extraction stage and can run in real time on an embedded device; the ResNet-18 network outputs a plurality of feature maps of different dimensions after feature extraction of the image, including a first backbone network feature map C2, a second backbone network feature map C3, a third backbone network feature map C4 and a fourth backbone network feature map C5 arranged in descending order of dimension;

[0015] The second backbone network feature map C3, the third backbone network feature map C4 and the fourth backbone network feature map C5 are respectively connected to a lightweight CBAM module, each CBAM module outputs a first CBAM feature map C3', a second CBAM feature map C4' and a third CBAM feature map C5' respectively, which are sent into the FPN network to improve the feature extraction capability of the model in the region of interest, and finally the FPN network generates five feature maps of different dimensions, including a first FPN feature map P3, a second FPN feature map P4, a third FPN feature map P5, a fourth FPN feature map P6 and a fifth FPN feature map P7 arranged in descending order of dimension, and the specific operation is as follows:

[0016] The third CBAM feature map C5' is subjected to 1x1 convolution to obtain the third FPN feature map P5, the third FPN feature map P5 is up-sampled so that the resolution of the third FPN feature map P5 becomes 2 times the original resolution, and is subjected to 1x1 convolution operation; the second CBAM feature map C4' is subjected to 1x1 convolution operation and added to the third FPN feature map P5 to obtain the second FPN feature map P4, and the second FPN feature map P4 is also up-sampled and subjected to 1x1 convolution; the first CBAM feature map C3' is subjected to 1x1 convolution operation and added to the second FPN feature map P4 to obtain the first FPN feature map P3; the third FPN feature map P5 is subjected to two 3x3 convolution operations to obtain the fourth FPN feature map P6 and the fifth FPN feature map P7, respectively.

[0017] Further, the target detection branch is composed of 4 3x3 convolution layers sharing a convolution channel, and then 3 3x3 convolutions are accessed to generate a class prediction tensor, a box prediction tensor and a segmentation mask coefficient prediction tensor respectively; the dimension size of the class prediction tensor is KxHxW, wherein K is the number of classes, the dimension size of the box prediction tensor is 4xHxW, and the dimension size of the segmentation mask coefficient is 32xHxW; wherein the resolution of each prediction tensor is the same as that of the input feature map, H is the height of the prediction tensor, and W is the width of the prediction tensor

[0018] Further, in step S3, the segmentation branch includes 3 convolution layers, a global attention upsampling module and 2 convolution layers, and the steps are as follows:

[0019] S101, in the segmentation branch, the first FPN feature map P3 is first subjected to 3 convolution operations to obtain a feature-enhanced feature map, i.e., the first segmentation branch feature map P3_1;

[0020] S102, the first backbone network feature map C2 with a resolution of 2Hx2W generated by the ResNet-18 network and the first segmentation branch feature map P3_1 are sent into the global attention upsampling module to obtain the second segmentation branch feature map P3_2 with a resolution of 2Hx2W;

[0021] S103, the second segmentation branch feature map P3_2 obtained in step S102 is subjected to 2 times of 3x3 convolution to obtain a prototype segmentation mask with a dimension of 32x2Hx2W.

[0022] Further, the global attention upsampling module includes 1 1x1 convolution layer, 1 3x3 convolution layer, 1 global average pooling layer and 1 DUpSampling module; the global attention upsampling module takes the first backbone network feature map C2 as a kind of attention mechanism, and fuses it with the first FPN feature map P3, and the specific operation is as follows:

[0023] S201, the first backbone network feature map C2 output by the ResNet-18 network is first subjected to global average pooling, so that the resolution of the first backbone network feature map C2 is consistent with that of the first FPN feature map P3, and the first backbone network feature map C2 after pooling is subjected to 1x1 convolution to obtain the first global attention feature map C2_1; the first global attention feature map C2_1 is multiplied by the first segmentation branch feature map P3_1 to obtain the second global attention feature map C2_2;

[0024] S202, the second global attention feature map C2_2 is subjected to upsampling by the DUpSampling module, so that the resolution of C2_2 is expanded by two times to obtain the third global attention feature map C2_3;

[0025] 203. Perform a 3×3 convolution on the first backbone network feature map C2 to obtain the fourth global attention feature map C2_4. Add the third global attention feature map C2_3 and the fourth global attention feature map C2_4 to obtain the second segmentation branch feature map P3_2. Finally, the global attention upsampling module outputs the second segmentation branch feature map P3_2.

[0026] Further, in step S4, the tracking branch includes an average pooling layer and two fully connected layers, and its specific operation is as follows:

[0027] The target segmentation mask predicted in the current frame is passed through an average pooling layer to obtain a first tracking branch feature map A_1 with a resolution of 72×72. Then, the first tracking branch feature map A_1 is fed into the first fully connected layer to obtain a second tracking branch feature vector A_2 with a length of 1024. Finally, the second tracking branch feature vector A_2 is fed into the second fully connected layer for further feature extraction and compression to obtain an appearance feature vector with a length of 128.

[0028] Furthermore, in the tracking branch, an additional storage queue ψ stores the detection boxes and appearance vectors generated in previous frames. The inner product between the appearance vector of the target instance in the current frame and the appearance vector of ψ, as well as the IoU value between the detection box in the current frame and the detection box in ψ, are calculated. The sum of the inner product value of the appearance vectors and the IoU value of the detection boxes is used to calculate the similarity score between the targets in the previous and current frames, and a score matrix is ​​constructed. The Hungarian algorithm is used to perform maximum matching on the score matrix to complete the association between the target instances in the previous and current frames.

[0029] Replace the matching bounding boxes and appearance vectors in ψ with the matching bounding boxes and appearance vectors in the current frame; while the unmatched bounding boxes and appearance vectors in the current frame are added to ψ as newly appearing targets.

[0030] Furthermore, the feature extraction network, object detection branch, segmentation branch, and tracking branch need to be jointly trained, and the training set for joint training is consecutive frame images from multiple videos.

[0031] Before training, the contours and categories of targets in the video images need to be labeled, generating target categories, bounding boxes, and segmentation masks, which are then written into a labeling file. During training, each labeled target instance is assigned to feature maps of different scales according to the size of the bounding box for prediction. The points of the labeled segmented instances are mapped onto the feature maps, and the points mapped onto the feature maps are used as the first positive sample points P'. Then, the Pseudo-IoU method is used to select the second positive sample point P.

[0032] Furthermore, the target detection branch needs to be trained using both positive and negative samples.

[0033] Further, the trained feature extraction network, target detection branch, segmentation branch and tracking branch are optimized by TensorRT during forward inference and deployed on Nvidia Jetson NX devices for real-time operation.

[0034] Compared with the prior art, the present application has the following advantages:

[0035] 1) In the feature extraction stage, the present application uses a lightweight backbone network ResNet-18 for image feature extraction and uses a CBAM module to enhance the feature extraction capability of ResNet-18, thereby improving the accuracy of the entire algorithm model.

[0036] 2) In the training, the present application uses a refined positive sample allocation strategy to select training positive samples, thereby improving the training quality.

[0037] 3) The present application proposes a data-dependent global attention up-sampling module, which can well supplement some detailed features lost during up-sampling of the feature map, thereby improving the overall segmentation accuracy of the algorithm.

[0038] 4) The present application embeds a tracking branch in the instance segmentation network to extract the appearance vector of the target instance, and correlates the target instances of the previous and subsequent frames through the inner product of the appearance vector and the IoU value of the target detection box.

[0039] 5) The present application can run at near real-time speed on an embedded device. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a flowchart of a joint instance segmentation multi-target tracking method according to an embodiment of the present application;

[0041] Figure 2 is a structure diagram of a lightweight attention mechanism module CBAM used in an embodiment of the present application;

[0042] Figure 3 is a feature extraction network structure diagram with CBAM according to an embodiment of the present application;

[0043] Figure 4 is a target detection branch structure diagram according to an embodiment of the present application;

[0044] Figure 5 is a sample selection strategy diagram of the target detection branch according to an embodiment of the present application;

[0045] Figure 6 is a Pseudo-IoU diagram used in an embodiment of the present application;

[0046] Figure 7is a segmentation branch structure diagram proposed by the embodiment of the present application;

[0047] Figure 8 is a tracking branch structure diagram proposed by the embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Other embodiments obtained by those skilled in the art based on the whole embodiments of the present application are within the scope of protection of the present application.

[0049] Embodiment:

[0050] Reference Figure 1 A pedestrian multi-target tracking method combined with instance segmentation provided by the embodiment of the present application has the characteristics that it includes the following steps:

[0051] S1, after normalizing the video frame data, the normalized video frame data is sent into a feature extraction network to extract multi-scale feature maps;

[0052] S2, the multi-scale feature maps extracted in step S1 are sent into a target detection branch to generate pedestrian detection boxes, classes and segmentation mask coefficients;

[0053] S3, the feature map with the largest resolution extracted in step S1 is sent into a segmentation branch to generate a segmentation prototype mask, the segmentation prototype mask is linearly multiplied by the segmentation mask coefficients generated in step S2, and is normalized through a Sigmoid function to obtain a final target segmentation mask;

[0054] S4, the segmentation mask of each target instance is sent into a tracking branch to generate an appearance vector, the IoU value between the current frame detection box and the tracking box and the inner product between the target appearance vector of the current frame and the target appearance vector of the previous frame are calculated, the sum of the IoU value and the inner product value constitutes a score matrix, and the Hungarian algorithm is used to match the same target of the previous frame and the current frame to form a tracking trajectory, as shown in the following formula: Figure 1 Each step of the embodiment of the present application will be clearly and completely described below, and the description is in a sequential manner, and each paragraph of description is a supplement to the previous paragraph of description.

[0055] Step S1, the resolution of the current picture is adjusted to the size of 576x576, and the picture data is normalized, and the specific formula of the normalization is as follows: Wherein, x is the value of RGB three channels of each pixel point on the image, the value of means is [103.94, 116.78, 123.6], and the value of stds is [57.38, 57.12, 58.40], The normalized image data is 1*3*576*576. The normalized image data is input into the feature extraction network. In order to improve the running efficiency of the whole algorithm model, ResNet-18 and FPN are used as the feature extraction network in the embodiment. ResNet-18 is a lightweight backbone network. In order to improve the feature extraction ability of ResNet-18, the feature maps output by ResNet-18, {the second backbone network feature map C3, the third backbone network feature map C4, and the fourth backbone network feature map C5} are connected to a CBAM attention mechanism module respectively, so as to improve the feature extraction ability of the feature extraction network. The structure diagram of the CBAM module is shown in Figure 2 The CBAM module is composed of a channel attention mechanism module and a spatial attention mechanism module. The calculation formula of the channel attention mechanism module is shown in formula (1).

[0056] M c (F)=sigmoid(MLP(AvgPool(F))+MLP(MaxPool(F))) (1)

[0057] M c (F) represents the channel weight map generated by the channel attention mechanism module for the feature map F. AvgPool(F) and MaxPool(F) represent the outputs obtained by the feature map F after the average pooling layer and the maximum pooling layer respectively. Then, the two outputs are input into the shared network MLP, and normalized by the Sigmoid function.

[0058] The calculation formula of the spatial attention mechanism module is shown in formula (2).

[0059] M s (F')=Sigmoid(f 7×7 ([AvgPool(F');MaxPool(F')])) (2)

[0060] Wherein, F' represents the output feature map of the channel attention mechanism module, M s (F') represents the spatial region weight map generated by the spatial attention mechanism module for F', f 7×7 represents a convolution layer with a convolution kernel size of 7*7, AvgPool(F') and MaxPool(F') represent the outputs obtained by the feature map F' after the average pooling layer and the maximum pooling layer respectively.

[0061] {the second backbone network feature map C3, the third backbone network feature map C4, the fourth backbone network feature map C5} are outputted as {the first CBAM feature map C3', the second CBAM feature map C4', the third CBAM feature map C5'} after passing through the CBAM module, as shown in Figure 3 {the first CBAM feature map C3', the second CBAM feature map C4', the third CBAM feature map C5'} are inputted into the FPN network, feature fusion is performed, and finally the FPN network outputs five feature maps of different scales {the first FPN feature map P3, the second FPN feature map P4, the third FPN feature map P5, the fourth FPN feature map P6, the fifth FPN feature map P7}. The specific operation is as follows: the third CBAM feature map C5' is subjected to 1x1 convolution to obtain the third FPN feature map P5, the third FPN feature map P5 is up-sampled so that the resolution of the third FPN feature map P5 becomes 2 times the original resolution, and 1x1 convolution operation is performed; the second CBAM feature map C4' is subjected to 1x1 convolution operation, and is added to the third FPN feature map P5 to obtain the second FPN feature map P4, and the second FPN feature map P4 is also subjected to up-sampling and 1x1 convolution; the first CBAM feature map C3' is subjected to 1x1 convolution operation, and is added to the second FPN feature map P4 to obtain the first FPN feature map P3; the third FPN feature map P5 is subjected to two 3x3 convolution operations to obtain the fourth FPN feature map P6 and the fifth FPN feature map P7, respectively.

[0062] After the feature extraction in step S1, multi-scale feature maps {the first FPN feature map P3, the second FPN feature map P4, the third FPN feature map P5, the fourth FPN feature map P6, the fifth FPN feature map P7} can be obtained, the down-sampling multiples of adjacent feature maps are 2, and the resolution of the first FPN feature map P3 is the largest; these feature maps will be inputted into a target detection branch to generate a target detection box, a target category and a segmentation mask coefficient. The structure diagram of the target detection branch is as shown in Figure 4 The branch is composed of 4 3x3 convolution layers to form shared convolution channels, and then 3 independent branches are connected to generate a category prediction tensor, a box prediction tensor and a segmentation mask coefficient prediction tensor, respectively. The dimension size of the category prediction tensor is KxHxW, where K is the number of categories; the dimension size of the box prediction tensor is 4xHxW; and the dimension size of the segmentation mask coefficient is 32xHxW. The resolution of each prediction tensor is consistent with the resolution of the feature map inputted into the target detection branch. In an embodiment: when the fifth FPN feature map P7 inputted has a dimension of 512x5x5, the dimension of the outputted category prediction tensor is Kx5x5, the dimension of the box prediction tensor is 4x5x5, and the dimension of the mask coefficient prediction tensor is 32x5x5.

[0063] Step S3, the first FPN feature map P3 with the largest resolution is sent into the segmentation branch to generate a prototype segmentation mask. The segmentation branch includes three 3x3 convolution layers, one global attention upsampling layer and two convolution layers, as shown in Figure 7 The steps are as follows: S101, the first FPN feature map P3 is subjected to three convolution operations to obtain a first segmentation branch feature map P3_1 with enhanced features; S102, the first backbone network feature map C2 with a resolution of 2Hx2W generated by the ResNet-18 and the first segmentation branch feature map P3_1 are sent into a global attention upsampling module to obtain a second segmentation branch feature map P3_2 with a resolution of 2Hx2W; S103, the second segmentation branch feature map P3_2 obtained in S102 is subjected to two 3x3 convolutions to obtain a prototype segmentation mask with a dimension of 32x2Hx2W. The prototype segmentation mask needs to be linearly multiplied with the mask coefficient generated in the target detection branch and activated by a Sigmoid function to obtain the final target segmentation mask.

[0064] In the segmentation branch, since the resolution of the input first FPN feature map P3 is 72x72, which is quite different from the resolution 576x576 of the input image, it is necessary to upsample the feature map in the segmentation branch to expand the resolution of the feature map. Upsampling is to obtain more pixel data and improve the quality of the prototype segmentation mask. However, simple bilinear upsampling is a data-independent operation that will lose a lot of detailed features. Therefore, the present embodiment proposes a global attention upsampling module that fuses a DUpSampling module, as shown in Figure 7The global attention up-sampling module comprises: a 1x1 convolution layer, a 3x3 convolution layer, a global average pooling layer, and a DUpSampling module. The global attention up-sampling module fuses the first backbone network feature map C2 as an attention mechanism with the first FPN feature map P3. The first backbone network feature map C2 mainly contains edge information and some shallow features of the image, and the edge information of the first backbone network feature map C2 is used to supplement the part of the details lost in the up-sampling process. The specific operation is as follows: S201, the first backbone network feature map C2 output by the ResNet-18 is first subjected to global average pooling, so that the resolution of the first backbone network feature map C2 is consistent with that of the first FPN feature map P3, and the first backbone network feature map C2 after the pooling is subjected to 1x1 convolution to obtain the first global attention feature map C2_1, the first global attention feature map C2_1 is multiplied with the first segmentation branch feature map P3_1 to obtain the second global attention feature map C2_2; S202, the second global attention feature map C2_2 is up-sampled by the DUpSampling module, so that the resolution of the second global attention feature map C2_2 is expanded by two times to obtain the third global attention feature map C2_3, the DUpSampling up-sampling module learns the interpolation value required for up-sampling through a convolutional neural network, and restores the pixel-level segmentation prediction from the convolutional neural network; S203, the first backbone network feature map C2 is subjected to 1 time 3x3 convolution to obtain the fourth global attention feature map C2_4, the third global attention feature map C2_3 and the fourth global attention feature map C2_4 are added to obtain the second segmentation branch feature map P3_2, and finally the global attention up-sampling module outputs the second segmentation branch feature map P3_2.

[0065] Step S4, after the prototype segmentation mask is generated by the segmentation branch, the segmentation mask coefficient generated by the detection branch is linearly multiplied to obtain the final segmentation mask. The segmentation mask of each target instance is sent to the tracking branch to generate the appearance vector of the target instance. The tracking branch comprises an average pooling layer and two fully connected layers, as shown in Figure 8 The predicted target segmentation mask in the current frame is subjected to the average pooling layer to obtain a first tracking branch feature map A_1 with a resolution of 72x72; then the first tracking branch feature map A_1 is sent to the first fully connected layer to obtain a second tracking branch feature vector A_2 with a length of 1024; finally, A_2 is sent to the second fully connected layer for further feature extraction and compression to obtain an appearance feature vector with a length of 128.

[0066] The tracking branch stores the bounding box and appearance vector generated by the previous frame through an additional storage queue ψ, calculates the inner product between the appearance vector of the target instance in the current frame and the appearance vector in the storage queue, and the IoU value between the bounding box of the current frame and the bounding box in the storage queue, sums the inner product value of the appearance vector and the IoU value of the bounding box to obtain the similarity score between the front and rear frames of the target, constructs a score matrix, uses the Hungarian algorithm to perform maximum matching on the score matrix, and completes the association of the target instances in the front and rear frames. The matched bounding box and appearance vector in the current frame are replaced by the matched bounding box and appearance vector in the current frame. The bounding box and appearance vector in the current frame which are not matched are added to the storage queue as new appearing targets.

[0067] The present application is a deep learning-based method, which needs to jointly train the feature extraction network, the target detection branch, the segmentation branch and the tracking branch. When training the target detection branch, positive and negative samples are selected for training. The training set is a plurality of continuous frame images of videos and a label file, and the label file gives the class, label box coordinates and segmentation mask of the target in each video frame image. Before training, the outline of the target and the target class in the video image are labeled using the VIA labeling tool, the class, label box and segmentation mask of the target are generated, and written into the label file. The target label information in the label file is read during training.

[0068] During the training of the target detection branch, each feature map input into the target detection branch will output three prediction tensors: a class prediction tensor, a box prediction tensor and a segmentation mask coefficient prediction tensor. In the present application example, five feature maps output by the FPN are sent into the target prediction branch, therefore, the target detection branch will finally generate five prediction tensors of different dimensions. In an embodiment: the dimensions of the five class prediction tensors are: Kx72x72, Kx36x36, Kx18x18, Kx9x9, Kx5x5; the dimensions of the five box prediction tensors are: 4x72x72, 4x36x36, 4x18x18, 4x9x9, 4x5x5; the dimensions of the five mask coefficient prediction tensors are: 32x72x72, 32x36x36, 32x18x18, 32x9x9, 32x5x5. Positive and negative sample points need to be selected from these prediction tensors for training.

[0069] According to the area of the label box of the target in the image, targets of different scales are assigned to different feature maps for prediction. In an embodiment, it is assumed that rl={512, 256, 128, 64, 32}, the label box area is S, and when The bounding box is assigned to the feature map l for prediction, where n is 2, and l is in {P3, P4, P5, P6, P7}. After assigning the feature map, the labeled instance points of the original image are mapped to the feature map to determine the first positive sample point. Assuming that the labeled instance point of the original image is p = (x, y), the point mapped to the feature map is where s is the down-sampling multiple between the original image and the feature map. Referring to FIG. 2(a) in the specification, Figure 5 Figure 5 FIG. 2(a) in the specification is a generated prediction tensor, Figure 5 FIG. 2(c) in the specification is a labeled segmentation mask of the original image size, where the white area represents the target, and the black area is the background. The labeled segmentation mask here has only one target. In order to make the resolution of the labeled segmentation mask consistent with that of the prediction tensor, the resolution of the labeled segmentation mask needs to be adjusted, and FIG. 2(b) in the specification can be obtained. Figure 5 Figure 5 C in FIG. 2(a) in the specification can be K, 4 or 32, representing three different prediction tensors. Taking a class prediction tensor as an example, in Figure 5 point ③ in FIG. 2(b) in the specification is a point on the target instance, and point ④ is a background point, so as to correspond to Figure 5 in FIG. 2(a) in the specification, point ① is a positive sample point, and point ② is a negative sample point. After that, similarly, point ① on the frame prediction tensor and the mask coefficient prediction tensor with the same resolution is also a positive sample point, and point ② is a negative sample point.

[0070] The above method of determining the positive sample point is relatively rough. Because the targets that occlude each other may have overlapping positive sample regions when mapped to the prediction tensor, if different labeled target supervision information is assigned to the overlapping positive samples each time the training is performed, the training direction will be inconsistent, making it difficult for the training to converge. Therefore, the present example determines the final positive sample point that needs to be trained by the Pseudo-IoU method, referring to FIG. 3 in the specification. Figure 6 Figure 6 In FIG. 3(a) in the specification, p is a pixel point belonging to a labeled instance in the original image, p' is the position of the positive sample point mapped back to the original image on the feature map, the black solid line box is a labeled target box, {lp, rp, tp, bp} are the distances from the point p to the left boundary, right boundary, upper boundary and lower boundary of the target box respectively, the black dashed line box is a pseudo-box generated according to {lp, rp, tp, bp} and the position of p', {lp', rp', tp', bp'} are the distances from the point p' to the left boundary, right boundary, upper boundary and lower boundary of the pseudo-box respectively, and the calculation formula of {lp', rp', tp', bp'} is: Then, the IoU value of the labeled box and the pseudo-box is calculated by formula (3).

[0071] ​​​

[0072] In the above formula, Sp and Sp' are the areas of the bounding box and the pseudo box, respectively, S ∩ is the intersection area of the bounding box and the pseudo box, S ∪ is the total area of the bounding box and the pseudo box. A threshold τ is set, and the positive sample points that satisfy IoU > τ are taken as the second positive sample points for regression training, and otherwise as negative sample points.

[0073] In training the tracking branch of the present embodiment, the input is two pictures, one of which is a key frame and the other is a reference frame. In training, the target instances labeled in the reference frame are separately sent into the tracking branch to generate the appearance vector of each target instance, and an additional storage queue ψ is used to save the appearance vector and the bounding box of the target instance. Then the key frame is sent into the instance segmentation network to obtain the predicted target instance segmentation mask and the detection box, and the target instance segmentation mask in the key frame is sent into the tracking branch to extract the appearance vector. Assuming that the number of target instances in the reference frame is N, and the appearance vector is The number of target instances in the key frame is M, and the appearance vector is The judgment of the tracking ID of a certain instance in the key frame can be converted into an N+1-class classification problem, where N represents the number of instances being tracked in the reference frame, and 1 represents that the instance is a new instance entering the video frame. The probability of similarity between the two appearance vectors can be calculated by formula (4).

[0074]

[0075] If a certain instance in the key frame is a new target that has just entered the video frame, its tracking ID does not belong to one of [1, N], and its tracking ID needs to be set to 0. The similarity is calculated by formula If a certain instance in the key frame is the same target as a certain instance in the reference frame, the similarity is calculated by formula .

[0076] In the training phase, the present method has a total of 4 loss values to be calculated, which are the class loss, the box regression loss, the segmentation mask loss, and the tracking loss. Therefore, in the classification task, how to handle the imbalance between positive and negative sample training is the key to training the network well. The classification task will output a KxHxW tensor, K is the number of classes in the data set, and is normalized by the Sigmoid function. If a target belongs to the i-th class (i∈[0,K-1]), the i-th probability of the corresponding K-dimensional vector is set to 1, and the rest is 0; if a target belongs to the background, the corresponding K-dimensional vector is 0. The present method selects the Focal Loss function to train the classification task, and the Focal Loss function is shown in formula (5).

[0077]

[0078] In equation (5), Let y be the predicted probability value for a certain category, α be the target probability for that category, and β be hyperparameters, where α takes the value 0.25 and β takes the value 2. For y = 1, when the predicted value... The closer it is to 1, the higher its weighting coefficient becomes. The smaller the value, the smaller the loss; conversely, the larger the value, the larger the loss. Therefore, when the predicted value is closer to the true label, the overall loss function focuses more on difficult samples.

[0079] The bounding box regression task outputs a 4×H×W tensor, where each point on the feature map predicts four values ​​to generate the target detection box. Assuming the positive sample of the selected region on the 4×H×W feature map is p', and its coordinates on the feature map are [x', y'], then its coordinates mapped to the original image are [r(x'+0.5), r(y'+0.5)], where r is the downsampling factor between the feature map and the original image. In the method of this invention, the value of r is [4, 8, 16, 32, 64]. Let the coordinates of the upper left and lower right corners of the corresponding bounding box of p' on the original image be [x1, y1] and [x2, y2], respectively. Then the formula for calculating the target value in the bounding box regression task is shown in equation (6).

[0080]

[0081]

[0082] The loss value for the box regression task is calculated using the GIoU Loss function. The formula for calculating GIoU Loss is shown in Equation (7).

[0083]

[0084] Loss bbox =1-GIoU (7)

[0085] In equation (7), C represents the smallest rectangular box containing detection boxes A and B. When calculating GIoU Loss, box A is the detection box predicted by the method of this invention, and box B is the annotation box in the image. When box A and box B do not intersect, the IoU value between box A and box B is 0. At this time, the value of GIoU Loss is not 0, thus ensuring that training can still be performed when box A and box B do not intersect.

[0086] The segmentation mask loss is trained using the Binary Cross Entropy Loss (BCE) function. The loss calculation formula is shown in Equation (8).

[0087] Loss mask= -(ylog(p(x)) + (1 - y)log(1 - p(x))) (8)

[0088] In the calculation of the segmentation mask loss, let the segmentation mask predicted by the method be M, and the ground truth of the segmentation mask be M gt The ground truth of the segmentation mask is a binary mask image with a resolution of 144x144, where the value of each pixel is 0 or 1, 0 represents the background, and 1 represents that the pixel belongs to the target instance. The resolution of the predicted segmentation mask is also 144x144, and the value of each pixel is normalized by the Sigmoid function and takes a value in the range (0, 1). Therefore, when calculating the loss value by formula (8), p(x) represents the value of each pixel in the segmentation mask M, and y represents the value of the corresponding pixel point in M gt , and finally the calculated BCE Loss of each pixel point is summed.

[0089] In the training of the tracking branch, only the inner product of the target instance appearance feature vector is used to form the score matrix, and the Hungarian algorithm is used for maximum matching. The loss value of the tracking branch needs to minimize the feature difference between instances with the same tracking ID and maximize the difference between appearance feature vectors with different tracking IDs. Therefore, the method adopts a TripletLoss function to calculate the loss of the tracking branch, as shown in formula (9).

[0090]

[0091] In formula (9), f i is the appearance feature vector of an instance in the current frame, is the appearance feature vector of an instance with the same tracking ID as f i , is the appearance feature vector of an instance with a different tracking ID from f i , and ε is a hyperparameter.

[0092] Finally, the total loss function of the method is shown in formula (10).

[0093]

[0094] In the inference stage, the input of the model is only one picture. After obtaining the target instance segmentation mask of the current frame, if it is the first frame, a tracking ID is assigned to all target instances, and the appearance vector and the detection box of all instances are saved through an additional queue ψ. In each subsequent frame, the inner product calculation of the appearance vectors of all target instances and the existing appearance vectors in ψ and the IoU calculation of the target detection boxes are performed to form a score matrix, and the Hungarian algorithm is used to perform maximum matching on the score matrix. If a certain instance in the current frame is the same target as a certain instance in the last frame, the appearance vector and the target detection box of the instance are replaced with the corresponding appearance vector and the detection box in ψ, otherwise a new target instance is assigned a new tracking ID, and its appearance vector and detection box are added to ψ.

[0095] In an embodiment, in order to implement an instance segmentation tracking algorithm with high effect on an embedded device (Jetson NX) with limited computing power, an acceleration engine of the algorithm is generated using the TensorRT framework, and the acceleration engine of the algorithm is deployed on the Jetson NX for running. Finally, the algorithm can run at a processing speed of 25 frames per second.

[0096] The following table shows the effect comparison of the method of the application.

[0097]

[0098] In summary, the multi-target tracking method combined with instance segmentation realized by the application has the following advantages:

[0099] 1) In the feature extraction stage, the CBAM module is used to enhance the feature extraction capability of ResNet-18, and the accuracy of the entire algorithm model is improved.

[0100] 2) In the detection branch, a refined positive sample assignment strategy is used to select training positive samples, improving the training quality.

[0101] 3) A data-dependent global attention upsampling module is proposed, which can well supplement some detailed features lost during upsampling of the feature map, thereby improving the overall segmentation accuracy of the algorithm.

[0102] 4) A tracking branch is embedded in the instance segmentation network to extract the appearance vector of the target instance, and the distance of the appearance vector and the IoU value of the target detection box are used to associate the target instances of the previous and subsequent frames.

[0103] 5) The method can run at near real-time speed on an embedded device.

[0104] Many specific details are set forth in the above description in order to provide a thorough understanding of the application. However, it will be appreciated by those skilled in the art that the application can be practiced in a variety of ways, and that the application is not limited to the details described above.

[0105] In summary, although the above embodiments are described, it should be understood that the application is not limited to the embodiments described above, but rather, that the application is intended to cover modifications and variations of the embodiments described above, provided they come within the scope of the appended claims.

Claims

1. A method of joint instance segmentation based multi-object tracking of pedestrians, characterized in that, Comprise the following steps: S1, after the video frame data is normalized, it is sent into the feature extraction network, and multi-scale feature maps are extracted; S2, the multi-scale feature maps extracted in step S1 are sent into the target detection branch to generate pedestrian detection boxes, classes and segmentation mask coefficients; S3, the largest resolution feature map extracted in step S1 is sent into the segmentation branch to generate a segmentation prototype mask, which is linearly multiplied with the segmentation mask coefficients generated in step S2, and normalized by a Sigmoid function to obtain the final target segmentation mask; S4, the segmentation mask of each target instance is sent into the tracking branch to generate an appearance vector, the IoU value between the current frame detection box and the tracking box is calculated, and the inner product between the target appearance vector of the current frame and the target appearance vector of the previous frame is calculated, the sum of the IoU value and the inner product value constitutes a score matrix, and the Hungarian algorithm is used to match the same target in the front and rear frames to form a tracking trajectory. 2.The method of claim 1, wherein, In step S1, the feature extraction network comprises a ResNet-18 network and a FPN feature pyramid network; the ResNet-18 network extracts features from the image and outputs a plurality of feature maps with different dimensions, including a first backbone network feature map C2, a second backbone network feature map C3, a third backbone network feature map C4 and a fourth backbone network feature map C5 arranged in descending order of dimension; Wherein, the second backbone network feature map C3, the third backbone network feature map C4 and the fourth backbone network feature map C5 are respectively connected to a CBAM module, and the first CBAM feature map C3', the second CBAM feature map C4' and the third CBAM feature map C5' corresponding to each CBAM module are respectively sent into the FPN network, and finally the FPN network generates five feature maps with different dimensions, including a first FPN feature map P3, a second FPN feature map P4, a third FPN feature map P5, a fourth FPN feature map P6 and a fifth FPN feature map P7 arranged in descending order of dimension, and the specific operation is: The third CBAM feature map C5' is subjected to 1×1 convolution to obtain the third FPN feature map P5, the third FPN feature map P5 is up-sampled so that the resolution of the third FPN feature map P5 becomes twice the original, and 1×1 convolution operation is performed; the second CBAM feature map C4' is subjected to 1×1 convolution operation, and is added to the third FPN feature map P5 to obtain the second FPN feature map P4, and the second FPN feature map P4 is also subjected to up-sampling and 1×1 convolution; the first CBAM feature map C3' is subjected to 1×1 convolution operation, and is added to the second FPN feature map P4 to obtain the first FPN feature map P3; the third FPN feature map P5 is subjected to two 3×3 convolution operations to obtain the fourth FPN feature map P6 and the fifth FPN feature map P7. 3.The method of claim 1, wherein, The target detection branch is composed of 4 3x3 convolution layers shared convolution channels, and then 3 3x3 convolutions are connected to generate a class prediction tensor, a box prediction tensor and a segmentation mask coefficient prediction tensor respectively; the dimension of the class prediction tensor is KxHxW, wherein K is the number of classes, the dimension of the box prediction tensor is 4xHxW, and the dimension of the segmentation mask coefficient is 32xHxW; wherein H is the height of the prediction tensor, W is the width of the prediction tensor, and the resolution of each prediction tensor is the same as that of the input feature map. 4.The method of claim 3, wherein, In step S3, the segmentation branch includes 3 convolution layers, a global attention upsampling module and 2 convolution layers, and the steps are as follows: S101, in the segmentation branch, first pass the first FPN feature map P3 through 3 convolution operations to obtain a feature-enhanced feature map first segmentation branch feature map P3_1; S102, the first backbone network feature map C2 generated by the ResNet-18 network has a resolution of 2Hx2W, and the first segmentation branch feature map P3_1 is sent into the global attention upsampling module to obtain a second segmentation branch feature map P3_2 with a resolution of 2Hx2W; S103, the second segmentation branch feature map P3_2 obtained in step S102 is subjected to 2 times of 3x3 convolution to obtain a prototype segmentation mask with a dimension of 32x2Hx2W.

5. The method of claim 4, wherein, The global attention upsampling module includes 1 1x1 convolution layer, 1 3x3 convolution layer, 1 global average pooling layer and 1 DUpSampling module; the global attention upsampling module takes the first backbone network feature map C2 as a kind of attention mechanism, and fuses the first FPN feature map P3, and the specific operation is as follows: S201, the first backbone network feature map C2 output by the ResNet-18 network is first subjected to global average pooling, so that the resolution of the first backbone network feature map C2 is consistent with that of the first FPN feature map P3, and the pooled first backbone network feature map C2 is subjected to 1x1 convolution to obtain a first global attention feature map C2_1, the first global attention feature map C2_1 is multiplied by the first segmentation branch feature map P3_1 to obtain a second global attention feature map C2_2; S202, the second global attention feature map C2_2 is upsampled by the DUpSampling module to expand the resolution of C2_2 by two times to obtain a third global attention feature map C2_3; S203, the first backbone network feature map C2 is subjected to 1 time 3x3 convolution to obtain a fourth global attention feature map C2_4, the third global attention feature map C2_3 and the fourth global attention feature map C2_4 are added to obtain the second segmentation branch feature map P3_2, and finally the global attention upsampling module outputs the feature map second segmentation branch feature map P3_2. 6.The method of claim 1, wherein, In step S4, the tracking branch includes an average pooling layer and two fully connected layers, and the specific operation is as follows: The target segmentation mask obtained by prediction in the current frame is passed through an average pooling layer to obtain a first tracking branch feature map A_1 with a resolution of 72x72; then the first tracking branch feature map A_1 is sent to a first fully connected layer to obtain a second tracking branch feature vector A_2 with a length of 1024; finally, the second tracking branch feature vector A_2 is sent to a second fully connected layer for further feature extraction and compression to obtain an appearance feature vector with a length of 128.

7. The method of claim 6, wherein, In the tracking branch, an additional memory queue is used The appearance vectors of the detected boxes generated in the previous frame are stored, and the appearance vectors of the target instances in the current frame are calculated The inner product between the appearance vectors of the target instances in the current frame and the previous frame is calculated, and the IoU value between the detected boxes in the current frame and the previous frame is calculated The sum of the inner product value of the appearance vectors and the IoU value of the detected boxes is taken as the similarity score between the target instances in the previous frame and the current frame, a score matrix is constructed, the Hungarian algorithm is used to perform maximum matching on the score matrix, and the association between the target instances in the previous frame and the current frame is completed. replacing the matched detection boxes and appearance vectors in the current frame with the matched detection boxes and appearance vectors in the previous frame ; and the detection boxes and appearance vectors in the current frame that do not match are added as new targets in the . 8.The method of any one of claims 1-7, wherein, The feature extraction network, the target detection branch, the segmentation branch and the tracking branch need to be jointly trained, and the training set of the joint training is the continuous frame images of multiple videos. Before training, the contour of the target in the video image and the target category need to be labeled, the category, the labeled box and the segmentation mask of the target are generated, and written into a label file; during training, each labeled target instance is allocated to a feature map of different scales according to the size of the labeled box for prediction, and the points of the labeled segmentation instance are mapped to the feature map, the points mapped to the feature map are taken as first positive sample points P', and then second positive sample points P are selected by using a Pseudo-IoU method. 9.The method of claim 8, wherein, The target detection branch needs to select positive and negative samples for training. 10.The method of claim 8, wherein, The trained feature extraction network, target detection branch, segmentation branch and tracking branch are accelerated and optimized by TensorRT during forward inference, and are deployed on an Nvidia Jetson NX device for real-time operation.

Images