A multi-stage video pedestrian multi-target tracking method and model assisted by re-identification

Abstract

This invention discloses a multi-stage video pedestrian multi-target tracking method and model with re-identification assistance, which constructs a system including a backbone network. ResNet50 A multi-layer feature aggregation module, including an attention-guided context aggregation module and an attention-guided module. IDAUP The submodule uses the MSMOT model to detect pedestrian bounding boxes and extracts pedestrian re-identification features; the attention guidance module includes a context attention module and a content attention module. IDAUP The submodule outputs shallow and deep feature information as inputs to the re-identification and detection branches, respectively. The detection branch employs three parallel... Head Used for estimation HeatMap The algorithm considers the offset of the center point and the size of the bounding box; it uses a data association algorithm based on Kalman filtering and the Hungarian algorithm to match the detected bounding box with the corresponding pedestrian trajectory, relying on the detected bounding box and the re-identification features.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision technology, and particularly relates to a multi-stage video pedestrian multi-target tracking method and model assisted by re-identification. BACKGROUND

[0002] In recent years, how to build a safe and harmonious society has become an important task of governments around the world. Among them, the video monitoring system is a widely used security means in the world. With the large-scale development of "safe city" and "smart city" engineering construction in China, the number of video monitoring cameras installed in China grows at a rate of more than 20% per year, which has made remarkable achievements in strengthening public security, optimizing traffic, serving city management, and innovating social governance, and has greatly promoted the development of public security video monitoring. However, the traditional video monitoring system needs to rely on a large number of human resources, and it is difficult to achieve real-time processing, and mainly plays a role in taking evidence after the event. Therefore, an intelligent monitoring system emerges as the times require. By adding an intelligent video analysis module to the monitoring system, with the powerful data processing capability of the computer, useless or interfering information in the video is filtered out, different objects are automatically identified, key information in the video is analyzed, abnormal situations in the monitoring picture are judged, and the fastest way is used to warn or trigger other actions, so as to realize a full-automatic, all-weather, real-time monitoring intelligent system that can effectively perform pre-warning, in-process processing, and timely evidence taking after the event.

[0003] Computer vision technology gives video monitoring the ability to recognize and analyze video sequences. Multi-target tracking, as an important direction in the field of computer vision, aims to identify, locate and assign corresponding identity labels to multiple targets in a video sequence, and associate the same target with the same identity label in consecutive video frames within a period of time to form a complete target trajectory. Multi-target tracking technology has a wide range of applications in real life, such as abnormal pedestrian detection, dense crowd analysis, and motion video analysis. Existing multi-target tracking technology has achieved great success, but still faces many problems in some scene conditions. In real scenes, especially in complex crowded scenes, obstacles and occlusions between pedestrians seriously affect the quality of pedestrian detection results. In addition, in high-traffic scenarios, it is very difficult to correctly associate the identities of all pedestrians. The influence of interference factors makes it easy to switch identities when occlusions occur.

[0004] The current methods for completing multi-target tracking tasks using computer vision can be divided into two categories. The first category is the detection-then-tracking framework (Tracking-By-Detection, TBD), and the second category is the joint detection and tracking framework (Join-Detection-Tracking, JDT).

[0005] Among them, the multi-target tracking model of TBD framework is the mainstream in current academic and industrial circles. With the rapid development of target detection field, the emergence of high-quality target detector provides more accurate detection results for TBD multi-target tracker. Therefore, the existing TBD multi-target tracker is mainly composed of high-quality target detector and data association algorithm. Specifically, the target detector is used to locate multiple pedestrians in the scene, and the rectangular detection bounding box is used for calibration. The Kalman filter is used to estimate the subsequent position of the target, and the feature distance or IoU between the estimated bounding box and the detected bounding box is used to form the cost matrix. Finally, the Hungarian algorithm is used for matching to complete the whole data association part. In order to establish a more long-term stable data relationship, the pedestrian re-identification technology also begins to become an auxiliary of multi-target task, which mainly extracts the overall appearance features of pedestrians through neural network, and then uses the distance between the features as the constraint condition of the cost matrix in the Hungarian algorithm matching process.

[0006] Yolov3, as an industrial widely used target detector, is combined with DeepSort data association method to form the basis of the current industrial mainstream multi-target tracking implementation. FairMOT is one of the representatives of multi-target tracking model combining detection and re-identification, which maintains stable association in a long time while ensuring tracking speed. Specifically, FairMOT uses the detection model CenterNet of Anchor-Free framework, which eliminates the time-consuming NMS operation in the detection process, effectively improving the tracking performance of pedestrians. The re-identification branch of FairMOT is parallel to the detection branch, and is linked at the tail of the multi-layer aggregation module (DLA), which is used to obtain the detection bounding box and extract the pedestrian re-identification features, respectively. The detection branch is composed of three different tasks Head, which are used for center point position prediction, down-sampling quantization error correction and bounding box size regression, respectively. The re-identification branch is used to extract 128-dimensional feature vectors. The tracking part of FairMOT uses the combination of widely used Kalman filter and Hungarian algorithm, in which the constraint of the cost matrix is composed of the IoU of the Kalman filter prediction bounding box and the actual detection bounding box and the cosine distance between the re-identification features.

[0007] Overall, existing pedestrian multi-target tracking technology (TBD or JDT) is mainly divided into two stages, the first stage is to use a target detector to locate and calibrate the pedestrian target in the scene, the second stage is to use Kalman filtering to predict the subsequent position of the pedestrian, and to use the Hungarian algorithm to associate the pedestrian detection with the previous pedestrian trajectory. However, this multi-target tracking method which only relies on the pedestrian detection result often shows low efficiency when facing poor pedestrian detection quality, temporary loss of pedestrian target and other situations, so as to be unable to cope with the problems such as occlusion interference, size change and temporary loss of target which often occur in actual scenes. SUMMARY

[0008] The purpose of the present application is to provide a multi-stage video pedestrian multi-target tracking method and model assisted by re-identification, which aims to improve the speed of pedestrian multi-target tracking while realizing a pedestrian multi-target tracking method with long-term association ability which is robust to size and angle changes, occlusion interference, based on the shortcomings of existing pedestrian multi-target tracking technology; the model takes FairMOT as the BaseLine, analyzes and solves some problems and suboptimal places existing in FairMOT, and realizes a more rapid and accurate pedestrian multi-target tracking model.

[0009] The present application is realized by the following technical scheme: a multi-stage video pedestrian multi-target tracking method assisted by re-identification, comprising the following steps:

[0010] 1) Build an MSMOT model, and use it to detect the detection boundary box of the pedestrian and extract the re-identification features of the pedestrian;

[0011] 2) Through a data association algorithm based on Kalman filtering and the Hungarian algorithm, relying on the detection boundary box and re-identification features obtained in step 1), the matching between the detection boundary box and the corresponding pedestrian trajectory is completed.

[0012] The MSMOT model comprises a ResNet50 as a backbone network, a multi-layer feature aggregation module (DLA) arranged at each level of the ResNet50 and used for converging local and global features, an attention-guided context aggregation module (AGC) added at the top layer of the multi-layer feature aggregation module (DLA) and comprising a context grabbing module (CEM) and an attention guiding module (AGM), and an IDAUP sub-module (used for reconstructing the image size and resolution from the converged features), wherein the attention guiding module (AGM) comprises a context attention module (CxAGM) and a content attention module (CnAGM), the IDAUP sub-module outputs shallow feature information and deep feature information as inputs of a re-identification branch and a detection branch respectively, the detection branch adopts three parallel Heads respectively for estimating a HeatMap, a center point offset and a size of a bounding box, the context grabbing module (CEM) comprises multiple levels for merging multi-scale information, the levels are densely connected, each level further obtains multiple feature maps in different receptive fields through a dilated convolution with different expansion rates, and a deformable convolution (DCN) is added in each path to improve the adaptability of the model to geometric transformation, the attention guiding module (AGM) comprises the context attention module (CxAGM) and the content attention module (CnAGM), the context attention module (CxAGM) focuses on the semantics between sub-regions of a given feature map, and the content attention module (CnAGM) focuses on ensuring spatial information.

[0013] Further, in order to better realize the multi-stage video pedestrian multi-target tracking method assisted by re-identification, the following arrangement is particularly adopted:

[0014] 1.1) ResNet50 is selected as the backbone network of the MSMOT model; the backbone network is used for extracting common basic features of an input video frame, ResNet50 is adopted as a basic common feature extraction network; ResNet50 transmits front layer information to rear layer information through the design of a residual block, so that the information of the front layer is not lost in the deepening process of the network, and ResNet50 is commonly used in various feature extraction tasks;

[0015] 1.2) a multi-layer feature aggregation module (DLA) is added after Level3- Level5 convolution blocks of the ResNet50, so as to converge features of different levels and different scales and enhance the multi-scale feature extraction capability of the backbone network; the multi-layer feature aggregation module (DLA) has a structure similar to a feature pyramid (FPN) and can converge features of different levels through a jump connection, and compared with the FPN, the DLA has more jump connections;

[0016] 1.3) Adding an attention-guided context aggregation module (AGC) on the top layer of the multi-layer feature aggregation module (DLA) to enhance the discriminability of global and local features; the attention-guided context aggregation module (AGC) is composed of a context extraction module (CEM) and an attention guidance module (AGM);

[0017] 1.4) Taking the output features of the top layer of the multi-layer feature aggregation module (DLA) as the input of the context extraction module (CEM) to search for large-scale context information from different receptive fields; the context extraction module (CEM) contains several additional levels, and the dense connections between the levels can better integrate multi-scale information. In addition, multiple feature maps are obtained in different receptive fields through different dilation rates of the hollow convolution, and a deformable convolution (DCN) is added in each path to improve the adaptability of the MSMOT model to geometric transformation;

[0018] 1.5) Since the output features of the context extraction module (CEM) contain rich receptive field information, but not all features are conducive to the detection and re-identification tasks, there is a lot of background redundant information, so the output features of the context extraction module (CEM) are taken as the input of the attention guidance module (AGM) to eliminate the negative effects of information redundancy in the receptive field. The use of the attention guidance module (AGM) can solve the misleading of the bounding box caused by the existence of background redundant information, and further improve the expression ability of the feature map;

[0019] Specifically, the attention guidance module (AGM) is composed of a context attention module (CxAGM) and a content attention module (CnAGM). The context attention module (CxAGM) mainly focuses on the semantics between sub-regions of a given feature map. However, due to the influence of deformable convolution (DCN), the position expression of these targets has changed. The content attention module (CnAGM) is designed to alleviate this problem, which focuses more on ensuring spatial information at the expense of a small amount of semantic information;

[0020] 1.6) The output features of the context extraction module (CEM) and the output features of the attention guidance module (AGM, composed of CxAGM and CnAGM) are input into the IDAUP sub-module for hierarchical feature extraction; that is, the mixed features (output features of the context extraction module (CEM) and output features of the attention guidance module (AGM)) are taken as the top layer output features of the multi-layer feature aggregation module (DLA), and then deeper feature extraction is performed after entering the IDAUP sub-module (specifically, in order to meet the different feature levels required by the detection task and the re-identification task, the detection task extracts deeper features, while the re-identification task extracts relatively shallow features);

[0021] 1.7) The feature information output by the IDAUP submodule for detection is used to estimate the center point position through HeatMap Head, to constrain the quantization error of the center point through Offset Head, and to regress the bounding box size through Box Size Head.

[0022] 1.8) The feature information output by the IDAUP submodule for re-identification is used to complete the re-identification feature extraction through the ReID Head.

[0023] To further improve the multi-stage video pedestrian multi-target tracking method with re-identification assistance described in this invention, the following configuration is specifically adopted: Step 1.7) includes the following steps:

[0024] 1.7.1) The feature information used for detection is used to estimate the center point location through HeatMap Head:

[0025] Define the height and width of the image frame as H. image and W image The generated HeatMap has a size of 1×H×W (H=H image / 4, W = W image / 4), the response of the generated HeatMap at the true location coordinates of the target center is 1, and it decays exponentially with distance from the center; for each detection bounding box DetBox i =(x 1i ,y 1i ,x 2i ,y 2i ), calculate the coordinates of its center point as C xi =(x 1i +x 2i ) / 2, C yi =(y 1i +y 2i ) / 2, which corresponds to the position (C) on the output feature map. xi ～ C yi ～ ) is ([C xi / 4],[C yi / 4]), where the symbol [] is used to indicate rounding down, and the response of HeatMap at (x, y) is represented as:

[0026]

[0027] Compared with the method of generating HeatMap by using a circular Gaussian kernel in the original FairMOT, formula (1) takes the aspect ratio of the detection bounding box into account and uses an elliptical Gaussian kernel to adapt to the aspect ratio of the bounding box. In general scenarios, the detection bounding box of pedestrians will present a condition that the height is greater than the width, and the ratio between the height and the width is often within a certain range, so the elliptical Gaussian kernel is better than the circular Gaussian kernel.

[0028] The loss function adopts Focal_Loss:

[0029]

[0030] N represents the number of targets in the picture, δ x 2 and δ y 2 are (αW / 6)2 and (αH / 6)2 respectively, where α is a hyperparameter, and its value is 0.54; α f and β f are hyperparameters in Focal_Loss, and their values are 2 and 4 respectively, M xy ～ is the HeatMap estimated by the network; M xy represents the HeatMap generated by formula (1).

[0031] 1.7.2) Feature information used for detection realizes the constraint on the quantization error of the center point through Offset Head:

[0032] For each GT bounding box GTBox i = (x 1i , y 1i , x 2i , y 2i ) in the image, the quantization error of the GT bounding box due to down-sampling is o i = (C xi ～ , C yi ～ )-([C xi / 4],[C yi / 4]), where o i ～ represents the estimated offset, and the quantization error of the center point is corrected by L1_Loss through the following formula:

[0033]

[0034] where, L offsetFor center point quantization loss, used to constrain the error caused by down-sampling; N is the number of targets (i.e. pedestrians) in the video frame; i is each target (i.e. pedestrian) in the video frame;

[0035] 1.7.3) Feature information for detection is realized by Box Size Head to regress the size of the bounding box:

[0036] The same elliptical Gaussian kernel as in the HeatMap generation process is used for the regression of the bounding box, and all pixels in the region A m with a Gaussian kernel response greater than 0 are regarded as samples for the regression of the size of the bounding box; compared with the method of using only the center point position as the regression sample in the original FairMOT, the regression samples in the region A m can accelerate the convergence of the model while reducing the random error caused by sample uncertainty; for each pixel (i,j) in the region A m and the down-sampling step r, the target to be regressed is defined as the distance from (ir,jr) to the four edges of the mth bounding box, represented by a 4-dimensional vector (wl,ht,wr,hb) ij m ; the predicted bounding box at (i,j) is represented as:

[0037]

[0038] where S is a fixed proportion value, and the predicted coordinates are the coordinates on the Image rather than the coordinates on the Feature-Map; if a pixel is not included in any elliptical Gaussian kernel region that meets the condition, it is ignored; if it is included in multiple elliptical Gaussian kernel regions, the target for its training is defined as the target in the smaller region; the GIoU value of the predicted bounding box and the GT bounding box is used as the constraint condition to obtain the following regression loss:

[0039]

[0040] where B ij ～ represents the predicted bounding box, B m represents the GT bounding box; W ij is the sample weight, used to balance the contribution of each sample to the loss; assuming that (i,j) is located in the A m region of the mth bounding box, the sample weight is:

[0041]

[0042] where G m (i,j) is the Gaussian response at (i,j), and a m is the Gaussian kernel target region of the mth bounding box.

[0043] Further, in order to better achieve the multi-stage video pedestrian multi-object tracking method assisted by re-identification, the following setting mode is particularly adopted: the step 1.8) is specifically:

[0044] 1.8.1) defining each GT bounding box GTBox in the image i =(x 1i ,y 1i ,x 2i ,y 2i );

[0045] 1.8.2) obtaining the center of the HeatMap as (C xi ～ ,C yi ～ );

[0046] 1.8.3) after step 1.8.2), re-identification features E(C xi ～ ,C yi ～ ) are grabbed from the point;

[0047] 1.8.4) after step 1.8.3), it is mapped to a class distribution vector P={p(k), k∈[1,K]};

[0048] 1.8.5) the real class label is represented by one-hot code L i (k), and the re-identification loss is:

[0049]

[0050] wherein K is the number of classes, and N is the number of samples in the image frame;

[0051] 1.8.6) all Head losses of the detection branch and the re-identification branch are weighted and summed to obtain the overall loss function L total in the training process of the MSMOT model, and

[0052] L total =W heat L heat +W offset L offset +W reg L reg +W id L id (8)。

[0053] Further, in order to better realize the multi-stage video pedestrian multi-target tracking method assisted by re-identification, the following setting mode is particularly adopted: the step 2) is specifically as follows: after the step 1), the detection bounding box information and the 128-dimensional re-identification feature are extracted, the detection bounding boxes are divided into three categories according to the confidence score of the detection bounding box, and three different confidence thresholds T h , T m , and T l are used to divide the detection bounding box intervals of the corresponding categories, and the specific division is as follows:

[0054] 2.1) First stage association (IoU or ReID): for the detection bounding box with the detection threshold greater than T h , it is considered that the influence of interference such as occlusion on the pedestrian detection task is small, and the quality of the extracted re-identification feature is high; therefore, when performing the data association task, it is considered that only one of the two conditions of IoU or re-identification feature distance needs to be met to complete the matching between the corresponding detection and track.

[0055] 2.2) Second stage association (IoU and ReID): for the detection bounding box with the detection threshold less than T h and greater than T m , it is considered that the influence of interference such as occlusion on the pedestrian detection task is small, and the quality of the extracted re-identification feature is high; therefore, in order to reduce the matching error, when performing the data association task, it is considered that the two conditions of IoU and re-identification feature distance are met at the same time to complete the matching between the corresponding detection and track.

[0056] 2.3) Third stage association (IoU): for the detection bounding box with the detection threshold less than T m and greater than T l , it is considered that the influence of interference such as occlusion on the pedestrian detection task is large, and the quality of the extracted re-identification feature is low; therefore, in order to avoid the damage of low-quality re-identification feature to the data association task, when performing the data association task, only IoU is used as the constraint condition, and the detection threshold less than T l is considered as background and discarded.

[0057] In the steps 2.1) and 2.3), the IoU is calculated from the Kalman filter predicted bounding box and the detection bounding box, the distance between the re-identification features adopts the Euclidean distance, and the Hungarian algorithm is used for bipartite graph matching based on the cost matrix; wherein, T h is the high confidence threshold for dividing high-quality detection bounding boxes, T m is the medium confidence threshold for dividing medium-quality detection bounding boxes, and T l is the low confidence threshold for dividing low confidence and background interference.

[0058] The application discloses a multi-stage video pedestrian multi-target tracking model assisted by re-identification, and relates to the technical field of video pedestrian multi-target tracking.

[0059] Further, in order to better realize the multi-stage video pedestrian multi-target tracking model assisted by re-identification, the following setting structure is particularly adopted: the attention guided context aggregation module (AGC) comprises a context capturing module (CEM) and an attention guiding module (AGM), the context capturing module (CEM) comprises multiple levels for merging multi-scale information, the levels are densely connected, each level further obtains multiple feature mappings in different receiving domains through a different expansion rate of a hollow convolution, and a deformable convolution (DCN) is added in each path to improve the adaptability of the model to geometric transformation.

[0060] Further, in order to better realize the multi-stage video pedestrian multi-target tracking model assisted by re-identification, the following setting structure is particularly adopted: the attention guiding module (AGM) comprises a context attention module (CxAGM) and a content attention module (CnAGM), the context attention module (CxAGM) focuses on the semantics between sub-regions of a given feature mapping, and the content attention module (CnAGM) focuses on ensuring spatial information.

[0061] Compared with the prior art, the application has the following advantages and beneficial effects:

[0062] The application adopts the ResNet50 network as a basic feature extraction backbone network, aggregates features of different levels through the multi-layer feature aggregation module (DLA), and enhances the performance of the features. In addition, the re-identification branch is earlier than the detection branch, can extract more fine-grained discriminative features, and the detection branch is located at a deeper position and can extract more global features; and the anchor-free detection framework can reduce the time overhead caused by anchors in the target detection process.

[0063] The application enhances the expression capability of HeatMap by using the elliptical Gaussian kernel, and is more suitable for actual scenes; and the use of the weighted GIoU loss function can better balance the detection bounding boxes of different sizes in actual scenes.

[0064] The MRMSA data association method adopted by the application fully considers the situation that the detection confidence score may be reduced due to interference such as occlusion in the scene, and a long-term stable data association is constructed by using re-identification features for auxiliary construction; in addition, the multi-stage data association is also a reasonable use of re-identification features under full consideration of the actual situation. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 The two stages are for the TBD multi-target tracking task, and the data association part is independent of the network model.

[0066] Figure 2 The MSMOT model of the application is a general network structure diagram.

[0067] Figure 3 The AGC structure diagram of the application is attention-guided context aggregation module (AGC).

[0068] Figure 4 The CxAGM structure diagram of the application is shown.

[0069] Figure 5 The CnAGM structure diagram of the application is shown.

[0070] Figure 6 The three task head visualization diagrams of the detection task are shown.

[0071] In Figure 6 , a is the detection bounding box of pedestrians in the scene, and b shows the difference between the elliptical Gaussian kernel region and the center point position.

[0072] Figure 7 The tracking situation diagram of pedestrians under different detection confidence is shown.

[0073] In Figure 7 , a is the detection result of different frames, b is the past tracking situation, and c is the ideal tracking situation (i.e. the data association method proposed by the application). DETAILED DESCRIPTION

[0074] The application will be further described in detail below in conjunction with the embodiments, but the embodiments of the application are not limited thereto.

[0075] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0076] Noun explanation:

[0077] Head: task head, which can be regarded as a task branch.

[0078] HeatMap: heat map.

[0079] Offset Head: offset correction task head, used to reduce the quantization error generated by the transmission of the target center point position through the network.

[0080] Box Size Head: bounding box regression task head, used to regress the bounding box size from the target center point.

[0081] MSMOT: a multi-stage multi-target tracking model, network structure: ResNet50+DLA+AGC, prediction part: elliptical Gaussian kernel+weighted GIoU loss.

[0082] MRMSA data association algorithm: IoU+ReID Feature Distances, multi-stage threshold: T h , T m , T l .

[0083] UP, Up: both are up-sampling modules.

[0084] CBL: ordinary convolution block.

[0085] MRMSA: multi-stage data association algorithm combined with motion information and re-identification assistance.

[0086] ReID: pedestrian re-identification.

[0087] IoU: intersection over union.

[0088] IDAUP: structure in a deep aggregation network for reconstructing target size and resolution.

[0089] Detection bounding box: the bounding box predicted by the network.

[0090] GT bounding box: the ground truth bounding box (i.e. the bounding box labeled on the dataset).

[0091] Anchor-Free detection framework: a kind of anchor-free detection framework without the step of generating pre-defined anchors.

[0092] Anchor: anchor box, in anchor-based detection framework, a certain number of candidate anchor boxes are generated on the whole image by a certain pre-defined means before the detection bounding box is obtained by regression, and the subsequent detection bounding box is also obtained by correcting these anchor boxes.

[0093] Weighted GIoU loss function: GIoU loss is equivalent to adding a closure penalty formed by GT bounding box and predicted bounding box in the function, and since IoU is a ratio concept, it is not sensitive to the size of the target object. GIoU loss can be regarded as an improvement.

[0094] MRMSA data association method: a multi-stage data association algorithm, which divides the set of to-be-associated detection boxes into different subsets by detection confidence, and calculates the cost matrix of different subsets in different ways.

[0095] The pedestrian multi-target tracking method described in the present application is divided into two stages. The first stage is the network architecture stage. This stage mainly obtains the detection bounding box of the pedestrian through the network model, and extracts the re-identification feature expression of the pedestrian, i.e. the detection and feature extraction stage. The second stage is the data association stage. This stage mainly uses the data association algorithm based on Kalman filter and Hungarian algorithm, relying on the detection bounding box and re-identification feature obtained in the first stage, to complete the matching between the detection bounding box and the corresponding pedestrian track, i.e. the tracking stage. The relationship between the two stages can be represented by Figure 1 Basic expression.

[0096] Network architecture stage: ResNet50 is used as the backbone network, and a multi-layer aggregation network (DLA) similar to FairMOT is added to each level of ResNet50 to converge local and global features. In addition, in order to enhance the expression of the network in extracting features, an attention-guided context aggregation module (AGC) is added to the upper layer of DLA. The output features of the top layer of the multi-layer feature aggregation module (DLA) are then input into the IDAUP sub-module for deeper feature extraction (specifically, in order to meet the different feature level requirements of the detection task and the re-identification task, the detection task (detection branch) extracts deeper features, while the re-identification task (re-identification branch) extracts relatively shallow features). Similar to FairMOT, the detection branch uses three parallel Heads to estimate the HeatMap, the offset of the center point, and the size of the bounding box, respectively. The overall network structure (MSMOT model) is shown in Figure 2 , where Level2-Level5 represent different convolutional blocks of the ResNet50 backbone network, the outputs of convolutional blocks Level3-Level5 are used for DLA calculation, AGC is an attention-guided context feature aggregation module, UP and Up are up-sampling modules with slightly different implementation methods. CBL is composed of convolutional layers, normalization layers, and activation functions, and the four different Heads are used to represent different tasks.

[0097] Data association stage: In the past, data association methods based on detection combined with Kalman filtering and the Hungarian algorithm have difficulty in associating occluded pedestrians when facing problems such as occlusion affecting detection quality. When facing detection bounding boxes with low confidence scores, the previous method usually considers them as background and discards them. However, interference factors such as occlusion may also cause the confidence score of the detection bounding box to decrease, and if it is discarded, it will affect the continuity of the tracking task. Figure 7 , (a) shows the detection results of three pedestrians in the scene in different frames. Due to the occlusion relationship between pedestrians, the detection confidence score of one of the pedestrians decreases (0.6 in Frame k1, 0.4 in Frame k2, and 0.2 in Frame k3). Figure 7 (b) shows that the previous data association method (traditional association method) will directly discard the bounding box with a low confidence score, so there is no longer a trajectory of this pedestrian in the subsequent frames, from Figure 7 (b) shows that the pedestrian in the solid line box is discarded in the tracking process, while the pedestrians in the dotted line box and the dashed line box are normally tracked. Combined with Figure 7 (c) shows that under actual occlusion conditions, even if there is partial occlusion interference, the trajectory of the occluded pedestrian should be estimated. For the above occlusion conditions, fromFigure 7 (c)It can be seen that the data association method proposed in the present application retains all the confidence score detection bounding boxes, that is, all the detection bounding boxes are included in the matching task with the trajectory (i.e. the solid line box pedestrians with lower scores caused by occlusion interference can also be tracked), so as to reduce the influence of occlusion interference. In addition, in order to maintain a long-term data association, the appearance information of the pedestrian, i.e. the re-identification feature, is used as an auxiliary for the data association task. Considering that the detection confidence degree reflects the quality of the detection and the quality of the re-identification feature to some extent, three thresholds T h , T m , T l are used to discuss the detection-trajectory matching method in different cases, which is called a multi-stage data association algorithm assisted by joint motion information and re-identification (MRMSA).

[0098] Embodiment 1:

[0099] A multi-stage video pedestrian multi-target tracking method assisted by re-identification, aiming at the shortcomings of the existing pedestrian multi-target tracking technology, the present application aims to improve the speed of pedestrian multi-target tracking while realizing a robust pedestrian multi-target tracking method with long-time association ability to size and angle changes and occlusion interference, comprising the following steps:

[0100] 1) Build a MSMOT model, and detect the detection bounding box of the pedestrian by using the model, and extract the re-identification feature of the pedestrian;

[0101] 2) Through a data association algorithm based on Kalman filtering and Hungarian algorithm, relying on the pedestrian detection bounding box and the pedestrian re-identification feature obtained in step 1), the matching between the detection bounding box and the corresponding pedestrian trajectory is completed.

[0102] The MSMOT model includes ResNet50 as a backbone network, a multi-layer feature aggregation module (DLA) arranged at each level of the ResNet50 and used for converging local and global features, an attention-guided context aggregation module (AGC) added at the top layer of the multi-layer feature aggregation module (DLA) and including a context grabbing module (CEM) and an attention guiding module (AGM), and an IDAUP sub-module (used for reconstructing the image size and resolution from the converged features), the attention guiding module (AGM) includes a context attention module (CxAGM) and a content attention module (CnAGM), the IDAUP sub-module outputs shallow feature information and deep feature information as inputs of a re-identification branch and a detection branch respectively, the detection branch adopts three parallel heads respectively for estimating a HeatMap, a center point offset and a bounding box size, the context grabbing module (CEM) includes multiple levels for merging multi-scale information, the levels are densely connected, each level also obtains multiple feature mappings in different receiving domains through different dilation rates of a hole convolution, and a deformable convolution (DCN) is added in each path to improve the adaptability of the model to geometric transformation, the attention guiding module (AGM) includes the context attention module (CxAGM) and the content attention module (CnAGM), the context attention module (CxAGM) focuses on the semantics between sub-regions of a given feature mapping, and the content attention module (CnAGM) focuses on ensuring spatial information.

[0103] Embodiment 2

[0104] This embodiment is further optimized on the basis of the above-mentioned embodiments, and the same as the foregoing technical solutions will not be repeated here. Further, in order to better realize the multi-stage video pedestrian multi-target tracking method assisted by re-identification, the following setting mode is particularly adopted:

[0105] 1.1) ResNet50 is selected as the backbone network of the MSMOT model; the backbone network is used for extracting common basic features of an input video frame, ResNet50 is adopted as a basic common feature extraction network; ResNet50 transmits front layer information to rear layer information through the design of a residual block, so that the information of the front layer is not lost in the deepening process of the network, and ResNet50 is commonly used in various feature extraction tasks;

[0106] 1.2) Add multi-level feature aggregation module (DLA) after Level3~Level5 convolution block of ResNet50 to converge features of different levels and different scales, enhance the multi-scale feature extraction ability of backbone network; the structure of multi-level feature aggregation module (DLA) is similar to feature pyramid network (FPN), which can converge features of different levels through skip connection, compared with FPN, DLA has more skip connections;

[0107] 1.3) In order to enhance the expression ability of extracted features and suppress the interference of background information, add attention guided context aggregation module (AGC) at the top layer of multi-level feature aggregation module (DLA) to enhance the discriminability of global features and local features; the attention guided context aggregation module (AGC) is composed of a context extraction module (CEM) and an attention guided module (AGM), and the structure of the attention guided context aggregation module (AGC) is as shown in Figure 3

[0108] 1.4) The output features of the top layer of multi-level feature aggregation module (DLA) are used as the input of context extraction module (CEM) to search large context information from different receptive fields; the context extraction module (CEM) contains several additional levels, and the dense connection between levels can better integrate multi-scale information, in addition, multiple feature maps are obtained in different receptive fields through different dilation rates of the hollow convolution, and deformable convolution (DCN) is added in each path to improve the adaptability of MSMOT model to geometric transformation;

[0109] 1.5) Since the output features of the context extraction module (CEM) contain rich receptive field information, but not all features are conducive to detection and re-identification tasks, there is a lot of background redundant information, therefore, the output features of the context extraction module (CEM) are used as the input of the attention guided module (AGM) to eliminate the negative effects of information redundancy in the receptive field, and the attention guided module (AGM) can solve the misleading of the bounding box caused by the existence of background redundant information, and further improve the expression ability of the feature map;

[0110] Specifically, the attention guided module (AGM) is composed of context attention module (CxAGM) and content attention module (CnAGM), the context attention module (CxAGM) mainly focuses on the semantics between sub-regions of a given feature map, however, due to the influence of deformable convolution (DCN), the position expression of these targets has changed; the content attention module (CnAGM) is to alleviate this problem, it pays more attention to ensuring spatial information, and sacrifices a small part of semantic information, and the structure of the context attention module (CxAGM) is as shown in Figure 4 ​As shown, the structure of the content attention module (CnAGM) is as follows Figure 5 As shown;

[0111] 1.6) The output features of the convergence context grabbing module (CEM) and the attention guiding module (AGM, composed of CxAGM and CnAGM) are mixed into features, which are input into the top layer output features of the multi-layer feature aggregation module (DLA, i.e. Figure 2 DLAUP) in IDAUP, and are input into the IDAUP submodule for hierarchical (shallow and deep) feature extraction; that is, the mixed features (output features of the context grabbing module (CEM) and the attention guiding module (AGM)) are taken as the top layer output features of the multi-layer feature aggregation module (DLA), and then are input into the IDAUP submodule for deeper feature extraction (specifically, in order to meet the different feature levels required by the detection task and the re-identification task, the detection task extracts deeper features, and the re-identification task extracts relatively shallow features);

[0112] 1.7) The feature information output by the IDAUP submodule for detection is used to estimate the center point position through the HeatMap Head, constrain the center point quantization error through the Offset Head, and regress the boundary box size through the Box Size Head;

[0113] 1.8) The feature information output by the IDAUP submodule for re-identification is used to extract the re-identification features through the ReID Head.

[0114] Embodiment 3:

[0115] This embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be repeated here. Further, in order to better realize the multi-stage video pedestrian multi-target tracking method assisted by re-identification, the following setting mode is particularly adopted:

[0116] 1.7.1) The feature information for detection is used to estimate the center point position through the HeatMap Head:

[0117] The height and width of the image frame are defined as H image and W image , the size of the generated HeatMap is 1xHxW (H=H image / 4, W=W image / 4), the response of the generated HeatMap at the real position coordinates of the target center is 1, and presents an exponential decay according to the distance from the center; for each detection boundary box DetBox i =(x 1iy 1i x 2i y 2i ), the center point coordinates are calculated as C xi =(x 1i +x 2i ) / 2, C yi =(y 1i +y 2i ) / 2, which correspond to the position (C xi ～ , C yi ～ ) on the output feature map ([C xi / 4], [C yi / 4]), where the symbol [] is used to represent the floor function. The response of HeatMap at (x, y) is represented as:

[0118]

[0119] Compared with the method of generating HeatMap by using a circular Gaussian kernel in the original FairMOT, formula (1) takes into account the aspect ratio of the detection bounding box and uses an elliptical Gaussian kernel to adapt to the aspect ratio of the bounding box. As shown in a in Figure 6 Under normal circumstances, the detection bounding box of a pedestrian will have a height greater than the width, and the ratio between the height and the width will often be within a certain range, so the elliptical Gaussian kernel is better than the circular Gaussian kernel.

[0120] The loss function uses Focal_Loss:

[0121]

[0122] where N represents the number of targets in the image, δ x 2 and δ y 2 are (αW / 6)2and (αH / 6)2, respectively, where α is a hyperparameter with a value of 0.54; α f and β f are hyperparameters in Focal_Loss, with values of 2 and 4, respectively, and M xy ～ is the estimated HeatMap; M xy represents the HeatMap generated by formula (1).

[0123] 1.7.2) Feature information used for detection The Offset Head is used to constrain the center point quantization error:

[0124] For each GT bounding box GTBox i =(x1i ,y 1i ,x 2i ,y 2i The quantization error of the GT bounding box due to downsampling is o. i =(C xi ～ C yi ～ )-([C xi / 4],[C yi / 4]), use o i ～ The estimated offset is represented by L1_Loss, which is used to correct the quantization error of the center point using the following formula:

[0125]

[0126] Among them, L offset The center point quantization loss is used to constrain the error caused by downsampling; N is the number of targets (i.e., pedestrians) in the video frame; i is each target (i.e., pedestrian) in the video frame;

[0127] 1.7.3) The feature information used for detection is used to regress the bounding box size through the Box Size Head:

[0128] like Figure 6 As shown in Figure b, the same elliptical Gaussian kernel as used in the HeatMap generation process is employed for bounding box regression, identifying regions A with a Gaussian kernel response greater than 0. m All pixels in region A are considered as samples for regressing the bounding box size; compared to the original FairMOT method which only uses the center point position as the regression sample, region A... m Within region A, regression samples can accelerate model convergence while reducing random errors caused by sample uncertainty; for region A m For each pixel (i,j) within the bounding box and the downsampling step size r, the regression objective is defined as the distance from (ir,jr) to the four sides of the m-th bounding box, represented by a 4-dimensional vector (wl,ht,wr,hb). ij m The predicted bounding box at (i,j) is represented as:

[0129]

[0130] Wherein, S is a fixed proportion value, the predicted coordinates are the coordinates on the Image rather than the coordinates on the Feature-Map; if a pixel is not contained in any one of the elliptical Gaussian kernel regions satisfying the condition, it is ignored; if it is contained in multiple elliptical Gaussian kernel regions, the training target is defined as the target in the smaller region; the GIoU value of the predicted bounding box and the detection GT bounding box is used as a constraint condition to obtain the following regression loss:

[0131]

[0132] Wherein, B ij ～ represents the predicted bounding box, B m represents the GT bounding box; W ij is the sample weight, used to balance the contribution of each sample to the loss; assuming that (i,j) is located in the A m region of the mth bounding box, the sample weight is:

[0133]

[0134] Wherein, G m (i,j) is the Gaussian response at (i,j), a m is the Gaussian kernel target region of the mth bounding box.

[0135] Embodiment 4:

[0136] This embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be repeated here. Further, in order to better realize the multi-stage video pedestrian multi-target tracking method assisted by re-identification, the following setting mode is particularly adopted:

[0137] 1.8.1) define each detection GT bounding box GTBox i = (x 1i , y 1i , x 2i , y 2i ) in the image;

[0138] 1.8.2) obtain the center (C xi ～ , C yi ～ ) in the HeatMap;

[0139] 1.8.3) after step 1.8.2), the re-identification feature E(C xi ～ , C yi ～ ) is grabbed from the point;

[0140] 1.8.4) After step 1.8.3), it is mapped to a class distribution vector P = {p(k), k ∈ [1, K]};

[0141] 1.8.5) The real class label is represented by one-hot code L i (k), and the re-identification loss is denoted as:

[0142]

[0143] where K is the number of classes, and N is the number of samples in the image frame;

[0144] 1.8.6) The total loss function L in the training process of the MSMOT model is obtained by weighted sum of all Head losses of the detection branch and the re-identification branch. total , and

[0145] L total = W heat L heat +W offset L offset +W reg L reg +W id L id (8).

[0146] Embodiment 5:

[0147] This embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be repeated here. Further, in order to better realize the multi-stage video pedestrian multi-target tracking method assisted by re-identification, the following setting mode is particularly adopted: the step 2) is specifically: after the training of the MSMOT model is completed in step 1), the detection bounding box information of the pedestrian and the 128-dimensional pedestrian re-identification feature are extracted. However, due to the size difference of the targets in the video frame, the number of samples generated by large targets is greater than that generated by small targets. Therefore, after the loss caused by all samples is normalized, the loss contribution of small targets will be ignored, which will damage the detection of small targets. Therefore, according to the confidence score of the detection bounding box, the detection bounding box is divided into three categories, and the detection bounding box interval of the corresponding category is divided by three different confidence thresholds T h , T m , and T l , which are specifically:

[0148] 2.1) First stage association (IoU or ReID): for the detection threshold greater than T hThe detection bounding box is assumed to have little impact on pedestrian detection tasks due to interference such as occlusion, and the extracted re-identification features are of high quality. Therefore, when performing data association tasks, it is believed that only one of the two conditions, IoU or re-identification feature distance, needs to be satisfied to complete the matching between the corresponding detection and trajectory.

[0149] 2.2) Second-stage association (IoUandReID): For detection thresholds less than T h Greater than T m The detection bounding box is assumed to have little impact on pedestrian detection tasks, and the extracted re-identification features are of high quality. Therefore, in order to reduce matching errors, when performing data association tasks, it is assumed that the matching between the corresponding detection and trajectory is completed when both the IoU and re-identification feature distance conditions are met.

[0150] 2.3) Third-stage association (IoU): For detection thresholds less than T m Greater than T l The detection bounding boxes are considered to have a significant impact on pedestrian detection tasks due to interference such as occlusion, and the quality of extracted re-identification features is low. Therefore, to avoid the damage of low-quality re-identification features to the data association task, only IoU is used as a constraint, and the detection threshold is less than T. l It was considered part of the background and discarded.

[0151] In steps 2.1) and 2.3), IoU is calculated between the bounding boxes predicted by the Kalman filter and the detected bounding boxes. The distance between the re-identified features is Euclidean distance, and the Hungarian algorithm is used for bipartite graph matching based on the cost matrix; where T h To define a high-confidence threshold for high-quality detection bounding boxes, T m To define the medium confidence threshold for dividing the quality inspection bounding box, T l The low-confidence threshold is used to distinguish between low confidence and background interference.

[0152] Example 6:

[0153] The application discloses a multi-stage video pedestrian multi-target tracking model assisted by re-identification, and relates to the technical field of video pedestrian multi-target tracking.

[0154] Embodiment 7:

[0155] The embodiment is further optimized on the basis of the above-mentioned embodiments, and the same parts as the previous technical solutions will not be repeated here.

[0156] Embodiment 8:

[0157] The embodiment is further optimized on the basis of the above-mentioned embodiments, and the same parts as the previous technical solutions will not be repeated here.

[0158] The above is only a preferred embodiment of the application, and does not limit the application in any form. Any simple modification or equivalent change based on the technical essence of the application to the above embodiment falls within the protection scope of the application.

Claims

1. A multi-stage video pedestrian multi-target tracking method with re-identification assistance, characterized in that: Includes the following steps: 1) Construct the MSMOT model and use it to detect pedestrian bounding boxes and extract pedestrian re-identification features; wherein, the MSMOT model includes ResNet50 as the backbone network, a multi-layer feature aggregation module, an attention-guided context aggregation module including a context capture module and an attention guidance module, and an IDAUP sub-module. The attention guidance module includes a context attention module and a content attention module. The IDAUP sub-module outputs shallow feature information and deep feature information as inputs to the re-identification branch and the detection branch, respectively. The detection branch uses three parallel heads to estimate the heatmap, the offset of the center point, and the size of the bounding box, respectively. 2) Using a data association algorithm based on Kalman filtering and the Hungarian algorithm, and relying on the detection bounding boxes and re-identification features obtained in step 1), the matching between the detection bounding boxes and the corresponding pedestrian trajectories is completed. Specifically, after step 1), the pedestrian detection bounding box information and 128-dimensional re-identification features are extracted. Based on the confidence score of the detection bounding boxes, the detection bounding boxes are divided into three categories, using three different confidence thresholds T. h T m T l The detection bounding box intervals for the corresponding categories are defined as follows: 2.1) First-stage association: For detection thresholds greater than T h The detection bounding box only needs to satisfy one of the two conditions, IoU or re-identified feature distance, to complete the corresponding detection and trajectory matching; 2.2) Second-stage association: For detection thresholds less than T h Greater than T m When the detection bounding box satisfies both the IoU and re-identified feature distance conditions, the matching between the corresponding detection and the trajectory is completed. 2.3) Third-stage association (IoU): For detection thresholds less than T m Greater than T l The detection bounding box uses only IoU as a constraint, and the detection threshold is less than T. l It was considered part of the background and discarded. In steps 2.1) and 2.3), IoU is calculated between the bounding boxes predicted by the Kalman filter and the detected bounding boxes. The distance between the re-identified features is Euclidean distance, and the Hungarian algorithm is used for bipartite graph matching based on the cost matrix; where T h To define a high-confidence threshold for high-quality detection bounding boxes, T m To define the medium confidence threshold for dividing the quality inspection bounding box, T l The low-confidence threshold is used to distinguish between low confidence and background interference.

2. The multi-stage video pedestrian multi-target tracking method with re-identification assistance according to claim 1, characterized in that: Step 1) includes the following specific steps: 1.1) ResNet50 was selected as the backbone network of the MSMOT model; 1.2) Add a multi-layer feature aggregation module after the Level 3 to Level 5 convolutional blocks in ResNet50; 1.3) Add an attention-guided context aggregation module at the top level of the multi-layer feature aggregation module; 1.4) Use the top-level output features of the multi-layer feature aggregation module as input to the context capture module to search for large-scale context information from different receptive domains; 1.5) Use the output features of the context capture module as the input of the attention guidance module to eliminate the negative impact of information redundancy in the receiving domain; 1.6) The output features of the context capture module and the attention guidance module are aggregated and input into the IDAUP submodule for hierarchical feature extraction; 1.7) The feature information output by the IDAUP submodule for detection is used to estimate the center point position through HeatMap Head, to constrain the quantization error of the center point through OffsetHead, and to regress the bounding box size through Box SizeHead. 1.8) The feature information output by the IDAUP submodule for re-identification is used to complete the re-identification feature extraction through the ReID Head.

3. The multi-stage video pedestrian multi-target tracking method with re-identification assistance according to claim 2, characterized in that: Step 1.7) includes the following steps: 1.7.1) The feature information used for detection is used to estimate the center point location through HeatMap Head: Define the height and width of the image frame as H. image and W image The generated HeatMap has a size of 1×H×W (H=H image / 4, W = W image / 4), the response of the generated HeatMap at the actual location coordinates of the target center is 1, and it decays exponentially with the distance from the center; For each detection bounding box DetBox i =(x 1i ,y 1i ,x 2i ,y 2i ), calculate the coordinates of its center point as C xi =(x 1i +x 2i ) / 2, C yi =(y 1i +y 2i ) / 2, which corresponds to the position (C) on the output feature map. xi ～ C yi ～ ) is ([C xi / 4],[C yi / 4]), where the symbol [] is used to indicate rounding down, and the response of HeatMap at (x, y) is represented as: The loss function used is Focal_Loss: Where N represents the number of targets in the image, δ x 2 and δ y 2 They are (αW / 6)² and (αH / 6)², respectively, where α is a hyperparameter with a value of 0.54; α f and β f The hyperparameters in Focal_Loss are 2 and 4 respectively, M xy ～ HeatMap estimated for the network; M xy This represents the HeatMap generated by formula (1); 1.7.2) The feature information used for detection constrains the quantization error of the center point through OffsetHead: For each GT bounding box GTBox in the image i =(x 1i ,y 1i ,x 2i ,y 2i The quantization error of the GT bounding box due to downsampling is o. i =(C xi ～ C yi ～ )-([C xi / 4],[C yi / 4]), use o i ～ The estimated offset is represented by L1_Loss, which is used to correct the quantization error of the center point using the following formula: in, L offset The center point quantization loss is used to constrain the error caused by downsampling; N is the number of targets in the video frame; i is each target in the video frame; 1.7.3) The feature information used for detection is used to regress the bounding box size through Box SizeHead: The same elliptical Gaussian kernel as used in the HeatMap generation process is employed for bounding box regression, identifying regions A with a Gaussian kernel response greater than 0. m All pixels in region A are considered as samples for regressing the bounding box size; m For each pixel (i,j) within the bounding box and the downsampling step size r, the regression objective is defined as the distance from (ir,jr) to the four sides of the m-th bounding box, represented by a 4-dimensional vector (wl,ht,wr,hb). ij m The predicted bounding box at (i,j) is represented as: in, S is a fixed scale value, and the predicted coordinates are coordinates on the Image rather than on the Feature-Map. If a pixel is not contained in any of the elliptical Gaussian kernel regions that meet the conditions, it is ignored. If it is contained in multiple elliptical Gaussian kernel regions, the training target is specified to be the target in the smaller region. The GIoU value between the predicted bounding box and the ground truth bounding box is used as a constraint, and the following regression loss is obtained: Among them, B ij ～ B represents the predicted bounding box. m Represents the GT bounding box; W ij These are sample weights, used to balance the contribution of each sample to the loss; assuming (i,j) is located in the m-th bounding box A m Within the region, the sample weights are: Among them, G m (i,j) is the Gaussian response at (i,j), a m Let m be the Gaussian kernel target region of the m-th bounding box.

4. The multi-stage video pedestrian multi-target tracking method with re-identification assistance according to claim 3, characterized in that: step 1.8) specifically comprises: 1.8.1) Define each GT bounding box (GTBox) in the image. i =(x 1i ,y 1i ,x 2i ,y 2i ); 1.8.2) Obtain the center of HeatMap as (C xi ～ C yi ～ ); 1.8.3) After step 1.8.2), extract the re-identification feature E(C) from this point. xi ～ C yi ～ ); 1.8.4) After step 1.8.3), map it to a category distribution vector P = {p(k), k ∈ [1, K]}; 1.8.5) Represent the true category label using one-hot encoding. i (k), the re-identification loss is denoted as: Where K is the number of categories and N is the number of samples in the image frame; 1.8.6) The total loss of all Head losses in the detection and re-identification branches is weighted and summed to obtain the overall loss function L during the training of the MSMOT model. total ,and L total =W heat L heat +W offset L offset +W reg L reg +W id L id (8)。 5. A re-identification-assisted multi-stage video pedestrian multi-target tracking model, characterized in that: it is used to implement the re-identification-assisted multi-stage video pedestrian multi-target tracking method as described in any one of claims 1 to 4, comprising a ResNet50 as the backbone network, a multi-layer feature aggregation module for pooling local and global features at each layer of the ResNet50, an attention-guided context aggregation module added at the top layer of the multi-layer feature aggregation module, the output of each layer of the multi-layer feature aggregation module being connected to an IDAUP sub-module, the shallow feature information output by the IDAUP sub-module being used as the input of the re-identification branch, and the deep feature information output by the IDAUP sub-module being used as the input of the detection branch; the detection branch uses three parallel heads to estimate the heatmap, the offset of the center point, and the size of the bounding box, respectively.

6. The multi-stage video pedestrian multi-target tracking model with re-identification assistance according to claim 5, characterized in that: The attention-guided context aggregation module includes a context capture module and an attention-guided module. The context capture module contains multiple layers for merging multi-scale information. The layers are densely connected. Each layer also obtains multiple feature maps in different receptive domains through dilated convolutions with different dilation rates. Deformable convolutions are added to each path to improve the model's adaptability to geometric transformations.

7. The multi-stage video pedestrian multi-target tracking model with re-identification assistance according to claim 6, characterized in that: The attention guidance module includes a context attention module and a content attention module. The context attention module focuses on the semantics between sub-regions of a given feature map, while the content attention module focuses on ensuring spatial information.

Images