A deep learning-based method for multi-target pedestrian tracking

Abstract

This invention provides a deep learning-based method for multi-target pedestrian tracking, relating to the field of computer vision technology. The method includes: S1 acquiring video data, extracting frames from the video data to convert them into image frames, and preprocessing the image frames; S2 using an improved YOLOv5-s model to detect pedestrian targets in the image frames, obtaining the pedestrian targets; S3 using an improved Strong SORT model to track the pedestrian targets in each frame, assigning an ID to each tracked pedestrian, and generating pedestrian tracking results. This invention can accurately locate each pedestrian in a video and track their trajectory.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method for multi-target pedestrian tracking based on deep learning. Background Technology

[0002] Multi-object tracking is a crucial task in computer vision. Its main objective is to detect and assign IDs to multiple specific objects in a video for trajectory tracking, without prior knowledge of the number of objects. Different objects possess different IDs to facilitate subsequent trajectory prediction and precise location. Pedestrian multi-object tracking involves real-time tracking of multiple pedestrian targets in complex environments, and has significant social and technological implications. The demand for accurate and efficient pedestrian multi-object tracking technology is increasing in fields such as urban traffic, security monitoring, intelligent transportation systems, and crowd management.

[0003] Based on their different algorithm structures, multi-target tracking algorithms can be roughly divided into three categories: multi-target tracking algorithms that separate detection and tracking, multi-target tracking algorithms based on joint detection and tracking, and multi-target tracking algorithms based on attention mechanisms.

[0004] This multi-target tracking algorithm, which separates detection and tracking, is built upon a tight integration of target detection and target association, aiming to achieve accurate tracking of multiple targets in complex scenes. First, for each frame of a video sequence, the algorithm analyzes the image using target detection technology to obtain the position information of all targets in the current frame and their corresponding bounding boxes. Targets are then cropped based on the bounding boxes to obtain all targets in the image. Subsequently, the problem transforms into target association. A similarity matrix is ​​constructed using Intersection over Union (IoU) and appearance features. The similarity matrix is ​​then solved using methods such as the Hungarian algorithm and greedy algorithms to achieve optimal matching for target association. SORT (Simple Online and Realtime Tracking) is a simple online real-time tracking method that introduces a Kalman filter for the first time to predict the position of a target in future frames based on its position in the current frame. While SORT has achieved some success in reducing interference from similar targets on the tracking model and can achieve high-speed inference due to its simplicity, it has some limitations in handling fast-moving targets and long-term occlusion problems. DeepSORT improves upon SORT by introducing deep learning feature representations and more powerful target association methods, thereby effectively reducing the number of target identity switching incidents and successfully alleviating the target re-identification problem.

[0005] Multi-object tracking algorithms based on joint detection and tracking achieve simultaneous object detection and feature extraction within a single network by adding a feature extraction branch to the object detection network. JDE integrates object detection and embedding learning into the same network, effectively avoiding redundant computation. The JDE framework offers a significant speed improvement while maintaining the same accuracy, enabling real-time multi-object tracking. FairMOT employs fair learning of detection and re-identification tasks, with two evenly distributed branches predicting pixel-level object scores and appearance features. This fairness between tasks results in high levels of detection and tracking accuracy.

[0006] Attention-based multi-object tracking algorithms apply attention mechanisms, such as the Transformer, which have become very popular in computer vision in recent years, to the field of multi-object tracking. Peize et al. built TransTrack, using a Transformer architecture to generate two sets of bounding boxes from two types of queries (object queries and trajectory queries), and determine the final set of bounding boxes, which is the tracking box for each object, through simple IoU matching. It is exactly the same as tracking via the detection paradigm. Furthermore, it utilizes a Transformer query key mechanism to track based on predictions from previous detection knowledge. Daitao et al. improved computation time by applying a lightweight attention layer to a Transformer model inserted into a pyramid network.

[0007] Chinese invention patent application number 202310529165.7 discloses a multi-target tracking method and system. This method tracks targets using point cloud deep learning, filters the target's position and angle, and uses similarity matching to associate targets in adjacent frames, stabilizing the target output state and improving the target detection rate. Simultaneously, based on the target's score, a high-low threshold algorithm is used to add the position, angle, and velocity information of new targets that meet the high-threshold strategy to the constructed tracking target information table before tracking, reducing false detections. This prior art addresses false detections but is not effective for missed detections. Summary of the Invention

[0008] In view of this, the present invention provides a deep learning-based method for multi-target pedestrian tracking. The main problem it solves is that when pedestrians are occluded by other objects or other pedestrians, pedestrian tracking may be lost or incorrect. The present invention can accurately locate each pedestrian in a video and track their trajectory.

[0009] The technical solution of this invention is implemented as follows:

[0010] This invention provides a deep learning-based method for multi-target pedestrian tracking, comprising:

[0011] S1 acquires video data, extracts frames from the video data to convert them into image frames, and preprocesses the image frames.

[0012] S2 uses the improved YOLOv5-s model to detect pedestrian targets in image frames and obtain pedestrian targets;

[0013] S3 uses an improved Strong SORT model to track pedestrian targets in each frame, assigns an ID to each tracked pedestrian, and generates pedestrian tracking results.

[0014] Based on the above technical solution, preferably, step S2 includes:

[0015] S21 constructs a YOLOv5-s model, including an input layer, a backbone network, a feature fusion network, and a prediction network;

[0016] S22 inputs the image frame into the YOLOv5-s model, and the backbone network extracts features from the image frame to obtain semantic features at different scales;

[0017] S23 inputs semantic features of different scales into a feature fusion network for fusion to obtain fused features;

[0018] S24 converts the fused features into the result of target detection, namely pedestrian targets, based on the prediction network.

[0019] Based on the above technical solution, the preferred network structure of the YOLOv5-s model is as follows:

[0020] The backbone network includes a first extraction module, a second extraction module, and a third extraction module, which extract semantic features at the first scale, the second scale, and the third scale, respectively. The first extraction module consists of three CBS modules and two C3-DCN modules, the second extraction module consists of one CBS module and one C3-DCN module, and the third extraction module consists of one CBS module, one C3-DCN module, one pyramid pooling layer, and one convolutional attention module.

[0021] The feature fusion network consists of four CBS modules, two upsampling modules, four C3-DCN modules, and four fusion modules. The feature fusion network performs split-path fusion of semantic features at different scales and obtains three fused features according to three fusion paths.

[0022] The prediction network consists of three convolutional modules, which detect pedestrian targets by processing three fused features.

[0023] Based on the above technical solution, preferably, the three fusion routes are the first fusion route, the second fusion route, and the third fusion route. The feature fusion network is defined according to the data flow direction as: first CBS module, first upsampling module, first fusion module, first C3-DCN module, second CBS module, second upsampling module, second fusion module, second C3-DCN module, third CBS module, third fusion module, third C3-DCN module, fourth CBS module, fourth fusion module, and fourth C3-DCN module;

[0024] The first fusion route is:

[0025] The semantic features of the third scale are sequentially input into the first CBS module and the first upsampling module. They are then fused with the semantic features of the second scale in the first fusion module. After fusion, they are input into the first C3-DCN module and the second CBS module to obtain the first intermediate feature. The first intermediate feature is then input into the second upsampling module and fused with the semantic features of the first scale in the second fusion module. After fusion, they are input into the second C3-DCN module to obtain the first fused feature.

[0026] The second fusion route is:

[0027] The first fusion feature is input into the third CBS module, where it is fused with the first intermediate feature. The fused feature is then input into the third C3-DCN module to obtain the second fusion feature.

[0028] The third integration route is:

[0029] The second fusion feature is input into the fourth CBS module, where it is fused with the semantic features of the third scale. The fused feature is then input into the fourth C3-DCN module to obtain the third fusion feature.

[0030] Based on the above technical solution, preferably, the semantic features of the third scale are features with added channel attention and spatial attention, wherein:

[0031] The formula for calculating channel attention is:

[0032]

[0033] In the formula, M c (F) represents the semantic features at the third scale with added channel attention, where σ represents the sigmoid function, MLP is a multilayer perceptron, AvgPool is average pooling, MaxPool is max pooling, μ1 and μ0 are the weight coefficients of the multilayer perceptron, and F is the original semantic features at the third scale. This represents the feature obtained after the average pooling operation. This represents the feature obtained after max pooling.

[0034] The formula for calculating spatial attention is:

[0035]

[0036] In the formula, M s (F) represents the semantic features at a third scale with added spatial attention, where σ represents the sigmoid function, AvgPool is average pooling, MaxPool is max pooling, and f 7×7 This indicates that the filter size in the convolution operation is 7×7, and F represents the original semantic features at the third scale. This represents the feature obtained after average pooling operation along the channel direction. This represents the feature obtained through the max pooling operation in the channel direction.

[0037] Based on the above technical solution, preferably, the YOLOv5-s model is a pre-trained model, and the loss function of the YOLOv5-s model during pre-training is:

[0038] L WIoU =rR WIoU L IoU

[0039]

[0040]

[0041] S u =wh+w gt h gt -W j H j

[0042] In the formula, L WIoU Let L be the loss function. IoU To calculate the crossover and union loss, R WIoU The penalty term is r, which is a non-monotonic focusing coefficient, and x and y represent the center coordinates of the predicted bounding box. gt ,y gt W represents the center coordinates of the true bounding box. g H g This represents the width and height of the smallest outer bounding box that contains both the predicted and ground truth boxes. The superscript * indicates separation from the computation graph. W j H j S represents the width and height of the overlapping region between the ground truth bounding box and the predicted bounding box. u Let w represent the area of ​​the union of the predicted bounding box and the ground truth bounding box, and let w and h represent the width and height of the predicted bounding box, respectively. gt ,h gt This represents the width and height of the actual bounding box.

[0043] Based on the above technical solution, preferably, step S3 includes:

[0044] S31 For each pedestrian target, initialize a predicted trajectory based on the Kalman filter. The trajectory includes the pedestrian's position, velocity, and related information.

[0045] S32 uses the Hungarian algorithm to match the predicted trajectory with the pedestrian target in the current image frame to obtain a matching pair. For each matching pair, a unique ID is assigned to the tracked pedestrian target.

[0046] For each tracked pedestrian target, S33 updates the trajectory of each pedestrian using a Kalman filter based on its historical trajectory and pedestrian features in the current image frame, including predicting the pedestrian's position, updating the state estimate, and updating the covariance matrix.

[0047] When the tracking stop condition is met, S34 outputs the pedestrian tracking results of the Strong SORT model.

[0048] Based on the above technical solution, preferably, in step S33, when updating the trajectory of each pedestrian using a Kalman filter, the appearance state of the i-th trajectory at image frame t is updated using a dynamic EMA method.

[0049]

[0050] α=α min +(α max -α min )×s d

[0051] In the formula, It represents the appearance state of the i-th trajectory at image frame t. It is the appearance embedding detected by the current matching pair, and α is the dynamic weight. min and α max It is a set constant, s d It represents the confidence level of the detection box.

[0052] Based on the above technical solution, preferably, in step S33, when updating the trajectory of each pedestrian using a Kalman filter, an adaptive noise update of the tracking state is adopted, and the adaptive noise covariance formula is:

[0053]

[0054] In the formula, It is the adaptive noise covariance, R k It is a pre-defined constant noise covariance, c kThis is the detection confidence score under state k. When there is little noise, the detection confidence score is c. k A higher score leads to Lower.

[0055] Based on the above technical solution, preferably, step S32 includes:

[0056] S321 If the pedestrian target in the current image frame matches the predicted trajectory successfully, a matching pair is formed, and step S33 is executed;

[0057] S322 If the pedestrian target in the current image frame and the predicted trajectory fail to match, a second matching is performed. If the second matching is successful, a matching pair is formed and step S33 is executed. If the second matching fails, the process proceeds to step S323.

[0058] S323 determines whether the second-order matching failure is due to pedestrian target matching failure or predicted trajectory tracking failure. If the second-order matching failure is due to pedestrian target matching failure, proceed to step S324. If the second-order matching failure is due to predicted trajectory tracking failure, proceed to step S325.

[0059] S324 creates a new predicted trajectory for the pedestrian target that failed to match, sets it to unconfirmed, verifies the new predicted trajectory three times, if the verification passes, changes it to confirmed to form a matching pair, and executes step S33; if the verification fails, delete the pedestrian target and the new predicted trajectory.

[0060] S325 For predicted trajectories that fail to match, check if their setting is unconfirmed. If it is unconfirmed, delete it. If it is set to confirmed, set the lifetime of the predicted trajectory and execute step S326.

[0061] S326 examines the predicted trajectories that fail to match within the lifetime. If the match still fails within the lifetime, the predicted trajectory is deleted. If the match is successful within the lifetime, a matching pair is formed, and step S33 is executed.

[0062] The method of the present invention has the following advantages over the prior art:

[0063] (1) This invention first utilizes the improved YOLOv5-s model for pedestrian target detection, which can quickly and accurately detect pedestrian targets in image frames. This helps the system quickly obtain target location information, providing an accurate initial position for subsequent target tracking. Then, the improved Strong SORT model is used to track pedestrian targets in each frame, achieving high-precision target tracking. This model can accurately estimate the target's position, velocity, and scale based on the target's motion pattern and observation information, and assign a unique ID to each tracked pedestrian to generate pedestrian tracking results;

[0064] (2) This invention reduces the number of frames processed by extracting frames from video data, thereby improving the real-time performance of the system. Simultaneously, the improved YOLOv5-s and Strong SORT models both have high processing speeds, enabling them to complete target detection and tracking tasks quickly, meeting the needs of real-time applications. Furthermore, the improved target detection and tracking models exhibit strong robustness, adapting to different scenarios and target variations. This means the system can perform pedestrian target detection and tracking in various complex environments, such as densely populated areas and environments with significant lighting changes.

[0065] (3) This invention utilizes an improved YOLOv5-s model for pedestrian target detection. By introducing the C3-DCN module with deformable convolutional design, the model can better adapt to pedestrian targets at different scales, thereby improving the detection accuracy. Traditional target detection models may have certain difficulties in detecting pedestrians at small or large scales, while the improved model can better adapt to these changes. By introducing the CBAM module, the model can generate attention feature map information in both channel and spatial dimensions, and multiply it with the original input feature map for adaptive feature correction. This allows the model to better focus on important features in the image and suppress unnecessary regional responses, thereby improving the detection accuracy of pedestrian targets;

[0066] (4) To enhance the robustness of tracking, this invention employs adaptive noise in the algorithm. Adaptive noise helps the tracking algorithm model and process uncertainties and noise during target tracking. In practical applications, factors such as target motion and external environment introduce noise and uncertainty. Adaptive noise can automatically adjust the noise model according to the current tracking situation, thereby better adapting to different tracking scenarios and improving the robustness of the tracking algorithm. Adaptive noise can dynamically adjust the weights and variance of state updates based on the current tracking state and observation information. By adaptively adjusting the noise model, the target state can be estimated more accurately, avoiding overfitting or underfitting and improving the accuracy of state updates.

[0067] (5) The matching process of the present invention adopts a secondary matching mechanism, a confirmation and deletion mechanism, and a lifespan mechanism, which can detect the failed matching trajectory in a timely manner and perform secondary verification or deletion, avoiding tracking invalid trajectories for too long, thereby improving the matching accuracy of pedestrian targets and the effect of trajectory tracking, improving the robustness and accuracy of the system, and thus better completing the task of tracking pedestrian targets. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0070] Figure 2 This is a diagram of the YOLOv5-s model network structure according to an embodiment of the present invention;

[0071] Figure 3 This is a flowchart of the matching process in target tracking according to an embodiment of the present invention. Detailed Implementation

[0072] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0073] like Figure 1 As shown, this invention provides a deep learning-based method for multi-target pedestrian tracking, comprising:

[0074] S1 acquires video data, extracts frames from the video data to convert them into image frames, and preprocesses the image frames.

[0075] S2 uses the improved YOLOv5-s model to detect pedestrian targets in image frames and obtain pedestrian targets;

[0076] S3 uses an improved Strong SORT model to track pedestrian targets in each frame, assigns an ID to each tracked pedestrian, and generates pedestrian tracking results.

[0077] Specifically, in one embodiment of the present invention, step S1 includes:

[0078] Acquire video data: Use video capture devices (such as cameras) or read existing video files to acquire video data.

[0079] Frame extraction and conversion to image frames: This involves extracting frames from the video data at a certain frame rate to obtain each frame of the image. Video processing libraries, such as OpenCV, can be used to perform frame extraction and conversion.

[0080] Image frame preprocessing: Preprocessing each frame of the image, including the following steps:

[0081] Image resizing: Adjust the image size to match the model input size to ensure the model meets input requirements.

[0082] Image enhancement: This can enhance aspects such as brightness, contrast, and saturation of an image to improve its quality.

[0083] Image normalization: Normalizing the pixel values ​​of an image so that they fall within a suitable range.

[0084] This embodiment transforms video data into image frame data that the model can process by performing frame extraction and image preprocessing, preparing it for subsequent target detection and tracking. Frame extraction reduces the number of frames that need to be processed, thereby improving the system's processing speed. Preprocessing optimizes the images, reducing noise and interference, and improving the model's detection and tracking performance. By preprocessing the image frames, the input image data can have consistent size and quality, which is beneficial for the stable operation of the model and the reliability of the results.

[0085] Specifically, in one embodiment of the present invention, step S2 includes:

[0086] S21 constructs a YOLOv5-s model, including an input layer, a backbone network, a feature fusion network, and a prediction network;

[0087] S22 inputs the image frame into the YOLOv5-s model, and the backbone network extracts features from the image frame to obtain semantic features at different scales;

[0088] S23 inputs semantic features of different scales into a feature fusion network for fusion to obtain fused features;

[0089] S24 converts the fused features into the result of target detection, namely pedestrian targets, based on the prediction network.

[0090] like Figure 2 As shown, the network structure of the YOLOv5-s model in this embodiment is as follows:

[0091] The backbone network includes a first extraction module, a second extraction module, and a third extraction module, which extract semantic features at the first scale, the second scale, and the third scale, respectively. The first extraction module consists of three CBS modules and two C3-DCN modules, the second extraction module consists of one CBS module and one C3-DCN module, and the third extraction module consists of one CBS module, one C3-DCN module, one pyramid pooling layer, and one convolutional attention module.

[0092] The feature fusion network consists of four CBS modules, two upsampling modules, four C3-DCN modules, and four fusion modules. The feature fusion network performs split-path fusion of semantic features at different scales and obtains three fused features according to three fusion paths.

[0093] The prediction network consists of three convolutional modules, which detect pedestrian targets by processing three fused features.

[0094] The three fusion routes are the first fusion route, the second fusion route, and the third fusion route. The feature fusion network is defined according to the data flow direction as follows: first CBS module, first upsampling module, first fusion module, first C3-DCN module, second CBS module, second upsampling module, second fusion module, second C3-DCN module, third CBS module, third fusion module, third C3-DCN module, fourth CBS module, fourth fusion module, and fourth C3-DCN module.

[0095] The first fusion route is:

[0096] The semantic features of the third scale are sequentially input into the first CBS module and the first upsampling module. They are then fused with the semantic features of the second scale in the first fusion module. After fusion, they are input into the first C3-DCN module and the second CBS module to obtain the first intermediate feature. The first intermediate feature is then input into the second upsampling module and fused with the semantic features of the first scale in the second fusion module. After fusion, they are input into the second C3-DCN module to obtain the first fused feature.

[0097] The second fusion route is:

[0098] The first fusion feature is input into the third CBS module, where it is fused with the first intermediate feature. The fused feature is then input into the third C3-DCN module to obtain the second fusion feature.

[0099] The third integration route is:

[0100] The second fusion feature is input into the fourth CBS module, where it is fused with the semantic features of the third scale. The fused feature is then input into the fourth C3-DCN module to obtain the third fusion feature.

[0101] This embodiment uses the YOLOv5-s model for object detection. The YOLOv5-s model is an end-to-end object detection model. This embodiment improves the YOLOv5-s model in the following ways: 1. A C3-DCN module is designed using deformable convolution to enhance the extraction of pedestrian features at different scales. The previous C3 module is replaced with C3-DCN; 2. A CBAM (Convolutional Attention Module) is added. CBAM can sequentially generate attention feature map information in both channel and spatial dimensions. Then, the two feature map information are multiplied with the original input feature map for adaptive feature correction to generate the final feature map, which focuses on the important features of the image and suppresses unnecessary regional responses.

[0102] Specifically, the CBS module includes one convolutional layer, one BN layer, and one activation layer, with the activation layer using the SiLU activation function. The C3-DCN module includes three DCBS modules, one Bottleneck module, and one Concat layer. The DCBS module replaces the convolutional layer with deformable convolutions; the rest of the structure is the same as the CBS module. The Bottleneck module has two modes: 1. Bottleneck True: First, a 1*1 CBS module, then a 3*3 CBS module, and finally, the input is added to the residual structure. 2. Bottleneck False: First, a 1*1 CBS module, then a 3*3 CBS module, without adding a residual structure. The pyramid pooling layer includes three 5*5 max pooling layers. The convolutional attention module includes two sub-modules: a channel attention sub-module, including a max pooling layer, an average pooling layer, and a shared fully connected layer; and a spatial attention sub-module, including a max pooling layer, an average pooling layer, and a convolutional layer.

[0103] In this embodiment, CBAM focuses on applying channel attention and spatial attention to the feature map output by the backbone network. Therefore, the semantic features at the third scale are represented as follows:

[0104] F∈R C×H×W

[0105] In the formula, F represents the semantic features of the third scale, and C, H, and W represent the number of channels, depth, and width of the feature map, respectively.

[0106] CBAM generates 1D channel attention feature maps M respectively. c and 2D spatial attention feature map M s :

[0107] M c ∈R c×H×W

[0108] M s∈R C×H×W

[0109] This process can be described by the following formula:

[0110]

[0111]

[0112] In the formula, This is element-wise multiplication. During multiplication, attention values ​​are broadcast (copied) accordingly; channel attention values ​​are broadcast along spatial dimensions and vice versa. F” is the final output.

[0113] Each channel of the feature map is treated as a feature detector. Channel features focus on what useful information is in the image. To calculate channel attention features more efficiently, the spatial dimension of the feature map is compressed, and average pooling and max pooling are used to learn the features of the target. The formula for calculating channel attention is:

[0114]

[0115] In the formula, M c (F) represents the semantic features at the third scale with added channel attention, where σ represents the sigmoid function, MLP is a multilayer perceptron, AvgPool is average pooling, MaxPool is max pooling, μ1 and μ0 are the weight coefficients of the multilayer perceptron, and F is the original semantic features at the third scale. This represents the feature obtained after the average pooling operation. This represents the feature obtained after max pooling.

[0116] Unlike channel attention, spatial attention focuses on where the effective information lies in the feature map. To compute spatial attention, average pooling and max pooling are first performed along the channel dimensions, and then the resulting feature maps are concatenated. A convolution operation is then applied to the concatenated feature map to generate the final spatial attention feature map, as shown in the following formula:

[0117]

[0118] In the formula, M s (F) represents the semantic features at a third scale with added spatial attention, where σ represents the sigmoid function, AvgPool is average pooling, MaxPool is max pooling, and f 7×7 This indicates that the filter size in the convolution operation is 7×7, and F represents the original semantic features at the third scale. This represents the feature obtained after average pooling operation along the channel direction. This represents the feature obtained through the max pooling operation in the channel direction.

[0119] The YOLOv5-s model in this embodiment is a pre-trained model. It utilizes the network structure of the YOLOv5-s model to extract features from sample images. The extracted feature maps are then processed by CBAM to output attention feature maps. Simultaneously, the image is divided into small squares and anchor boxes are generated. The labeled predicted boxes are associated with the feature maps. Finally, a loss function is established and end-to-end training is initiated. YOLOv5-s uses CIOU to measure the loss of the bounding boxes; to improve the overall performance of the model, it is replaced with WIoU. The loss function is shown below:

[0120] L WIoU =rR WIoU L IoU

[0121]

[0122]

[0123] S u =wh+w gt h gt -W j H j

[0124] In the formula, L WIoU Let L be the loss function. IoU To calculate the crossover and union loss, R WIoU The penalty term is r, which is a non-monotonic focusing coefficient, and x and y represent the center coordinates of the predicted bounding box. gt ,y gt W represents the center coordinates of the true bounding box. g H g This represents the width and height of the smallest outer bounding box that contains both the predicted and ground truth boxes. The superscript * indicates separation from the computation graph. W j H j S represents the width and height of the overlapping region between the ground truth bounding box and the predicted bounding box. u Let w represent the area of ​​the union of the predicted bounding box and the ground truth bounding box, and let w and h represent the width and height of the predicted bounding box, respectively. gt ,h gt This represents the width and height of the actual bounding box.

[0125] Where r is constructed based on β, and the formula for calculating r is as follows:

[0126]

[0127] In the formula, This is the weighting parameter, β, which is a parameter value calculated from the monotonic focusing coefficient and the moving average of momentum m in the original YOLO model. Its calculation formula is:

[0128]

[0129] In the formula, L IoU * That is, the monotonic focusing coefficient in the original YOLO model. That is, the moving average with momentum m.

[0130] In summary, this embodiment utilizes an improved YOLOv5-s model for pedestrian target detection. By introducing the C3-DCN module with deformable convolutional design, the model can better adapt to pedestrian targets at different scales, thereby improving detection accuracy. Traditional target detection models may face difficulties in detecting pedestrians at small or large scales, while the improved model can better adapt to these variations. By introducing the CBAM module, the model can generate attention feature map information in both channel and spatial dimensions, and multiply it with the original input feature map for adaptive feature correction. This allows the model to better focus on important features in the image and suppress unnecessary region responses, thus improving the accuracy of pedestrian target detection. Overall, the improved YOLOv5-s model in this embodiment can improve detection accuracy and robustness. The model can better adapt to pedestrian targets at different scales and focus on important features in the image, thereby reducing false positives and false negatives. This is highly beneficial for pedestrian target detection tasks, improving the system's performance and reliability in real-world scenarios.

[0131] Specifically, in one embodiment of the present invention, step S3 includes:

[0132] S31 For each pedestrian target, initialize a predicted trajectory based on the Kalman filter. The trajectory includes the pedestrian's position, velocity, and related information.

[0133] S32 uses the Hungarian algorithm to match the predicted trajectory with the pedestrian target in the current image frame to obtain a matching pair. For each matching pair, a unique ID is assigned to the tracked pedestrian target.

[0134] For each tracked pedestrian target, S33 updates the trajectory of each pedestrian using a Kalman filter based on its historical trajectory and pedestrian features in the current image frame, including predicting the pedestrian's position, updating the state estimate, and updating the covariance matrix.

[0135] When the tracking stop condition is met, S34 outputs the pedestrian tracking results of the Strong SORT model.

[0136] Strong SORT is an algorithm model for target tracking. It is an improved version of the Deep SORT algorithm, designed to enhance the accuracy and robustness of target tracking. Strong SORT uses a multi-feature fusion method to improve target tracking performance. It fuses multiple features; in this embodiment, it extracts appearance and motion features to better describe and distinguish different targets. This reduces confusion between targets and improves the accuracy of target identification and tracking.

[0137] This embodiment modifies the Kalman filter in Strong SORT, using width and height instead of the previously used aspect ratio. This modification helps improve the fit between the bounding box width and the object, resulting in a better fit of the generated detection box to the target. To selectively incorporate appearance information into the orbital model only under high-quality conditions, this embodiment proposes a dynamically weighted EMA (Exponential Moving Average) to replace the previous EMA.

[0138] Specifically, in step S33, when updating the trajectory of each pedestrian using a Kalman filter, the appearance state of the i-th trajectory at image frame t is updated using a dynamic EMA method.

[0139]

[0140] α=α min +(α max -α min )×s d

[0141] In the formula, It represents the appearance state of the i-th trajectory at image frame t. It is the appearance embedding detected by the current matching pair. α is a dynamic weight generated dynamically using a formula based on the confidence of the detection box, thus ensuring that only high-quality appearance embeddings are accepted. min and α max It is a set constant, s d It represents the confidence level of the detection box.

[0142] Furthermore, when updating the trajectory of each pedestrian using a Kalman filter, adaptive noise is used to update the tracking state. The formula for the adaptive noise covariance is:

[0143]

[0144] In the formula, It is the adaptive noise covariance, R k It is a pre-defined constant noise covariance, c k This is the detection confidence score under state k. When there is little noise, the detection confidence score is c. kA higher score leads to Lower.

[0145] lower This means that detection will have a higher weight in the state update step, and vice versa. This helps improve the accuracy of updating the state.

[0146] To enhance the robustness of tracking, this embodiment employs adaptive noise in the algorithm. Adaptive noise helps the tracking algorithm model and handle uncertainties and noise during target tracking. In practical applications, factors such as target motion and the external environment introduce noise and uncertainty. Adaptive noise automatically adjusts the noise model based on the current tracking situation, thereby better adapting to different tracking scenarios and improving the robustness of the tracking algorithm. Adaptive noise dynamically adjusts the weights and variance of state updates based on the current tracking state and observation information. By adaptively adjusting the noise model, the target's state can be estimated more accurately, avoiding overfitting or underfitting and improving the accuracy of state updates.

[0147] Specifically, please refer to Figure 3 In one embodiment of the present invention, step S32 includes:

[0148] S321 If the pedestrian target in the current image frame matches the predicted trajectory successfully, a matching pair is formed, and step S33 is executed;

[0149] S322 If the pedestrian target in the current image frame and the predicted trajectory fail to match, a second matching is performed. If the second matching is successful, a matching pair is formed and step S33 is executed. If the second matching fails, the process proceeds to step S323.

[0150] S323 determines whether the second-order matching failure is due to pedestrian target matching failure or predicted trajectory tracking failure. If the second-order matching failure is due to pedestrian target matching failure, proceed to step S324. If the second-order matching failure is due to predicted trajectory tracking failure, proceed to step S325.

[0151] S324 creates a new predicted trajectory for the pedestrian target that failed to match, sets it to unconfirmed, verifies the new predicted trajectory three times, if the verification passes, changes it to confirmed to form a matching pair, and executes step S33; if the verification fails, delete the pedestrian target and the new predicted trajectory.

[0152] S325 For predicted trajectories that fail to match, check if their setting is unconfirmed. If it is unconfirmed, delete it. If it is set to confirmed, set the lifetime of the predicted trajectory and execute step S326.

[0153] S326 examines the predicted trajectories that fail to match within the lifespan. If the match still fails within the lifespan, the predicted trajectory is deleted. If the match is successful within the lifespan, a match pair is formed and step S33 is executed.

[0154] Illustrate with a specific example:

[0155] If the detection box and the already confirmed trajectory are successfully matched, an update is performed, and then prediction-observation-update is repeated. The first case of a failed match is that for the unmatched trajectories and detection boxes, IoU matching is performed again. Those that succeed in IoU matching are updated, and the tracking process of prediction–observation–update continues.

[0156] Failed IoU matching is divided into failed detection box matching and failed tracking of tracks. For the detection boxes that still fail to match after re-matching, the reason for the failed match may be that newly emerged targets did not have their tracks before, or targets with long-term occlusion did not have tracks either. So, a new track is established for it, set to unconfirmed, and it is examined three times. If it is an actual target, it is modified to confirmed, and then prediction-observation-update is performed. For the tracks that still fail to match after re-matching, it may be that the detector missed the target. Check whether it is confirmed. If it is unconfirmed, delete it; otherwise, set its lifespan. If there is no change within the lifespan (>max_age), it is deleted, considering that it has moved out of the camera's view; if within the lifespan (<max_age), it is also examined three times to see if it is a tracked target, and then prediction-observation-update is performed.

[0157] The matching process of this embodiment adopts a secondary matching mechanism, a confirmation and deletion mechanism, and a lifespan mechanism, which can timely detect the trajectories that fail to match and perform secondary verification or deletion, avoid tracking invalid trajectories for too long, improve the matching accuracy of pedestrian targets and the effect of trajectory tracking, and enhance the robustness and accuracy of the system, thus better completing the task of pedestrian target tracking.

[0158] When the target tracking algorithm reaches the stop condition, the pedestrian tracking results of the Strong SORT model are output. The stop condition can be:

[0159] Time cutoff: When the preset time limit is reached, stop target tracking.

[0160] Manual stop: When it is considered that the tracking results meet the requirements, manually stop tracking or trigger a stop signal to stop tracking.

[0161] The output pedestrian tracking results include:

[0162] Target location and bounding box: Outputs the target's location and bounding box information in each frame for subsequent analysis and processing.

[0163] Target trajectory: Outputs the trajectory of the target throughout the tracking process, which can be a series of location points or a continuous bounding box.

[0164] Target ID: Assign a unique ID to each tracked target and associate the ID with the target's location and trajectory for subsequent identification and tracking.

[0165] Finally, the generated tracking results are displayed on the system interface, allowing users to view the tracking results by the name of the image set or the name of the video.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A pedestrian multi-target tracking method based on deep learning, characterized in that, include: S1 acquires video data, extracts frames from the video data to convert them into image frames, and preprocesses the image frames. S2 uses the improved YOLOv5-s model to detect pedestrian targets in image frames and obtain pedestrian targets. The YOLOv5-s model includes an input layer, a backbone network, a feature fusion network, and a prediction network. The feature fusion network adopts three fusion paths, namely the first fusion path, the second fusion path, and the third fusion path. The feature fusion network is defined according to the data flow structure as follows: first CBS module, first upsampling module, first fusion module, first C3-DCN module, second CBS module, second upsampling module, second fusion module, second C3-DCN module, third CBS module, third fusion module, third C3-DCN module, fourth CBS module, fourth fusion module, and fourth C3-DCN module. The first fusion route is as follows: the semantic features of the third scale are sequentially input into the first CBS module and the first upsampling module, fused with the semantic features of the second scale in the first fusion module, and then input into the first C3-DCN module and the second CBS module to obtain the first intermediate feature. The first intermediate feature is input into the second upsampling module, fused with the semantic features of the first scale in the second fusion module, and then input into the second C3-DCN module to obtain the first fused feature. The second fusion route is as follows: input the first fusion feature into the third CBS module, fuse it with the first intermediate feature in the third fusion module, and then input the fused feature into the third C3-DCN module to obtain the second fusion feature; The third fusion route is as follows: input the second fusion feature into the fourth CBS module, fuse it with the semantic features of the third scale in the fourth fusion module, and then input the fused feature into the fourth C3-DCN module to obtain the third fusion feature; S3 uses the improved Strong SORT model to track pedestrian targets in each frame, assigns an ID to each tracked pedestrian, and generates pedestrian tracking results; Specifically, the appearance state of the i-th trajectory at image frame t is updated using a dynamic EMA method. : ； ； In the formula, It represents the appearance state of the i-th trajectory at image frame t. It is the appearance embedding detected by the current matching pair. It is a dynamic weight. and It is a set constant. It is the confidence level of the detection box; The tracking state is updated using adaptive noise. The adaptive noise covariance formula is: ； In the formula, It is adaptive noise covariance. It is a pre-set constant noise covariance. This is the detection confidence score under state k. When there is little noise, the detection confidence score is higher. A higher score leads to Lower; In step S321, if the pedestrian target in the current image frame matches the predicted trajectory successfully, a matching pair is formed, and step S33 is executed. S322 If the pedestrian target in the current image frame fails to match the predicted trajectory, a second matching is performed. If the second matching is successful, a matching pair is formed, and step S33 is executed. If the second matching fails, the process proceeds to step S323. S323 Determine whether the second-order matching failure is due to pedestrian target matching failure or predicted trajectory tracking failure. If the second-order matching failure is due to pedestrian target matching failure, proceed to step S324. If the second-order matching failure is due to predicted trajectory tracking failure, proceed to step S325. S324 Create a new predicted trajectory for the pedestrian target that failed to match, set it to unconfirmed, verify the new predicted trajectory three times. If the verification is successful, change it to confirmed to form a matching pair and execute step S33. If the verification fails, delete the pedestrian target and the new predicted trajectory. S325 For predicted trajectories that fail to match, check if their setting is unconfirmed. If it is unconfirmed, delete it. If it is set to confirmed, set the lifetime of the predicted trajectory and execute step S326. S326 Examine the predicted trajectory that failed to match within the lifetime. If the match still fails within the lifetime, delete the predicted trajectory. If the match is successful within the lifetime, a matching pair is formed, and step S33 is executed.

2. The pedestrian multi-target tracking method based on deep learning as described in claim 1, characterized in that, Step S2 includes: S21 Construct the YOLOv5-s model, including the input layer, backbone network, feature fusion network, and prediction network; S22 inputs the image frame into the YOLOv5-s model, and the backbone network extracts features from the image frame to obtain semantic features at different scales; S23 inputs semantic features of different scales into a feature fusion network for fusion to obtain fused features; S24 The fused features are converted into the result of target detection, i.e., pedestrian targets, based on the prediction network.

3. A deep learning-based multi-target pedestrian tracking method as described in claim 2, characterized in that, The network structure of the YOLOv5-s model is as follows: The backbone network includes a first extraction module, a second extraction module, and a third extraction module, which extract semantic features at the first scale, the second scale, and the third scale, respectively. The first extraction module consists of three CBS modules and two C3-DCN modules, the second extraction module consists of one CBS module and one C3-DCN module, and the third extraction module consists of one CBS module, one C3-DCN module, one pyramid pooling layer, and one convolutional attention module. The feature fusion network consists of four CBS modules, two upsampling modules, four C3-DCN modules, and four fusion modules. The feature fusion network performs split-path fusion of semantic features at different scales and obtains three fused features according to three fusion paths. The prediction network consists of three convolutional modules, which detect pedestrian targets by processing three fused features.

4. The pedestrian multi-target tracking method based on deep learning as described in claim 1, characterized in that, The semantic features at the third scale are those with added channel attention and spatial attention, where: The formula for calculating channel attention is: ； In the formula, To add a third scale of semantic features to channel attention, This represents the sigmoid function, and MLP stands for Multilayer Perceptron. For average pooling, For max pooling, and These are the weight coefficients of the multilayer perceptron, and F represents the original semantic features at the third scale. This represents the feature obtained after the average pooling operation. This represents the feature obtained after max pooling. The formula for calculating spatial attention is: ； In the formula, To add a third scale of semantic features to spatial attention, This represents the sigmoid function. For average pooling, For max pooling, This indicates that the filter size in the convolution operation is 7×7, and F represents the original semantic features at the third scale. This represents the feature obtained after average pooling operation along the channel direction. This represents the feature obtained through the max pooling operation in the channel direction.

5. The pedestrian multi-target tracking method based on deep learning as described in claim 2, characterized in that, The YOLOv5-s model is a pre-trained model. The loss function of the YOLOv5-s model during pre-training is: ； ； ； ； In the formula, For loss function, To compare the losses, As a penalty item, The non-monotonic focusing coefficient, Indicates the center coordinates of the prediction box. Represents the coordinates of the center of the true bounding box. This represents the width and height of the smallest bounding box that contains both the predicted and ground truth boxes. (Superscript) This indicates separation from the computation graph. This represents the width and height of the overlapping area between the ground truth bounding box and the predicted bounding box. This represents the area of ​​the union of the predicted bounding box and the ground truth bounding box. This indicates the width and height of the prediction box. This represents the width and height of the actual bounding box.

Images