Pedestrian tracking method based on improved YOLOv8 and Hybrid SORT
By improving the YOLOv8 model and HybridSORT algorithm, combining the details enhancement convolution layer and feature selection layer, the problems of YOLOv8's pedestrian detection accuracy decrease and HybridSORT's dependence on high-precision detection in complex environments are solved, and the high accuracy of pedestrian detection and tracking is achieved and stability.
Patent Information
- Application Number
- CN202411271571.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-11-29
AI Technical Summary
YOLOv8's pedestrian detection accuracy decreases in complex environments. The HybridSORT algorithm relies on high-precision object detection results. The ReID model fails to fully extract detailed features, affecting the accuracy of pedestrian tracking.
Improved YOLOv8 model through weighted cross entropy loss and focus loss, combined with HybridSORT algorithm and improved ReID model, adopts detailed enhancement convolution layer and feature selection layer, combined with Kalman filtering, Hungarian algorithm and strong and weak clue modeling, improves pedestrian detection and Track performance.
It significantly improves the accuracy and robustness of pedestrian detection in complex contexts, optimizes the extraction of pedestrian appearance features, ensures the stability and reliability of target tracking in complex environments, and reduces the false detection and missed detection rates.
Smart Images

Figure CN119048565A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and artificial intelligence, and in particular to a pedestrian tracking method based on improved YOLOv8 and HybridSORT. Background Art
[0002] Pedestrian target detection and tracking technology has important application value in the fields of intelligent monitoring, autonomous driving, public safety, etc. At present, target detection algorithms based on deep learning have become mainstream. Among them, the YOLO (You Only LookOnce) series has been widely used due to its end-to-end detection method and efficient real-time processing capabilities. As a widely recognized version of the YOLO series, YOLOv8 has significantly improved detection accuracy and speed, and performs well in many application scenarios.
[0003] In the pedestrian target detection task, YOLOv8 can handle complex problems in multi-target scenes through its improved feature extraction and prediction mechanism. However, in environments with dense crowds, complex backgrounds, or large light changes, YOLOv8 still faces challenges, such as pedestrian occlusion, background fusion, and scale changes, which may lead to a decrease in detection accuracy and increase missed detections and false detections. In addition, if the model fails to fully capture the detailed features of the target, it will affect the subsequent tracking effect.
[0004] In multi-target tracking tasks, the HybridSORT algorithm can effectively integrate detection results and trajectory predictions by combining Kalman filters, Hungarian algorithms, and weak clue modeling strategies, showing good tracking performance. However, the HybridSORT algorithm still relies on high-precision target detection results in practical applications, and its performance is limited to a certain extent by the feature extraction capabilities of the ReID model. Although existing ReID models have made some progress in feature extraction, they often fail to fully extract detailed features in complex environments, especially when pedestrians are wearing similar clothes or the background is complex, which affects the accuracy of target matching and tracking.
[0005] Therefore, it is very necessary to develop a pedestrian tracking method based on improved YOLOv8 and HybridSORT to improve the accuracy and stability of pedestrian target detection and tracking. Summary of the invention
[0006] The purpose of the present invention is to provide a pedestrian tracking method based on improved YOLOv8 and HybridSORT to solve the above defects.
[0007] In order to achieve the above object, the present invention provides the following technical solutions:
[0008] A pedestrian tracking method based on improved YOLOv8 and HybridSORT includes the following steps:
[0009] S100, input the video sequence, normalize each frame of the image, convert it into a size and format that meets the input requirements of the YOLOv8 model, and input it into the detector;
[0010] S200, using an improved YOLOv8 model for target detection, wherein the YOLOv8 model consists of four layers: Input, Backbone, Neck, and Head, to ensure efficient detection of pedestrian targets;
[0011] S300, after completing the target detection, cropping the region containing the target from the original image, inputting the obtained region into the pre-trained improved ReID model for learning, and outputting a feature vector integrating the detailed appearance features of the pedestrian;
[0012] S400, performing target tracking based on the HybridSORT algorithm, wherein the HybridSORT algorithm combines appearance features extracted by the ReID model, strong clue modeling, weak clue modeling, Kalman filtering and the Hungarian algorithm to achieve robust and stable pedestrian target tracking in complex environments;
[0013] S500: Output the detection and tracking results in each frame of the image, including the location, identity information and trajectory of the pedestrian target.
[0014] Preferably, in step S200, the YOLOv8 model includes four layers: Input, Backbone, Neck and Head, specifically including:
[0015] S201, at the Input layer: perform data enhancement operations, including Mosaic enhancement, Mixup, random perturbation, and color perturbation HSV strategies to enhance the generalization ability of the model;
[0016] S202, Backbone network: Use the improved CSPDarknet53 network to extract features from preprocessed images, generate feature maps of different scales, and effectively reduce the amount of calculation and improve feature expression capabilities through the Cross Stage Partial (CSP) structure;
[0017] S203, Neck feature fusion network: Introducing PAN-FPN as a feature pyramid, through bidirectional path fusion, the feature map contains richer context and semantic information, enhancing the model's detection ability for targets of different scales, especially helping to identify and accurately locate small-sized pedestrians;
[0018] S204, Head detector network: Generate candidate boxes through an anchor-free mechanism, use non-maximum suppression (NM) to remove duplicate candidate boxes, and use an improved loss function to optimize the model weights during training to make it more suitable for pedestrian detection, and finally output the target's category confidence and the precise regression results of the bounding box.
[0019] Preferably, in step S204, the improvement or optimization strategy of the loss function of the YOLOv8 model specifically includes:
[0020] Introducing weighted cross entropy loss to give higher weights to pedestrian categories and effectively deal with the category imbalance problem;
[0021] Introducing focal loss to guide the model to pay more attention to pedestrian targets that are difficult to detect;
[0022] Combining weighted cross entropy loss with focal loss, a new classification loss function is defined to better deal with the problem of weight distribution of category imbalance and sample difficulty.
[0023] Preferably, in step S300, the improved ReID model specifically includes:
[0024] S301, cropping the bounding box of the pedestrian target output by YOLOv8 and standardizing the image to preprocess the subsequent network input;
[0025] S302, input the processed image into the ResNet50 basic convolutional network to extract the initial pedestrian appearance features;
[0026] S303, introduce a detail enhancement convolution layer, which uses multi-scale convolution, batch normalization, ReLU activation and residual connection to improve the model's sensitivity to subtle differences and enhance the ability to capture detailed features of pedestrian images;
[0027] S304: Introduce a feature selection layer to select the feature area most relevant to the pedestrian identity based on the self-attention mechanism, suppress background information and irrelevant content, highlight key features, reduce redundancy, and improve the quality of feature expression;
[0028] S305, averaging the weighted feature map in the spatial dimension through global average pooling to generate a fixed-length feature vector representation;
[0029] S306, inputting the generated feature vector into a fully connected layer for further processing to optimize the expression of the feature vector;
[0030] S307, using the EMA strategy to update the appearance feature vector to reduce feature fluctuations caused by short-term changes;
[0031] S308, finally outputting the pedestrian appearance feature vector after EMA smoothing, which is used for pedestrian matching and tracking in the HybridSort algorithm.
[0032] Preferably, in step S400, the pedestrian target tracking process of the HybridSORT algorithm is as follows:
[0033] S401, input the pedestrian target detection result obtained by the YOLOv8 model of the current frame and the appearance feature vector extracted by the ReID model.
[0034] S402, using the Kalman filter to predict the state of the tracker, estimate its position and speed in the current frame, and calculate the covariance matrix;
[0035] S403, based on the target category confidence score output by YOLOv8, the confidence threshold set by HybridSORT is compared to divide the detection results into high-confidence objects and low-confidence objects, specifically including:
[0036] If the result is greater than the confidence threshold, the sequential operations of steps S404-S406 are directly performed to perform strong clue modeling and data association on the high confidence object;
[0037] If the result is less than or equal to the confidence threshold, jump to step S407 and perform sequential operations of steps S407-S409 to perform weak clue modeling and data association for low confidence objects through TCM and HMIoU.
[0038] S404, calculating the standard IoU between the predicted position and the current frame detection result;
[0039] S405, normalize and vectorize the obtained IoU value, the motion state information predicted by the Kalman filter, and the appearance feature, and splice the three features into a comprehensive feature vector according to different weights:
[0040] S406, calculating the Euclidean distance between the current frame detection result and the comprehensive feature vector to construct a comprehensive similarity matrix of high-confidence objects, and then performing step S410;
[0041] S407, calculate the confidence score for each tracker track through track confidence modeling TCM. For tracks below the track confidence threshold, TCM will increase the diagonal elements of the covariance matrix to adapt to the instability of the target state; at the same time, use Height Modulated IoU to introduce target height information, adjust IoU calculation, and improve matching accuracy when the target height changes;
[0042] S408, normalizing and vectorizing the obtained HMIoU value, the motion state information after TCM adjustment, and the appearance feature, and splicing the three features into a comprehensive feature vector;
[0043] S409, calculating the Euclidean distance between the current frame detection result and the comprehensive feature vector to construct a comprehensive similarity matrix of the low-confidence object, and then performing step S410;
[0044] S410, normalizing the comprehensive similarity matrices of the high-confidence and low-confidence objects to the same range and merging them into a global similarity matrix to obtain a global similarity score between each detected object and the tracker;
[0045] S411, inputting the fused global similarity matrix into the Hungarian algorithm to perform Hungarian matching to determine the best matching combination between the detection result and the tracker;
[0046] After the above matching process is completed, for the successfully matched detection results and trackers, the operation of step S412 is performed; for the unmatched detection results, the operation of step S413 is performed; for the unmatched trackers, the operations of steps S414-416 are performed, including:
[0047] S412: If the tracker and the detection result are successfully matched, the Kalman filter is used to update the state of each tracker, and the next state prediction is performed, and then the process jumps to step S417;
[0048] S413, if the detection result of the current frame does not find an existing tracker that matches it, initialize a new tracker for it, and then jump to step S417;
[0049] S414, if a tracker fails to successfully match the detection result in the current frame, it is marked as lost, and the target position, appearance features and motion state information successfully detected by the tracker for the last time are saved, and the number of lost frames is recorded;
[0050] S415, in subsequent frames, if the new detection result is similar to the last detection result of the lost target, then try to re-associate, and if the match is successful, update the tracker state;
[0051] S416: If the new detection target is still not successfully matched and the number of lost frames exceeds the predetermined threshold, the tracker is completely deleted from the tracking list, and then the operation continues to step S417;
[0052] S417: Update the tracker list and repeat the above operation when the next frame detection result is input.
[0053] The beneficial effects of the present invention are:
[0054] (1) Improvement of pedestrian detection performance in complex backgrounds: Based on the improved YOLOv8 and HybridSORT pedestrian tracking methods, this paper introduces a combination of weighted cross entropy loss and focal loss in the YOLOv8 model, which greatly improves the detection performance of pedestrians in complex backgrounds. The weighted cross entropy loss increases the model's attention to the pedestrian category, while the focal loss focuses on targets that are difficult to detect. This improvement significantly enhances the detection accuracy and robustness of the model in complex environments, enabling it to effectively cope with challenges such as complex backgrounds and pedestrian occlusion.
[0055] (2) Optimization of the quality of pedestrian appearance feature extraction: This paper introduces detail enhancement convolutional layers and feature selection layers into the ReID model, which greatly improves the effect of pedestrian appearance feature extraction. The detail enhancement convolutional layer enhances the sensitivity to pedestrian image details by capturing subtle differences in the image, while the feature selection layer reduces the interference of redundant features by suppressing background information and irrelevant content. This improvement enables the ReID model to accurately extract and express pedestrian appearance features in complex environments, such as multiple occlusions or rapid pedestrian movement.
[0056] (3) Improvement of the overall performance of the detection and tracking system: The present invention combines the improved YOLOv8 with the HybridSORT algorithm, making full use of the advantages of YOLOv8 in multi-scale feature fusion and the robustness of HybridSORT in multi-strategy data association. This not only enables the system to accurately detect pedestrian targets in complex environments, but also ensures continuous tracking of targets through strong and weak clue modeling and lost target recovery mechanisms, greatly improving the reliability and stability of the overall system and making up for the shortcomings of the existing technology in pedestrian detection and tracking performance in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 : Overall flow chart of the method of the present invention;
[0058] Figure 2 : Schematic diagram of the process of extracting pedestrian appearance features using the improved ReID model and EMA in the present invention;
[0059] Figure 3 : Schematic diagram of the process of the HybridSORT algorithm in the present invention. DETAILED DESCRIPTION
[0060] The present invention is further described below in conjunction with the embodiments. It should be noted that these are merely examples and illustrations of the concept of the present invention. Those skilled in the art may make various modifications or additions to the described specific embodiments or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the claims, they should be deemed to fall within the protection scope of the present invention.
[0061] Embodiment 1:
[0062] See also Figure 1 As shown in the figure, the pedestrian tracking method based on improved YOLOv8 and HybridSORT includes the following steps:
[0063] S100, input the video sequence, normalize each frame of the image, convert it into a size and format that meets the input requirements of the YOLOv8 model, and input it into the detector;
[0064] Specifically, the YOLOv8 model requires that the input image size is fixed to 640x640 pixels, the pixel values are normalized to between 0 and 1, and the channel order is BGR float32 type image.
[0065] S200, use the improved YOLOv8 model for target detection.
[0066] The YOLOv8 model consists of four levels: input, backbone network, feature fusion network, and detector network. Specifically, it performs data enhancement at the input level, extracts multi-scale feature maps at the backbone level, performs multi-scale feature fusion and enhancement at the neck level, and predicts target category confidence and target bounding box precise regression at the head level.
[0067] Step S200 is as follows:
[0068] S201, Input input, including data enhancement strategies such as Mosaic, Mixup, random perturbation, and HSV enhancement, to improve the generalization ability and robustness of the model;
[0069] S202, Backbone network, uses the improved CSPDarknet53 network to extract features from preprocessed images and output feature maps of different scales. The CSPDarknet53 network effectively reduces the amount of calculation and improves the feature expression capability through the Cross Stage Partial (CSP) structure;
[0070] S203, Neck feature fusion network, introduces PAN-FPN as its feature pyramid network. Through the fusion of bidirectional paths, the feature map contains richer contextual information and semantic information, further enhancing the model's detection ability for targets of different scales, which is helpful for small-sized pedestrian recognition and precise positioning of targets;
[0071] S204, Head detector network, generates candidate boxes through the anchor-free mechanism, and uses NMS to remove duplicate candidate boxes to obtain the final detection results, including the category confidence information of each target and the accurate regression of the bounding box (including the center point coordinates (x, y) and width and height (ω, h) of the target), which will be used for subsequent pedestrian tracking tasks. During the training process, the model compares the detection results with the true labels, and guides the model to perform back propagation and update the model weights by calculating various loss functions (including classification loss, regression loss and confidence loss), gradually improving the detection performance of the model.
[0072] In step S204, in order to further improve the model's detection performance for pedestrians in complex backgrounds, the loss function of YOLOv8 is improved or optimized, and weighted cross entropy loss is introduced to effectively deal with the problem of category imbalance; focal loss is introduced to guide the model to pay more attention to pedestrian targets that are difficult to detect; a new classification loss is defined by combining weighted cross entropy loss and focal loss to better deal with category imbalance and the problem of weight distribution of difficult and easy samples. The overall improvement or optimization strategy specifically includes:
[0073] YOLOv8's original loss function L YOLOv8 By the classification loss L class , target confidence loss L conf and bounding box regression loss Composition, defined as: L YOLOv8 =λ loc L loc +λ conf L conf +λ class L class , where λ loc , conf , class are the weight parameters corresponding to each type of loss.
[0074] The optimization part is for L class Then, the weighted cross entropy loss L WCE and the focal loss L FL Combined, define a new classification loss function: In the formula, the weight coefficient σ is a balancing factor, which is used to adjust the relative importance of the two loss functions. The new classification loss effectively ensures that the model can better learn the characteristics of pedestrians in complex backgrounds during training. Specifically:
[0075] The weighted cross entropy loss L WCE By giving a higher weight to the pedestrian category, the problem of category imbalance can be effectively dealt with and the accuracy of detection can be improved. Specifically, the total number of target categories is set to C, where the weight of category c is ω c , the true label of the sample is y c , the predicted probability is Then the expression of the weighted cross entropy loss function is: Among them, ω c Set it according to the proportion of categories in the actual video, such as:
[0076] The focus loss L FL By reducing the weight of easy-to-classify samples and enhancing the influence of difficult-to-classify samples, the model is guided to pay more attention to those pedestrian targets that are difficult to detect. The formula is as follows: Among them, σ i is a balancing factor used to adjust the weights of positive and negative samples; γ is a modulation factor used to control the weights of easy-to-classify samples, usually set to 2.
[0077] The final total loss function is defined as The improved loss function introduces weighted cross entropy loss and focal loss to better handle the problems of class imbalance and weight distribution of difficult and easy samples. Using the optimized loss function for back propagation during model training and adjusting hyperparameters through actual input can significantly improve the pedestrian detection performance of the YOLOv8 model in complex backgrounds.
[0078] S300, after completing the target detection, the region (patch) containing the target is cropped from the original image, the obtained region (patch) is input into the pre-trained improved ReID model for learning, and a feature vector integrating the detailed appearance features of the pedestrian is output.
[0079] In the process of pedestrian detection and tracking, the ReID model is used to enhance the matching ability of pedestrian features. The present invention uses a method combining an improved ReID model with an exponential moving average (EMA) to extract appearance features, generating a high-dimensional feature vector that integrates the appearance features of pedestrians, which is used to improve the matching accuracy under background changes or posture changes.
[0080] The improved ReID model, see Figure 2 As shown, specifically including:
[0081] S301, perform data preprocessing operations such as target bounding box cropping and image normalization on the image output by YOLOv8 to prepare for subsequent network input;
[0082] S302, input the processed image into the basic convolutional network, use ResNet50 as the basic network of the ReID model, and extract the initial pedestrian appearance feature representation;
[0083] S303, introduce the detail enhancement convolution layer, and extract more detailed features of pedestrians (such as subtle clothing differences and body postures) through multi-scale convolution and residual connection, so as to improve the expressiveness of the features and the robustness of the model.
[0084] Specifically, the detail enhancement convolutional layer structure is designed as follows:
[0085] Input: accept the feature map output by ResNet50;
[0086] First convolution layer: First, a 3×3 convolution layer is applied to further extract fine-grained features, and batch normalization is performed on them, and then nonlinear mapping is introduced through the ReLU activation function;
[0087] Second convolution layer: Another 5×5 convolution layer is applied, combined with batch normalization and ReLU activation function to further enhance the network's ability to capture pedestrian features.
[0088] Residual connection: The output features of the first layer of convolution are directly added to the features of the second layer of convolution output to form a skip connection. The formula is Output = Conv(x) + x, where x represents the output features of the 3×3 convolution layer, Conv(x) represents the output features of the second layer of 5×5 convolution, and Output represents the final output after the residual connection is added;
[0089] Output: Pass the output features to the feature selection layer.
[0090] S304. Add a feature selection layer to the ReID model. Through the self-attention mechanism, select the feature areas most relevant to each pedestrian identity from the feature map, and perform weighted amplification on these areas to highlight key visual information, suppress background or irrelevant content, reduce redundant information, and improve the quality of feature expression.
[0091] Specifically, the feature selection layer structure is designed as follows:
[0092] Input: Receive the feature map output from the detail enhancement convolution layer, which contains the preliminary feature representation of pedestrians, such as posture, clothing, accessories, etc.
[0093] Self-attention mechanism: Calculate the attention weight of each position. Generate an attention map through a 1x1 convolution layer, which is used to represent the importance of each feature position in distinguishing pedestrian identities, and then use the Softmax function to normalize the weights to ensure that the model can handle different features in a balanced manner, especially in diverse pedestrian images;
[0094] Feature weighting: The calculated attention weight is multiplied by the corresponding feature position to amplify features that are strongly related to the pedestrian's identity (such as unique clothing, accessories, or posture) and suppress background or irrelevant information to generate a weighted feature map;
[0095] Output: Output the weighted feature map after feature selection.
[0096] S305, averaging the weighted feature map in the spatial dimension through global average pooling to generate a fixed-length feature vector representation;
[0097] S306, inputting the generated feature vector into a fully connected layer for further processing to optimize the expression of the feature vector;
[0098] S307. Use the EMA strategy to update the appearance feature vector. EMA is applied in the feature update stage after the appearance feature extraction. Its main function is to smooth and update the appearance features of the target and reduce the feature fluctuations caused by short-term changes (such as illumination changes, angle changes, partial occlusion, etc.). This feature update method is particularly effective when processing long-term tracking tasks. It can ensure the efficient tracking performance of the system in complex environments and enhance the stability and generalization ability of the model.
[0099] Specifically, the EMA application steps include:
[0100] Input: appearance feature vector f extracted from the current frame current ;
[0101] Update: Update the appearance features through EMA, the formula is: f updated =μ·f current +(1-μ)·f previous , where f previous represents the historical feature vector, μ is the smoothing factor of EMA, and represents the weight distribution of the current frame features and historical features;
[0102] Output: Get the updated feature vector f updated , and save it as the new appearance feature of the target for subsequent tracking.
[0103] S308, finally outputting the pedestrian appearance feature vector after EMA smoothing, which is used for pedestrian matching and tracking in the HybridSort algorithm.
[0104] S400, after completing target detection and appearance feature extraction, enter the tracking phase and use the HybridSORT algorithm for tracking.
[0105] The HybridSORT algorithm combines the appearance features extracted by the improved ReID model, strong clue modeling, weak clue modeling, Kalman filter algorithm and Hungarian algorithm. It predicts and updates the state of pedestrian targets through the Kalman filter, associates data through strong and weak clue modeling, and uses the Hungarian algorithm for data matching, thereby achieving robust and stable pedestrian target tracking in complex environments.
[0106] See also Figure 3 As shown in Figure 2, the pedestrian target tracking process is as follows:
[0107] S401, inputting the pedestrian target detection result obtained by the YOLOv8 model of the current frame and the appearance feature vector extracted by the ReID model;
[0108] S402, predicting the tracker state in the previous frame using a Kalman filter, estimating the position and velocity information of the tracker in the current frame, and calculating the covariance matrix;
[0109] S403, based on the target category confidence score output by the YOLOv8 algorithm and the confidence threshold Conf set by the HybridSORT algorithm threshold Compare and classify high-confidence objects and low-confidence objects.
[0110] Specifically, the threshold selection method for distinguishing high and low confidence objects is:
[0111] High confidence threshold: Objects with high confidence usually mean more reliable detection results. The association strategy for such objects can be more direct, usually relying on conventional methods such as appearance features and position prediction for association. Common confidence thresholds range from 0.5 to 0.7, depending on the requirements of the detection scenario.
[0112] Low confidence threshold: Objects with low confidence may have increased uncertainty due to occlusion, complex background, small target, etc. At this time, the use of conventional association methods may lead to mis-association or missed association. Therefore, a more conservative or complex strategy is used for low-confidence objects to reduce errors. Specifically, the discrimination method for low-confidence objects can be, for example, if the threshold of high-confidence objects is 0.7, then the threshold of low-confidence objects may be between 0.3 and 0.7, and objects below 0.3 will be directly discarded.
[0113] In the data association stage, the HybridSORT algorithm will adopt different association strategies according to the different target confidences. That is, for high-confidence objects, the sequential operations of steps S404-S406 are directly performed, and strong clue modeling and data association are performed through information such as IoU; for low-confidence objects, it jumps to step S407 and performs sequential operations of steps S407-S409, and weak clue modeling and data association are performed through TCM and HMIoU.
[0114] Specifically, the data association process includes:
[0115] S404, calculating the standard IoU between the predicted position and the current frame detection result;
[0116] S405: normalize and vectorize the obtained IoU value, the motion state information predicted by the Kalman filter, and the appearance feature, and splice the three features into a comprehensive feature vector V according to different weights. high , the formula is:
[0117] V high =α h ·V IoU +β h ·V motion +γ h ·V appearance , where V IoU is the vectorized representation of the IoU value, V motion is the motion state vector predicted by the Kalman filter, V appearance is the vector representation of the appearance feature, α h , β h and γ h are the weights of the corresponding features. These three weights can be dynamically adjusted according to the needs of the target detection scenario to ensure that the contribution of different features in matching is reasonable;
[0118] S406, calculating the Euclidean distance between the current frame detection result and the comprehensive feature vector to construct a comprehensive similarity matrix of high-confidence objects, and then performing step S410;
[0119] S407: Calculate a confidence score for each tracker track through TCM, which helps to evaluate the reliability of the track during target tracking, especially when dealing with long-term tracking. threshold For the trajectory, TCM will optimize the prediction results of the Kalman filter by increasing the diagonal elements of the prediction covariance matrix, which means that when predicting the future state of the target, greater uncertainty will be introduced to adapt to the instability of the target state;
[0120] At the same time, HMIoU is used to introduce target height information and adjust the IoU calculation. HMIoU can improve the association accuracy when dealing with targets with large scale changes, thereby improving the matching accuracy when the target height changes.
[0121] S408: normalize and vectorize the obtained HMIoU value, the motion state information after TCM adjustment, and the appearance feature, and concatenate the three features into a comprehensive feature vector V low , the formula is:
[0122] V low =α l ·V HMIoU +β l ·V TCM-motion +γ l ·V appearance , where V HMIoU is the vectorized representation of the HMIoU value, V TCM-motion is the motion state vector predicted by the Kalman filter, V appearance is the vector representation of the appearance feature, α l , β l and γ l are the weights of the corresponding features. These three weights can be dynamically adjusted according to the needs of the target detection scenario to ensure that the contribution of different features in matching is reasonable;
[0123] S409, calculating the Euclidean distance between the current frame detection result and the comprehensive feature vector to construct a comprehensive similarity matrix of the low-confidence object, and then performing step S410;
[0124] S410: Normalize the comprehensive similarity matrices of high-confidence objects and low-confidence objects to the same range and merge them into a global similarity matrix V global , get the global similarity score between each detected object and the tracker, the formula is:
[0125] V global =w high ·V high +w low ·V low , where the corresponding weights w of the high and low confidence similarity matrices are high 、w low It can be dynamically adjusted according to the confidence of the detected object or the scene requirements.
[0126] S411, input the fused global similarity matrix into the Hungarian algorithm for matching to determine the best matching combination between the detection result and the tracker. If a detection result has a high similarity with a tracker, it is considered that they are the same target. The Hungarian algorithm maximizes the overall similarity score to determine the best matching result and associate the detection result with the corresponding tracker.
[0127] After the above matching process is completed, for the successfully matched detection results and trackers, the operation of step S412 is performed; for the unmatched detection results, the operation of step S413 is performed; for the unmatched trackers, the operations of steps S414-416 are performed, including:
[0128] S412: If the tracker successfully matches the detection result, the Kalman filter is used to update the state of each tracker, including the position, velocity information and covariance matrix, and then the process jumps to step S417;
[0129] S413, if the detection result of the current frame does not find an existing tracker that matches it, initialize a new tracker for it, and then jump to step S417;
[0130] S414. If a tracker fails to successfully match the detection result in the current frame, it will be marked as lost. However, the system will save the key information of the last successful match of the track and record the number of lost frames lost_frames. The retained information specifically includes:
[0131] Target location information: save the last successfully detected location coordinates;
[0132] Appearance feature information: retain the last extracted appearance feature vector for subsequent matching;
[0133] Motion state information: records the target’s motion state, such as speed and direction, to predict the target’s potential location.
[0134] S415. In subsequent frames, if the retained information of the tracker marked as lost is similar to a certain detection result, the system will try to re-associate it with the detection result. If the match is successful, the state of the tracker is updated;
[0135] S416: If the new detection target is still not successfully matched and the number of lost frames exceeds the preset threshold max_lost_frames, the tracker is completely deleted from the tracking list, and then the operation continues to step S417.
[0136] S417: Update the tracker list and repeat the above operation when the next frame detection result is input.
[0137] S500: Output the detection and tracking results of each frame of image, including the location, identity information and trajectory of the pedestrian target.
[0138] (1) The present invention is based on an improved pedestrian tracking method based on YOLOv8 and HybridSORT. By introducing a combination of weighted cross entropy loss and focal loss into the YOLOv8 model, the detection performance of pedestrian targets in complex backgrounds is greatly improved. The weighted cross entropy loss enhances the focus on pedestrian categories, while the focal loss focuses on targets that are difficult to detect. This improvement significantly improves the accuracy and robustness of the model in pedestrian detection in complex environments.
[0139] (2) The present invention significantly improves the quality of pedestrian feature extraction by introducing a detail enhancement convolution layer and a feature selection layer into the ReID model; the detail enhancement convolution layer improves the sensitivity to pedestrian image details, and the feature selection layer reduces the interference of redundant information. These improvements enable the ReID model to extract features efficiently and accurately even in complex environments such as multiple occlusions and rapid motion, significantly enhancing the ReID model's ability to extract pedestrian appearance features, thereby improving the stability and accuracy of target tracking and significantly reducing the incidence of false detection and missed detection.
[0140] (3) The present invention combines the improved YOLOv8 with the HybridSORT algorithm, making full use of the advantages of YOLOv8 in multi-scale feature fusion and the robustness of HybridSORT in multi-strategy data association. It not only enables the system to accurately detect pedestrian targets in complex environments, but also ensures continuous tracking of targets through the strong and weak clue modeling and lost target recovery mechanism of HybridSORT, greatly improving the reliability and stability of the overall system and making up for the shortcomings of the existing technology in detection and tracking performance in complex environments.
[0141] (4) The present invention is particularly suitable for environments with dense crowds, fast-moving pedestrians or complex backgrounds, and can achieve more efficient and reliable pedestrian detection and tracking in these scenarios.
[0142] The above is an exemplary description of the invention. Obviously, the specific implementation of the present invention is not limited to the above-mentioned method. As long as such non-substantial improvements are made by adopting the method concept and technical solution of the present invention, or the concept and technical solution of the invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.
Claims
1. A pedestrian tracking method based on improved YOLOv8 and HybridSORT, It is characterized in that The following steps are involved: S100, input the video sequence, normalize each frame of the image, convert it into a size and format that meets the input requirements of the YOLOv8 model, and input it into the detector; S200, using an improved YOLOv8 model to perform target detection, wherein the YOLOv8 model consists of four layers: Input, Backbone, Neck, and Head; S300, after completing the target detection, cropping the region containing the target from the original image, and inputting the region into the pre-trained improved ReID model to extract a feature vector integrating the detailed appearance features of the pedestrian; S400, using a HybridSORT algorithm for tracking, wherein the HybridSORT algorithm combines appearance features extracted by an improved ReID model, strong clue modeling, weak clue modeling, Kalman filtering, and a Hungarian algorithm to achieve robust and stable pedestrian target tracking in a complex environment; S500: Output the detection and tracking results in each frame of the image, including the location, identity information and trajectory of the pedestrian target.
2. According to the pedestrian tracking method based on improved YOLOv8 and HybridSORT according to claim 1, It is characterized in that In step S200, the YOLOv8 model includes four layers: Input, Backbone, Neck and Head, specifically including: S201, perform data enhancement operations at the input level, including mosaic enhancement, mixed enhancement, random perturbation, and color perturbation strategies; S202, extracting multi-scale feature maps at the Backbone level, wherein the YOLOv8 uses an improved CSPDarknet53 network to extract features and generate feature maps of different scales; S203, multi-scale feature fusion and enhancement are performed at the Neck level. YOLOv8 uses feature pyramid network and path aggregation network to fuse feature information of different scales, further enhancing the detection capability of targets of different scales. S204: Predict the target category and position at the Head level, generate candidate regions based on the multi-scale feature map, and update the model weights based on the improved loss function, ultimately obtaining the target category confidence prediction and accurate regression of the bounding box.
3. According to the pedestrian tracking method based on improved YOLOv8 and HybridSORT according to claim 1, It is characterized in that In step S200, the loss function improvement or optimization strategy of the YOLOv8 model specifically includes: Introducing weighted cross entropy loss to give higher weights to pedestrian categories and effectively deal with the category imbalance problem; Introducing focal loss to guide the model to pay more attention to pedestrian targets that are difficult to detect; Combining weighted cross entropy loss with focal loss, a new classification loss is defined to further optimize the problems of category imbalance and sample difficulty weight distribution.
4. The pedestrian tracking method based on improved YOLOv8 and HybridSORT according to claim 1, It is characterized in that In step S300, the ReID model is used to extract the pedestrian appearance feature vector, specifically including: S301, cropping the bounding box of the pedestrian target output by YOLOv8 and standardizing the image to preprocess the subsequent network input; S302, inputting the processed image into the ResNet50 basic convolutional network to extract the initial pedestrian appearance feature representation; S303, introducing a detail enhancement convolution layer to improve the ability to capture details of pedestrian images; S304, introduce a feature selection layer to reduce redundant information and improve the quality of feature expression; S305, performing spatial dimensional average processing on each channel of the weighted feature map through global average pooling to generate a fixed-length feature vector; S306, inputting the feature vector into a fully connected layer for further processing to optimize the expression of the feature vector; S307, using the exponential moving average EMA strategy to update the appearance feature vector to reduce feature fluctuations caused by short-term environmental changes; S308, finally outputting the feature vector after EMA smoothing, which is used for pedestrian matching and tracking by HybridSort algorithm.
5. The pedestrian tracking method based on improved YOLOv8 and HybridSORT according to claim 1, It is characterized in that In step S300, the improvement strategy of the ReID model specifically includes: Introducing detail enhancement convolutional layers to improve the sensitivity to pedestrian image details through multi-scale convolution and residual connections; The feature selection layer is introduced to select the feature area most relevant to each pedestrian's identity from the feature map through the self-attention mechanism, thereby reducing redundant information and improving the quality of feature expression. The appearance feature vector is updated using an exponential moving average strategy.
6. The pedestrian tracking method based on improved YOLOv8 and HybridSORT according to claim 1, It is characterized in that In step S400, the process of using the HybridSORT algorithm to track pedestrians is as follows: S401, input the detection result of YOLOv8 of the current frame and the pedestrian appearance feature vector extracted by the ReID model; S402, using the Kalman filter to predict the state of the tracker, estimate its position and speed in the current frame, and calculate the covariance matrix; S403, based on the target category confidence score output by YOLOv8, compare it with the confidence threshold set by HybridSORT. If it is greater than the threshold, it is a high-confidence object, and the operations of steps S404-S406 are performed; If it is less than or equal to the threshold, it is a low confidence object, and the process directly jumps to the operation of steps S407-S409; S404, calculating the standard IoU between the predicted position and the current frame detection result; S405, normalize and vectorize the obtained IoU value, the motion state information predicted by the Kalman filter, and the appearance feature, and splice the three features into a comprehensive feature vector according to different weights: S406, calculating the Euclidean distance between the current frame detection result and the comprehensive feature vector, constructing a comprehensive similarity matrix of high-confidence objects, and then performing step S410; S407, calculate the confidence score for each tracker track through trajectory confidence modeling TCM, and for tracks with confidence scores below a threshold, TCM will adjust the covariance matrix of the Kalman prediction; at the same time, use the target height information to adjust the IoU to calculate the HMIoU, so as to improve the matching accuracy when the target height changes; S408, normalizing and vectorizing the obtained HMIoU value, the motion state information after TCM adjustment, and the appearance feature, and splicing the three features into a comprehensive feature vector; S409, calculating the Euclidean distance between the current frame detection result and the comprehensive feature vector, constructing a comprehensive similarity matrix of low-confidence objects, and then performing step S410; S410, normalizing the comprehensive similarity matrices of the high-confidence and low-confidence objects to the same range and merging them into a global similarity matrix; S411, inputting the fused global similarity matrix into the Hungarian algorithm to perform Hungarian matching to determine the best matching combination between the detection result and the tracker; After the above matching process is completed, for the successfully matched detection results and trackers, the operation of step S412 is performed; for the unmatched detection results, the operation of step S413 is performed; for the unmatched trackers, the operations of steps S414-416 are performed, as follows: S412: If the tracker and the detection result are successfully matched, the Kalman filter is used to update the state of each tracker, and the next state prediction is performed, and then the process jumps to step S417; S413, if the detection result of the current frame does not find an existing tracker that matches it, initialize a new tracker for it, and then jump to step S417; S414, if a tracker fails to successfully match the detection result in the current frame, it is marked as lost, and the target position, appearance features and motion state information successfully detected by the tracker for the last time are saved, and the number of lost frames is recorded; S415, in subsequent frames, if the new detection result is similar to the last detection result of the lost target, then try to re-associate, and if the match is successful, update the tracker state; S416: If the new detection target is still not successfully matched and the number of lost frames exceeds the predetermined threshold, the tracker is completely deleted from the tracking list, and then the operation continues to step S417; S417: Update the tracker list and repeat the above operation when the next frame detection result is input.
Citation Information
Cited By
Visual analysis method for traffic intersection collaborative perception
CN119810555A
Three-dimensional positioning obstacle avoidance method for mountain forest prevention spraying unmanned aerial vehicle
CN122468125A
