A multi-target tracking method, device, equipment, storage medium and product

Abstract

The application discloses a multi-target tracking method, device, equipment, storage medium and product. The method comprises the following steps: acquiring a to-be-tested video; performing feature extraction and fusion on the to-be-tested video based on a YOLOv8 backbone and neck network structure to obtain multi-scale features; performing multi-target detection according to the multi-scale features to obtain a detection result; and determining the motion trajectory of each target in the to-be-tested video based on the detection result. The backbone network of YOLOv8 is used to improve the multi-target tracking algorithm, so that the target tracking precision can be improved under the premise of ensuring real-time performance.

Description

Technical Field

[0001] This application relates to the field of target tracking technology, and in particular to a method, apparatus, device, storage medium and product for multi-target vehicle tracking. Background Technology

[0002] Multi-target vehicle tracking is a key technology in intelligent transportation systems for real-time monitoring and management of road vehicles, analysis of traffic flow and vehicle operating status, and is also an important foundation for autonomous driving environmental perception and decision-making. Existing vehicle tracking solutions mainly include two categories: traditional trackers based on correlation filtering and deep learning-based trackers. Due to the complexity of real-world road scenarios and the frequent changes in posture and scale of vehicles during movement, leading to problems such as image blurring and trajectory loss, vehicle tracking models need to possess high inference speed and the ability to extract robust target features. Deep learning-based methods have achieved significant progress in tracking accuracy; however, existing methods are still prone to trajectory switching and poor accuracy in dense traffic and complex occlusion scenarios, and there is a challenge in balancing real-time performance and accuracy. Summary of the Invention

[0003] To address the above technical issues, this application provides a multi-target tracking method, apparatus, device, storage medium, and product. By utilizing the YOLOv8 backbone network to improve the multi-target tracking method, the target tracking accuracy can be enhanced while ensuring real-time performance.

[0004] This application provides a multi-target tracking method, including: Obtain the video to be tested; Based on the backbone and neck network structure of YOLOv8, feature extraction and fusion are performed on the video under test to obtain multi-scale features; Multi-target detection is performed based on the multi-scale features to obtain detection results, and the motion trajectory of each target in the video to be tested is determined based on the detection results.

[0005] As an improvement to the above scheme, in the step of feature extraction and fusion of the video under test based on the backbone and neck network structure of YOLOv8, the entire downsampling network for feature extraction and fusion adopts a cross-stage local network structure, including a convolution-batch normalization-activation function (CBS) module, a bifractal cross-stage C2F module, and a fast spatial pyramid pooling (SPPF) structure; the feature fusion upsampling network adopts a C2F structure.

[0006] As an improvement to the above scheme, the step of performing multi-target detection based on the multi-scale features to obtain detection results, and determining the motion trajectory of each target in the video to be tested based on the detection results, includes: The multi-scale features of the current video frame are input into a parallel branching structure without anchor boxes for target detection and re-identification feature extraction, so as to obtain the detection results and re-identification features of each target respectively. Based on the existing trajectory of the current video frame, a motion model is used to predict the position of the trajectory in the next video frame, thus obtaining the predicted trajectory. Based on the detection results, the re-identification features, and the predicted trajectory, a hierarchical data association algorithm is used to associate the trajectories and generate the motion trajectories of each target.

[0007] As an improvement to the above scheme, the multi-scale feature input of the current video frame is used in a parallel branching structure without anchor boxes for target detection and re-identification feature extraction, to obtain the detection result and re-identification features of each target, including: The multi-scale features are input into the target detection branch to obtain the target detection result; the detection head of the target detection branch includes a heatmap head, a width and height head, and an offset head. The heatmap head is used to detect the target center point position, the width and height head is used to detect the target bounding box position, and the offset head is used to detect the target center point offset. The multi-scale features are input into the re-identification feature extraction branch, and the re-identification feature extraction head is used to encode the appearance information of the target to obtain the re-identification features of the target.

[0008] As an improvement to the above scheme, the loss function of the re-identification feature extraction branch is the gradient coordination mechanism (GHM) loss function; the GHM loss function is used to set the sample loss weight according to the sample density intervals under different confidence levels. The loss function of the parallel branch structure is an uncertainty-based multi-task adaptive weighted loss function, which is used to jointly optimize the loss of the target detection branch and the loss of the re-identification feature extraction branch.

[0009] As an improvement to the above scheme, the step of generating the motion trajectory of each target by performing trajectory association using a hierarchical data association algorithm based on the detection results, the re-identification features, and the predicted trajectory includes: Obtain the historical re-identification features of the existing trajectories, and calculate the cosine distance between the re-identification features of each target in the current video frame and the historical re-identification features of each trajectory; Calculate the Mahalanobis distance between the detection result of each target in the current video frame and each of the predicted trajectories; Based on the cosine distance and the Mahalanobis distance, the Hungarian algorithm is used to perform the first round of matching between the detection results and the trajectory. After the first round of matching, for the unmatched detection results and trajectories, the intersection-union algorithm is used to perform a second round of matching of the detection results and trajectories, and the detection results that are still unmatched after the second round of matching are initialized as new trajectories; Based on the trajectory association results of the first round of matching and the second round of matching, the motion trajectory of each target is generated.

[0010] This application also provides a multi-target tracking device, including: The data acquisition module is used to acquire the video to be tested; The feature extraction module is used to extract and fuse features from the video under test based on the backbone and neck network structure of YOLOv8 to obtain multi-scale features; The target tracking module is used to perform multi-target detection based on the multi-scale features to obtain detection results, and to determine the motion trajectory of each target in the video under test based on the detection results.

[0011] This application also provides a computer device including a processor and a memory, wherein the memory stores a computer program and the computer program is configured to be executed by the processor, wherein the processor executes the computer program to implement the multi-target tracking method described in any of the preceding claims.

[0012] This application also provides a computer-readable storage medium storing a computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the multi-target tracking method described in any of the above claims.

[0013] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-target tracking method described in any of the preceding claims.

[0014] Compared with existing technologies, the beneficial effects of the multi-target tracking method, apparatus, device, storage medium, and product provided in this application are as follows: By extracting and fusing features from the video under test based on the backbone and neck network structure of YOLOv8, multi-scale features are obtained. Then, multi-target detection is performed based on the multi-scale features to obtain detection results. Based on the detection results, the motion trajectory of each target in the video under test is determined. It introduces the backbone and neck network structure of YOLOv8 on the basis of the multi-target tracking model, which can reduce the amount of computation and enhance gradient information, improve multi-scale adaptability, and thus improve accuracy while ensuring real-time performance. It is suitable for multi-target vehicle tracking in dense traffic and complex occlusion scenarios. Attached Figure Description

[0015] Figure 1This is a flowchart illustrating a multi-target tracking method provided in an embodiment of this application; Figure 2 This is a structural diagram of the algorithm model of a multi-target tracking method provided in an embodiment of this application; Figure 3 This is a network structure diagram for feature extraction and fusion provided in an embodiment of this application; Figure 4 This is a schematic diagram of a C2F structure provided in an embodiment of this application; Figure 5 This is a schematic diagram of an SPPF structure provided in an embodiment of this application; Figure 6 This application provides a visualization diagram of online vehicle trajectory association. Figure 7 This is a schematic diagram of the structure of a multi-target tracking device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0017] Please see Figure 1 , Figure 1 This is a flowchart illustrating a multi-target tracking method provided in an embodiment of this application. The multi-target tracking method includes: S1: Obtain the video to be tested; S2: Based on the backbone and neck network structure of YOLOv8, feature extraction and fusion are performed on the video under test to obtain multi-scale features; S3: Perform multi-target detection based on the multi-scale features to obtain detection results, and determine the motion trajectory of each target in the video to be tested based on the detection results.

[0018] Specifically, this application improves the detection-based online multi-target tracking model by introducing the backbone and neck network structure of YOLOv8 to enhance the model's real-time performance and balance accuracy. It can be applied to technologies such as multi-target vehicle tracking, and can reduce the number of trajectory switching due to occlusion caused by dense traffic flow, thereby improving tracking accuracy.

[0019] The detection-based online multi-object tracking model includes a feature extraction model, an appearance model, a motion model, a data association algorithm, and an online update mechanism. For the test video, the multi-object tracking model processes each video frame. Based on the first input frame, the model obtains the initial detection result and initial motion trajectory. Subsequently, it uses the contextual information of the video frames to model the appearance features and motion of the targets, thereby estimating the target's position in subsequent frames.

[0020] The tracking system comprises several key components: a feature extraction model, an object detection network, and an appearance model. The feature extraction model extracts depth appearance features for trajectory matching and decision-making, serving as the cost of trajectory association. Effectively combining these two models improves the overall tracking system's timeliness. The motion model describes the relationship between target motion states between frames, predicting the target's location in subsequent video frames. A data association algorithm takes the detection results from the object detection network, the trajectory predictions from the motion model, and the depth appearance features extracted by the appearance model as input to generate the target's entire motion trajectory. An online update mechanism continuously updates the target's motion trajectory appearance and motion model parameters based on the updated online video, recording information changes during tracking and ensuring more accurate trajectory association in subsequent video frames.

[0021] Preferably, in this embodiment of the application, the FairMOT (Fair Multiple Object Tracking) algorithm is used to construct the multi-object tracking model. This algorithm achieves fairer multi-object tracking by balancing feature learning of detection and re-identification tasks.

[0022] Based on the FairMOT multi-target tracking model, this application embodiment focuses on improving the feature extraction and fusion part, and introduces the backbone and neck network structure of YOLOv8 to reduce the amount of computation and enhance gradient information, improve multi-scale adaptability, and thus improve accuracy while ensuring real-time performance.

[0023] YOLOv8 (You Only Look Once version 8) is an object detection model whose network mainly consists of a backbone network for feature extraction, a neck network for feature fusion, and a detection head. In this embodiment, the YOLOv8 backbone network downsamples the input video frames, progressively extracting features at different scales. The neck network then performs multi-scale feature fusion as the feature extraction and fusion part. Further, the feature extraction and fusion part is followed by branches such as the object detection network and appearance model, enabling the multi-object tracking model to track objects based on the multi-scale features extracted by the YOLOv8 backbone and neck network, generating the motion trajectories of each object.

[0024] For example, please refer to Figure 2 This is a structural diagram of the algorithm model of the multi-target tracking method provided in this application embodiment. The Encoder is a feature extraction network, i.e., the backbone network of YOLOv8; the Decoder is a feature fusion network, i.e., the neck network of YOLOv8; the Head is the detection head, specifically including four detection heads: heatmap, box_size, offset, and ReID (re-identification). The first three detection heads are target detection branches, corresponding to the target detection network, and the last detection head is the re-identification feature extraction branch, corresponding to the appearance model. Then, a Kalman filter is used as the motion model for trajectory prediction. Finally, based on a hierarchical data association method and the Hungarian algorithm, the detection results, re-identification features, and predicted trajectory are correlated to generate the motion trajectory of the tracked target.

[0025] This application addresses the problem of uneven speed and accuracy in multi-target vehicle tracking in existing technologies by proposing an improved FairMOT target tracking algorithm model based on the YOLOv8 network structure. This model uses the backbone and neck network in YOLOv8 to balance speed and accuracy, and performs feature extraction and fusion to extract accurate vehicle features, significantly improving the real-time performance and accuracy of vehicle tracking.

[0026] As one of the optional embodiments, in the step of feature extraction and fusion of the video under test based on the backbone and neck network structure of YOLOv8, the entire downsampling network for feature extraction and fusion adopts a cross-stage local network structure, including a convolution-batch normalization-activation function (CBS) module, a bifractal cross-stage C2F module, and a fast spatial pyramid pooling (SPPF) structure; the feature fusion upsampling network adopts a C2F structure.

[0027] Specifically, the feature extraction downsampling network and the feature fusion upsampling network use a C2F (bifractal cross-stage module) structure with richer gradient flow, and adjust the number of channels for different scales. The feature fusion upsampling network also uses a C2F structure with richer gradient flow. The C2F module achieves feature fusion by concatenating features from different branches along the channel dimension. The concatenated features contain information from different branches, enriching the expressive power of the features and making the model more adaptable to targets of different sizes and shapes while maintaining a lightweight design. The entire downsampling network adopts a cross-stage local network structure, including CBS (Convolution, Batch Normalization, SiLU activation), the C2F module, and the SPPF (Spatial Pyramid Pooling–Fast) structure. The feature extraction downsampling network performs five downsampling passes with channel numbers of 64, 128, 256, 512, and 1024, respectively.

[0028] Please see Figure 3 The diagram shows the feature extraction and fusion network structure. On the left is the backbone network of YOLOv8, which includes a feature extraction network primarily based on downsampling. This network consists of stacked CBS and C2F units, with SPPF at the end. CBS serves as the basic convolutional and downsampling units, used for progressive feature extraction; C2F is used for cross-stage local fusion to improve feature propagation; and finally, SPPF expands the receptive field and aggregates multi-scale information. On the right is the neck network of YOLOv8, which includes an upsampling and downsampling feature fusion network. Upsampling units are used for multi-scale feature alignment, while downsampling enhances cross-scale semantic propagation.

[0029] Specifically, such as Figure 4 As shown, the C2F module first passes the input data through a first convolutional layer and an additional splitting operation (i.e., Split), then divides the output into two parts. One part is directly passed to the output, while the other part is processed by multiple Bottleneck modules. Finally, the results of the two parts are concatenated along the channel dimension and passed through a second convolutional layer to obtain the final output. This not only enriches the gradient backpropagation process and strengthens the feature information, but also helps to increase the network's nonlinearity and representation capabilities, thereby improving the network's ability to model complex data. For the SPPF structure, as shown... Figure 5As shown, when processing data, the SPPF structure performs serial max-pooling (i.e., MaxPool2d) on the feature map, with each max-pooling layer having a kernel size of 5 to achieve higher serial structure efficiency and improve computational speed. Simultaneously, the SPPF operation divides the feature map into different spatial scales, enabling the network to perceive targets of varying sizes. In this embodiment, both the C2F module and the SPPF structure contribute to improving the real-time performance of vehicle tracking. Furthermore, the downsampling network effectively blends local and global features of the feature map, resulting in richer extracted feature information. This addresses the issue of multi-scale target variations to some extent, improving the overall model's detection performance in complex multi-target scenarios.

[0030] As one optional embodiment, the step of performing multi-target detection based on the multi-scale features to obtain detection results, and determining the motion trajectory of each target in the video under test based on the detection results, includes: The multi-scale features of the current video frame are input into a parallel branching structure without anchor boxes for target detection and re-identification feature extraction, so as to obtain the detection results and re-identification features of each target respectively. Based on the existing trajectory of the current video frame, a motion model is used to predict the position of the trajectory in the next video frame, thus obtaining the predicted trajectory. Based on the detection results, the re-identification features, and the predicted trajectory, a hierarchical data association algorithm is used to associate the trajectories and generate the motion trajectories of each target.

[0031] Specifically, in this embodiment, after feature extraction and fusion of the input video frames, a multi-task learning method based on anchor boxes is combined, employing parallel object detection and re-identification feature extraction branches to obtain detection results and re-identification features, thereby reducing ambiguity caused by anchor boxes. With this parallel branch structure, the vehicle tracking model can output the detection results and identity information required for vehicle trajectory matching, association, and updating within a single network, meeting real-time requirements. Furthermore, a motion model is used to predict the trajectory's position in the next video frame, and a hierarchical online data association algorithm is used to update the trajectory's re-identification features frame by frame, performing trajectory association to ultimately obtain the real-time vehicle tracking trajectory.

[0032] Preferably, a Kalman filter is used as the motion model. The Kalman filter is a recursive state-space model based on an estimation algorithm, and it is an optimal recursive data processing algorithm. "Optimal" refers to resolving the weight relationship between the estimated and measured values, thereby obtaining a result closer to the true value. In detection-based target tracking tasks, the Kalman filter enables the tracker to predict the future trajectory of the target based on its current trajectory.

[0033] As one optional embodiment, the step of inputting the multi-scale features of the current video frame into a parallel branching structure without anchor boxes for target detection and re-identification feature extraction, to obtain the detection result and re-identification features of each target, includes: The multi-scale features are input into the target detection branch to obtain the target detection result; the detection head of the target detection branch includes a heatmap head, a width and height head, and an offset head. The heatmap head is used to detect the target center point position, the width and height head is used to detect the target bounding box position, and the offset head is used to detect the target center point offset. The multi-scale features are input into the re-identification feature extraction branch, and the re-identification feature extraction head is used to encode the appearance information of the target to obtain the re-identification features of the target.

[0034] Specifically, a parallel object detection branch and a re-identification feature extraction branch are connected at the end of the feature fusion network. The object detection branch includes three detection heads: a heatmap head, a width and height head, and an offset head, used to detect the target location in the image. In the vehicle tracking method, the heatmap head is responsible for estimating the center position of the vehicle target object, the width and height head estimates the height and width of the vehicle target bounding box at each position, and the center point offset head recovers the accurate position information lost due to the feature downsampling process. The object detection branch does not rely on preset anchor boxes; it obtains the target center by decoding the peak values ​​of the heatmap and directly generates the bounding box by combining the regressed width, height, and offset. Therefore, non-maximum suppression is not required at the inference end, and no grouping or post-processing is needed after extracting the center point, which saves resources and avoids excessive ambiguity in crowded scenes.

[0035] The heatmap header is mainly for estimating the target heatmap. The probability that each location contains the center of the target object decreases rapidly as the distance between the predicted location and the object's center increases. Specifically, the center point of the vehicle object is predicted by calculating the peak points of the heatmap.

[0036] For example, target heatmap The construction process is as follows: For the i-th ground truth bounding box in the image The real center is :

[0037]

[0038] in, Let be the coordinates of the top-left corner of the i-th target. Let be the coordinates of the bottom right corner of the i-th target. and Let x and y be the x and y coordinates of the center of the i-th target. Its position on the feature map. , where S is the downsampling step size of the output features of the feature fusion network. Then the position... Heatmap response at the location as follows:

[0039] in, This indicates the number of target objects in the image. It represents the standard deviation, which varies depending on the size of the object.

[0040] Final heatmap It is obtained by merging the Gaussian values ​​of all targets element by element to avoid the probability superposition when they overlap, which would make it difficult to distinguish the centers.

[0041] Loss function of heatmap header The loss function for pixel-wise logistic regression to reduce penalties is as follows:

[0042] in, , It is a hyperparameter in the loss function.

[0043] Furthermore, the width and height are regressed based on the target center point to reconstruct the bounding box of the vehicle object. The coordinates of each ground truth bounding box in the image are represented as... The actual width and height are: The estimated width and height at the corresponding position are expressed as... .

[0044] Loss function of width and height head The L1 loss is used, which involves averaging the absolute values ​​of the estimated and actual values. The formula is as follows:

[0045] in, This indicates the number of target objects in the image.

[0046] Furthermore, since downsampling an image causes center point shift, resulting in errors, an offset header is used to recover the accurate positional information lost during the feature downsampling process. Specifically, horizontal and vertical offsets are applied to the reduced target center to precisely adjust the target object's position. By comparing the position of each pixel with the target center, the offset of the model's output is determined, thereby mitigating the errors generated during downsampling.

[0047] Assuming the true offset is The estimated offset at the corresponding position is expressed as The loss function of the offset head Using the L1 loss, the formula is as follows:

[0048] in, This indicates the number of target objects in the image.

[0049] Furthermore, the loss function of the entire object detection branch... The uncertainty weighted sum of the heatmap header, width and height header, and offset header is expressed by the following formula:

[0050] in, , , These are the weight values ​​for the heatmap header, width and height header, and offset header, respectively, used to balance the contributions of the three tasks.

[0051] Furthermore, in order to simultaneously obtain vehicle detection results and learn vehicle identity embedding information in a single network, after feature extraction and fusion, this embodiment decouples the parallel branch and adds a re-identification feature extraction head for extracting vehicle identity features. This head encodes the vehicle's appearance information into an appearance embedding vector for vehicle re-identification, enabling subsequent trajectory matching. Specifically, re-identification features are obtained by first performing a convolution with a kernel size of 3, followed by a convolution with a kernel size of 1.

[0052] Furthermore, embodiments of this application also attach a fully connected layer with the same number of class output channels to the re-identification feature extraction head, so that the task is treated as a classification task.

[0053] As one optional embodiment, the loss function of the re-identification feature extraction branch is the gradient coordination mechanism (GHM) loss function; the GHM loss function is used to set the sample loss weight according to the sample density intervals under different confidence levels. The loss function of the parallel branch structure is an uncertainty-based multi-task adaptive weighted loss function, which is used to jointly optimize the loss of the target detection branch and the loss of the re-identification feature extraction branch.

[0054] Specifically, the loss function for the re-identification feature extraction branch adopts GHM Loss (Gradient Harmonizing Mechanism), which can alleviate the training bias problem caused by unbalanced sample gradient density. The GHM loss function balances location and target class, setting sample loss weights based on sample density intervals at different confidence levels. Specifically, it uses the gradient magnitude of the samples as the dividing criterion, assigning weights to samples according to gradient magnitude and inversely proportional to the gradient density within each interval. This prevents the model from over-focusing on a very small number of outliers, thus avoiding the training process being dominated by a few abnormal samples. Specifically, it is expressed as follows:

[0055] in, It is a binary cross-entropy function. To predict the class probability of a sample for the model, For sample category information, For density compatibility function, This represents the number of samples.

[0056] Furthermore, to achieve an adaptive trade-off between object detection and re-identification feature extraction, the loss function of the parallel branch structure employs an uncertainty-based multi-task adaptive weighted loss function. Specifically, it is expressed as follows:

[0057] in, and To balance the learnable parameters of the two tasks.

[0058] This application embodiment inputs multi-scale features extracted from video frames into parallel branches for target detection and re-identification feature extraction, enabling the vehicle tracking model to obtain the target detection results and vehicle appearance features required for vehicle tracking trajectory association in an efficient network, effectively improving the real-time performance and accuracy of vehicle tracking.

[0059] As one optional embodiment, the step of generating the motion trajectory of each target by performing trajectory association using a hierarchical data association algorithm based on the detection results, the re-identification features, and the predicted trajectory includes: Obtain the historical re-identification features of the existing trajectories, and calculate the cosine distance between the re-identification features of each target in the current video frame and the historical re-identification features of each trajectory; Calculate the Mahalanobis distance between the detection result of each target in the current video frame and each of the predicted trajectories; Based on the cosine distance and the Mahalanobis distance, the Hungarian algorithm is used to perform the first round of matching between the detection results and the trajectory. After the first round of matching, for the unmatched detection results and trajectories, the intersection-union algorithm is used to perform a second round of matching of the detection results and trajectories, and the detection results that are still unmatched after the second round of matching are initialized as new trajectories; Based on the trajectory association results of the first round of matching and the second round of matching, the motion trajectory of each target is generated.

[0060] Specifically, for the first frame of the video to be tested, the target trajectory and corresponding re-identification features are initialized based on the detected target object. In subsequent video frames, a two-stage matching strategy is used to link the detected target in each video frame to the existing trajectory to complete the trajectory matching and association.

[0061] In the first stage, i.e., the first round of matching, the minimum cosine distance between the i-th trajectory and the j-th detected target in the appearance space is measured based on the detection results and re-identification features obtained from the current video frame. Existing trajectories are determined by the detection results of previous video frames, and the historical re-identification features corresponding to the trajectories are the re-identification features corresponding to previous video frames. Matching is performed by calculating the cosine distance between the re-identification features of the current detected target and the historical re-identification features of the existing trajectories. The formula for the cosine distance is as follows:

[0062] in, Let be the cosine distance between the i-th trajectory and the j-th detected target. For the re-identification features of the j-th detected target, Let i be the re-identification feature of the i-th trajectory. This is the set of features for trajectory re-identification.

[0063] On the other hand, based on the existing trajectories of previous video frames, a Kalman filter is used to predict the trajectory position of the next frame, thus obtaining the predicted trajectory. Then, based on the detection results of the current video frame and the predicted trajectory, the Mahalanobis distance between the detected and predicted positions is calculated to match the current detected and predicted trajectories. The formula for the Mahalanobis distance is as follows:

[0064] in, Let be the Mahalanobis distance between the i-th trajectory and the j-th detected target; It is the projection of the i-th trajectory into the measurement space after Kalman filtering. It is the corresponding covariance matrix; It is the measurement vector of the j-th detected target.

[0065] Finally, the weighted distance D is obtained based on the cosine distance and Mahalanobis distance mentioned above, and the formula is expressed as follows:

[0066] in, These are weighting parameters. If the Mahalanobis distance is greater than a preset threshold, then... The value is set to infinity to avoid obtaining trajectories with large movements. Based on the weighted distance D, the Hungarian algorithm is used to find the optimal matching scheme, completing the first stage of trajectory matching.

[0067] The Hungarian algorithm systematically enumerates possible matching relationships between tasks and executors to find the optimal solution, resolving the task allocation problem in less time. The task allocation problem involves assigning n tasks to n executors to maximize efficiency. During the matching process, an n×n matrix is ​​established as the processing model, stipulating that the i-th executor can only complete one of the n tasks, and the j-th task can only be completed by a single executor. Then, based on the premise that selecting an element from each row and column minimizes the evaluation function, elements are chosen from the processing matrix. This allocation problem model can be represented as follows:

[0068] in, This represents the efficiency of the i-th executor in performing the j-th task. In the detection-based multi-target tracking task of this embodiment, the Hungarian algorithm is used to solve the problem of allocating trajectories and detection results, ultimately aiming to minimize the total allocation cost, which is determined by the weighted distance D.

[0069] In the second stage, or the second round of matching, a second matching is performed on the unmatched detection results and trajectories. Specifically, the IoU (Intersection over Union) algorithm is used for matching, which is based on the overlap ratio between the target box of the trajectory and the detection result. A matching threshold is set as the input of the Hungarian algorithm to achieve the second round of supplementary matching.

[0070] Finally, the detection results that still fail to match after two rounds of matching are initialized as new trajectories for target tracking in the next frame. Unmatched trajectories are saved for a certain number of frames to prevent future occurrences. By performing frame-by-frame target detection and two-stage trajectory matching and association in the video under test, the vehicle's target tracking trajectory can be obtained. For example, a visualization of online vehicle trajectory association using an embodiment of this application is shown below. Figure 6 As shown in the figure, the target tracking method of this application can quickly and accurately track and locate moving vehicles.

[0071] Compared to existing technologies, the embodiments of this application can improve accuracy while ensuring real-time vehicle tracking. By using the backbone and neck branches in YOLOv8 to balance speed and accuracy, feature extraction and fusion are performed to extract target features. Downsampling is performed using the C2F module and the SPPF module, and then the features are fused by combining the C2F structure, ensuring tracking accuracy. The introduction of the C2F module can effectively mix local and global features of video frames. By performing feature transformation on the input data through two convolutional layers, it helps to extract features of different levels and abstractions in the input data. While maintaining lightweight design, it obtains richer gradient flow information, and adjusts the number of channels according to the model scale, making the extracted feature information richer. This can solve the problem of multi-scale changes of targets to a certain extent, improve the detection effect of the entire model in complex multi-target scenes, and thus significantly improve the model performance. Furthermore, by feeding the extracted multi-scale features into a parallel branch using anchor boxes for target detection and re-identification feature extraction, the vehicle tracking model can obtain the target detection results and vehicle appearance features required for vehicle tracking trajectory association in an efficient network. Finally, a hierarchical data association algorithm is used for trajectory association, which significantly improves the real-time performance of vehicle tracking while balancing accuracy. It can reduce the number of trajectory switching due to obstruction caused by dense traffic flow and improve tracking accuracy.

[0072] Accordingly, this application also provides a multi-target tracking device capable of implementing all the processes of the multi-target tracking method in the above embodiments.

[0073] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a multi-target tracking device provided in an embodiment of this application. The multi-target tracking device includes: Data acquisition module 701 is used to acquire the video to be tested; Feature extraction module 702 is used to extract and fuse features from the video under test based on the backbone and neck network structure of YOLOv8 to obtain multi-scale features; The target tracking module 703 is used to perform multi-target detection based on the multi-scale features to obtain detection results, and to determine the motion trajectory of each target in the video to be tested based on the detection results.

[0074] Preferably, in the step of feature extraction and fusion of the video under test based on the backbone and neck network structure of YOLOv8, the entire downsampling network for feature extraction and fusion adopts a cross-stage local network structure, including a convolution-batch normalization-activation function (CBS) module, a bifractal cross-stage C2F module, and a fast spatial pyramid pooling (SPPF) structure; the feature fusion upsampling network adopts a C2F structure.

[0075] Preferably, the step of performing multi-target detection based on the multi-scale features to obtain detection results, and determining the motion trajectory of each target in the video to be tested based on the detection results, includes: The multi-scale features of the current video frame are input into a parallel branching structure without anchor boxes for target detection and re-identification feature extraction, so as to obtain the detection results and re-identification features of each target respectively. Based on the existing trajectory of the current video frame, a motion model is used to predict the position of the trajectory in the next video frame, thus obtaining the predicted trajectory. Based on the detection results, the re-identification features, and the predicted trajectory, a hierarchical data association algorithm is used to associate the trajectories and generate the motion trajectories of each target.

[0076] Preferably, the step of inputting the multi-scale features of the current video frame into a parallel branching structure without anchor boxes for target detection and re-identification feature extraction, to obtain the detection result and re-identification features of each target, includes: The multi-scale features are input into the target detection branch to obtain the target detection result; the detection head of the target detection branch includes a heatmap head, a width and height head, and an offset head. The heatmap head is used to detect the target center point position, the width and height head is used to detect the target bounding box position, and the offset head is used to detect the target center point offset. The multi-scale features are input into the re-identification feature extraction branch, and the re-identification feature extraction head is used to encode the appearance information of the target to obtain the re-identification features of the target.

[0077] Preferably, the loss function of the re-identification feature extraction branch is the gradient coordination mechanism (GHM) loss function; the GHM loss function is used to set the sample loss weight according to the sample density intervals under different confidence levels. The loss function of the parallel branch structure is an uncertainty-based multi-task adaptive weighted loss function, which is used to jointly optimize the loss of the target detection branch and the loss of the re-identification feature extraction branch.

[0078] Preferably, the step of generating motion trajectories for each target by associating trajectories using a hierarchical data association algorithm based on the detection results, the re-identification features, and the predicted trajectory includes: Obtain the historical re-identification features of the existing trajectories, and calculate the cosine distance between the re-identification features of each target in the current video frame and the historical re-identification features of each trajectory; Calculate the Mahalanobis distance between the detection result of each target in the current video frame and each of the predicted trajectories; Based on the cosine distance and the Mahalanobis distance, the Hungarian algorithm is used to perform the first round of matching between the detection results and the trajectory. After the first round of matching, for the unmatched detection results and trajectories, the intersection-union algorithm is used to perform a second round of matching of the detection results and trajectories, and the detection results that are still unmatched after the second round of matching are initialized as new trajectories; Based on the trajectory association results of the first round of matching and the second round of matching, the motion trajectory of each target is generated.

[0079] In specific implementation, the working principle, control process and technical effects of the multi-target tracking device provided in this application are the same as those of the multi-target tracking method in the above embodiments, and will not be repeated here.

[0080] See Figure 8 , Figure 8 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device includes a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the processor 801 executes the computer program, it implements the steps in the above-described multi-target tracking method embodiments. Alternatively, when the processor 801 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments.

[0081] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 802 and executed by the processor 801 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0082] The computer device may include, but is not limited to, a processor 801 and a memory 802. Those skilled in the art will understand that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0083] The processor 801 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 801 is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0084] The memory 802 can be used to store the computer programs and / or modules. The processor 801 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 802 and calling the data stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0085] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 801, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0086] This application also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the multi-target tracking method described in any of the above embodiments.

[0087] This application also provides a computer program product, including a computer program / instruction, which, when executed by the computer program / instruction processor, implements the multi-target tracking method described in any of the above embodiments.

[0088] The methods in this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in this application are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, or other programmable devices.

[0089] The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; or an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0090] The computer programs / instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards those instructions to computer-readable storage media within the respective computing / processing device.

[0091] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays, are personalized by utilizing the status information of the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0092] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams according to embodiments thereof. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0093] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0094] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0095] Each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0096] This application provides a multi-target tracking method, apparatus, device, storage medium, and product. Its advantages lie in the following: By extracting and fusing features from the video under test using a YOLOv8-based backbone and neck network structure, multi-scale features are obtained. Then, multi-target detection is performed based on these multi-scale features to obtain detection results. The motion trajectories of each target in the video under test are determined based on these detection results. This method introduces the YOLOv8 backbone and neck network structure into the multi-target tracking model, which reduces computational load and enhances gradient information, improving multi-scale adaptability. This, in turn, improves accuracy while ensuring real-time performance, making it suitable for multi-target vehicle tracking in dense traffic and complex occlusion scenarios.

[0097] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A multi-target tracking method, characterized in that, include: Obtain the video to be tested; Based on the backbone and neck network structure of YOLOv8, feature extraction and fusion are performed on the video under test to obtain multi-scale features; Multi-target detection is performed based on the multi-scale features to obtain detection results, and the motion trajectory of each target in the video to be tested is determined based on the detection results.

2. The multi-target tracking method as described in claim 1, characterized in that, In the step of feature extraction and fusion of the video under test based on the backbone and neck network structure of YOLOv8, the entire downsampling network for feature extraction and fusion adopts a cross-stage local network structure, including a convolution-batch normalization-activation function (CBS) module, a bifractal cross-stage C2F module, and a fast spatial pyramid pooling (SPPF) structure; the feature fusion upsampling network adopts a C2F structure.

3. The multi-target tracking method as described in claim 1, characterized in that, The step of performing multi-target detection based on the multi-scale features to obtain detection results, and determining the motion trajectory of each target in the video under test based on the detection results, includes: The multi-scale features of the current video frame are input into a parallel branching structure without anchor boxes for target detection and re-identification feature extraction, so as to obtain the detection results and re-identification features of each target respectively. Based on the existing trajectory of the current video frame, a motion model is used to predict the position of the trajectory in the next video frame, thus obtaining the predicted trajectory. Based on the detection results, the re-identification features, and the predicted trajectory, a hierarchical data association algorithm is used to associate the trajectories and generate the motion trajectories of each target.

4. The multi-target tracking method as described in claim 3, characterized in that, The process involves inputting the multi-scale features of the current video frame into a parallel branching structure without anchor boxes for target detection and re-identification feature extraction, yielding the detection result and re-identification features for each target, including: The multi-scale features are input into the target detection branch to obtain the target detection result; the detection head of the target detection branch includes a heatmap head, a width and height head, and an offset head. The heatmap head is used to detect the target center point position, the width and height head is used to detect the target bounding box position, and the offset head is used to detect the target center point offset. The multi-scale features are input into the re-identification feature extraction branch, and the re-identification feature extraction head is used to encode the appearance information of the target to obtain the re-identification features of the target.

5. The multi-target tracking method as described in claim 4, characterized in that, The loss function of the re-identification feature extraction branch is the gradient coordination mechanism (GHM) loss function; the GHM loss function is used to set the sample loss weight according to the sample density intervals under different confidence levels. The loss function of the parallel branch structure is an uncertainty-based multi-task adaptive weighted loss function, which is used to jointly optimize the loss of the target detection branch and the loss of the re-identification feature extraction branch.

6. The multi-target tracking method as described in claim 3, characterized in that, The step of generating motion trajectories for each target by associating trajectories using a hierarchical data association algorithm based on the detection results, the re-identification features, and the predicted trajectory includes: Obtain the historical re-identification features of the existing trajectories, and calculate the cosine distance between the re-identification features of each target in the current video frame and the historical re-identification features of each trajectory; Calculate the Mahalanobis distance between the detection result of each target in the current video frame and each of the predicted trajectories; Based on the cosine distance and the Mahalanobis distance, the Hungarian algorithm is used to perform the first round of matching between the detection results and the trajectory. After the first round of matching, for the unmatched detection results and trajectories, the intersection-union algorithm is used to perform a second round of matching of the detection results and trajectories, and the detection results that are still unmatched after the second round of matching are initialized as new trajectories; Based on the trajectory association results of the first round of matching and the second round of matching, the motion trajectory of each target is generated.

7. A multi-target tracking device, characterized in that, include: The data acquisition module is used to acquire the video to be tested; The feature extraction module is used to extract and fuse features from the video under test based on the backbone and neck network structure of YOLOv8 to obtain multi-scale features; The target tracking module is used to perform multi-target detection based on the multi-scale features to obtain detection results, and to determine the motion trajectory of each target in the video to be tested based on the detection results.

8. A computer device, characterized in that, The system includes a processor and a memory, the memory storing a computer program configured to be executed by the processor, wherein the processor, when executing the computer program, implements the multi-target tracking method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the multi-target tracking method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the multi-target tracking method as described in any one of claims 1 to 6.

Images