Multi-target tracking method based on time memory and GNN
By combining the LSTM model of time memory and feature extraction of GhostNetV3 and combining the GCN network processing matching tasks, the problems of the existing technology in complex motion trajectories and high computing complexity are solved, and efficient and accurate target tracking is achieved, suitable for complex scenarios and real-time applications.
Patent Information
- Application Number
- CN202411209061.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-06
AI Technical Summary
The existing target tracking technology has reduced prediction accuracy in complex motion trajectories and occlusion scenarios. Kalman filtering is suitable for linear systems and has high computational complexity. Traditional feature extraction networks cannot maintain high frame rates and accuracy in real-time processing.
The LSTM model based on time memory is used for prediction, combined with GhostNetV3 for feature extraction, and two GCN networks are used to process matching tasks with different confidence levels, and target detection is used for pre-trained YOLOX, and matching relationships are built through the feature vectors of LSTM and GhostNetV3. Improve matching efficiency and accuracy.
On the premise of ensuring operational accuracy, quickly extract features, maintain high frame rate and robustness, and is suitable for complex motion scenarios and real-time applications, improving the stability and accuracy of target tracking.
Smart Images

Figure CN119090917A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target tracking technology, and in particular to a multi-target tracking method based on time memory and GNN. Background Art
[0002] Target detection and target tracking technologies can detect and track objects in video streams in real time, identify target positions and motion trajectories, and provide important support for security monitoring, traffic management, and intelligent driving. For target tracking technology, there are generally two solutions: one is detection first and then tracking, and the other is detection and tracking at the same time.
[0003] For example, "TrackFormer: Multi-Object Tracking with Transformers" published in the IEEE Conference on Computer Vision and Pattern Recognition in 2022 and "TransCenter: Transformers with dense representations for multiple-object tracking" published in the IEEE Transactions on Pattern Analysis and Machine Intelligence are algorithms that combine detection and tracking. There are many algorithms that detect first and then track, such as "Multi-Object Tracking Meets Moving UAV" published in the IEEE Conference on Computer Vision and Pattern Recognition in 2022. Practice and application have shown that the former has better results in production environments. The most widely used method is to use the recognized best target detector YOLO for target detection, and then combine methods similar to DeepSORT or ByteTrack to track the upstream detection results, which has certain defects in some application scenarios.
[0004] Most algorithms use Kalman filtering to predict and update the target position, such as "Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking" published in the IEEE Conference on Computer Vision and Pattern Recognition in 2023. The Kalman filter is suitable for state estimation of linear systems, and it is generally assumed that the object moves in a uniform straight line. However, when the target motion trajectory is complex, the motion speed changes, or there is occlusion, the prediction accuracy of the Kalman filter decreases, resulting in the accumulation of tracking errors.
[0005] In addition, some methods use the Hungarian algorithm to match prediction results and detection results, such as "ByteTrack: Multi-object Tracking by Associating Every Detection Box" published in the European Conference on Computer Vision in 2022. In scenes with dense targets, this algorithm needs to process a large number of nodes, resulting in high computational complexity, low matching efficiency, and reduced accuracy and robustness. At the same time, traditional feature extraction networks cannot achieve a good balance between inference time and accuracy. It is difficult to meet the needs of real-time processing on some embedded devices, resulting in a low video frame rate. Summary of the invention
[0006] The present invention provides a multi-target tracking method based on temporal memory and GNN, which can extract features faster while maintaining a high frame rate and comfortable visual effects while ensuring the operating accuracy.
[0007] A multi-target tracking method based on temporal memory and GNN includes the following steps:
[0008] (1) Train the LSTM model for subsequent prediction;
[0009] (2) Train the GhostNetV3 model for subsequent feature extraction;
[0010] (3) Train the GCN model for subsequent matching of detection boxes and prediction boxes. During the training process, the confidence of the detection boxes in the dataset is classified with a boundary of 0.5, and two GCNs are trained, namely GCN1 and GCN2.
[0011] (4) Use the pre-trained YOLOX to perform target detection and obtain the detection frame;
[0012] (5) Send the tracking frame of the previous frame to the corresponding LSTM network to predict the tracking result of the current frame and obtain the predicted frame;
[0013] (6) Match the detection frame obtained by YOLOX and the prediction frame obtained by LSTM, classify the target detection results of the current frame, and perform different processing respectively;
[0014] (7) For the detection frame of the current frame obtained in step (4) with a confidence greater than or equal to 0.5, match it with the prediction frame of the previous frame tracking frame obtained in step (5) output by LSTM, and construct them into a bipartite graph. The vertices of the bipartite graph are the feature vectors output by GhostNetV3 of these frames trained in step (2), and the edge value is the distance between the feature vectors. The goal is to minimize the edge value. The graph is sent to GCN1, and the output is still a graph to obtain the matching relationship;
[0015] (8) For the detection boxes with a confidence score less than 0.5 obtained in step (4), construct a bipartite graph together with the prediction boxes that were not matched in step (7). The vertices are the coordinates of these boxes, and the edge values are the intersection and union ratios of the boxes. The goal is to maximize the edge values. The graph is sent to GCN2, and the output is still a graph to obtain the matching relationship.
[0016] (9) According to the matching relationship obtained in step (7) and step (8), the matched boxes are directly drawn in the current frame, and the unmatched detection boxes with a confidence level greater than or equal to 0.5 are regarded as new targets, and other unmatched boxes are discarded.
[0017] In step (1), the input of the LSTM model is the positions of an object in the video sequence, which are arranged in chronological order. The positions are represented by boxes. The position at each moment is a vector. The output of the LSTM model is the position of the object in the next frame at the corresponding moment.
[0018] In step (2), the input of the GhostNetV3 model is the block diagram of each object, and the output of the GhostNetV3 model is a vector indicating which category it belongs to, that is, which object it belongs to; the last layer of the trained GhostNetV3 is removed, and the output of the removed GhostNetV3 is used as the feature vector of the object.
[0019] The specific process of step (3) is as follows:
[0020] Use the multi-target tracking dataset to create a new dataset and train two GCNs with different weights. For each frame of the multi-target tracking dataset video, consider the two consecutive frames before and after, and classify the detection boxes in the two frames whose confidence is greater than or equal to 0.5 into one category, named category A, and the remaining objects into another category, named category B.
[0021] For category A, the box detected in the previous frame is intercepted and sent to GhostNetV3 to obtain the corresponding feature vector. The box detected in the next frame is intercepted and sent to GhostNetV3 to obtain the corresponding feature vector. The feature vectors of the boxes detected in the previous and next frames are used as the vertex values of GCN, and the Mahalanobis distance between the vertices is used as the edge value to construct a bipartite graph; this bipartite graph is used as the input of GCN. The output of GCN is also a graph, but the edge is a number between 0 and 1, indicating the probability of the edge being selected. The output value of the matched edge is set to 1, and the others are set to 0. The trained network is called GCN1;
[0022] For category B, the position information of the boxes detected in the previous and next frames is used as the vertex value of GCN, and the EIoU between the vertices is used as the edge value to construct a bipartite graph; this bipartite graph is used as the input of GCN. The output of GCN is also a graph, but the edge is a number between 0 and 1, indicating the probability of the edge being selected. The output value of the matching edge is set to 1, and the others are set to 0. The trained network is called GCN2.
[0023] In step (4), the pre-trained YOLOX is used to perform object detection on each frame of the video to obtain detection boxes containing specific objects. These boxes are represented by vectors.
[0024] In step (5), the input tracking box is represented by a number of vectors, and the output is the predicted box of the corresponding object in the current frame, also represented by a number of vectors.
[0025] The specific process of step (7) is as follows:
[0026] A bipartite graph is constructed, in which the vertices are the prediction boxes and the detection boxes with a confidence greater than or equal to 0.5 in the current frame, and the values of the vertices are their feature vectors after GhostNetV3. Edges are established between the prediction boxes and the detection boxes with a confidence greater than or equal to 0.5 in the current frame. The value of the edge is the Mahalanobis distance of the two vectors. The prediction boxes and detection boxes of the previous and next frames are matched and modeled into a bipartite graph. The goal is to minimize the edge value. This graph is sent to GCN1, and the output is still a graph with unchanged graph structure. The value of the edge is the probability of the edge being selected, that is, the probability of the corresponding two boxes matching. First, find the edge with the largest value, confirm their matching relationship, and then delete all the edges associated with the two vertices of this edge. For the remaining edges, repeatedly find the edge with the largest value among the remaining edges, confirm their matching relationship, and then delete all the edges associated with the two vertices of this edge. Until all edges are deleted, all the optimal matching relationships are determined.
[0027] The specific process of step (8) is as follows:
[0028] A bipartite graph is constructed, where the vertices are the detection boxes with confidence less than 0.5 and the unmatched prediction boxes. Since the confidence of these detection boxes is too low and they are no longer suitable for feature extraction, the value of each vertex is their position. An edge is established between the unmatched prediction boxes of the previous frame and the detection boxes with confidence less than 0.5 of the current frame. The value of the edge is the EIoU value between the two boxes. The detection boxes with confidence less than 0.5 and the unmatched prediction boxes are modeled into a bipartite graph. The goal is to minimize the edge value. This graph is sent to GCN2. The output is still a graph with unchanged graph structure. The value of the edge is the probability of the edge being selected, that is, the probability of the corresponding two boxes matching. First, find the edge with the largest value, confirm their matching relationship, and then delete all the edges associated with the two vertices of this edge. For the remaining edges, repeatedly find the edge with the largest value among the remaining edges, confirm their matching relationship, and then delete all the edges associated with the two vertices of this edge. Until all edges are deleted, all the optimal matching relationships are determined.
[0029] The specific process of step (9) is as follows:
[0030] According to the matching relationship obtained in step (7) and step (8), for the detection frame matched in the current frame, write the ID of the prediction frame corresponding to the previous frame on the detection frame. For the detection frame that is not matched in the current frame, treat it as a newly detected object, assign a new ID to each of them, and allocate an independent LSTM network to each of them; the prediction frame and detection frame that are not matched in the end are regarded as tracking loss.
[0031] At this point, the tracking result of the current frame is obtained through the tracking result of the previous frame. The above operation is performed on all the previous and next frames in sequence to obtain the tracking result of the entire video.
[0032] Compared with the prior art, the present invention has the following beneficial effects:
[0033] In the present invention, LSTM is used for prediction. LSTM can better learn and predict complex motion patterns through its internal memory unit and gating mechanism. LSTM can also memorize long-term sequence information, reduce prediction errors, and improve tracking stability and accuracy. This is more robust in scenes where the motion speed changes quickly and is complex and occluded for a long time. The advantages of LSTM can be fully utilized by applying LSTM to target frame feature extraction and EIoU calculation at the same time. Two GCN networks are used to process matching tasks with different confidence levels. GCN can efficiently handle complex relationships between multiple nodes and improve matching efficiency and accuracy. In multi-target scenes, such as crowded scenes, matching relationships can be quickly calculated, and matching is more accurate. GhostNetV3 is used to extract features from the target frame. GhostNetV3 is a lightweight neural network structure that can significantly reduce computational complexity while ensuring feature extraction effects, and is suitable for real-time applications. The method of the present invention can extract features faster while ensuring operating accuracy, while still maintaining a high frame rate and comfortable visual effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 This is a schematic diagram of LSTM training in an embodiment of the present invention;
[0035] Figure 2 This is a schematic diagram of GhostNetV3 training in an embodiment of the present invention;
[0036] Figure 3 This is a schematic diagram of GCN1 training in an embodiment of the present invention;
[0037] Figure 4 This is a schematic diagram of GCN2 training in an embodiment of the present invention;
[0038] Figure 5 The present invention is a flowchart of target tracking according to an embodiment of the present invention. DETAILED DESCRIPTION
[0039] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.
[0040] The present invention first trains the LSTM model, using MOT17 as the dataset, focusing on the different positions of the same object before and after it appears in the video. Figure 1As shown in the figure, the input of the LSTM model is the position of an object in the video sequence, which is arranged in chronological order. The position is represented by a box. The position at each moment is a vector of the form (cx, cy, h, w), where cx and cy are the center pixel coordinates of the box, and h and w are the height and width of the box respectively. The output of the LSTM model is the position of the object in the next frame at the corresponding moment. In this way, LSTM can learn the motion characteristics of the object and prepare for subsequent position prediction.
[0041] Train the GhostNetV3 model using the ReID dataset, which contains different poses of the same object, such as Figure 2 As shown in the figure, the input of the GhostNetV3 model is the block diagram of each object, and the output of the GhostNetV3 model is a vector indicating which category it belongs to, that is, which object; each category is an object, and the same object may appear in different frames. There are as many categories as there are objects in the data set. The dimension of the last fully connected layer of GhostNetV3 is changed to the number of objects in the ReID data set. When the block diagram of each object is input, the output is a vector indicating which category it belongs to, that is, which object. The last layer (fully connected layer) of the trained GhostNetV3 is removed, and the output of the removed GhostNetV3 is used as the feature vector of the object. In this way, GhostNetV3 finally has the ability to classify.
[0042] Train the GCN model, create a dataset based on MOT17, and train two GCNs with different weights; for each frame of the MOT17 video, consider the two consecutive frames before and after, and classify the detection boxes in the two frames with confidence greater than or equal to 0.5 into one category, named category A, and the remaining objects into another category, named category B.
[0043] like Figure 3 For category A, the box detected in the previous frame is intercepted and sent to GhostNetV3 to obtain the corresponding feature vector. The box detected in the next frame is intercepted and sent to GhostNetV3 to obtain the corresponding feature vector. The feature vectors of the boxes detected in the previous and next frames are used as the vertex values of GCN, and the Mahalanobis distance between the vertices is used as the edge value. Since the detection boxes between the previous and next frames are associated, a bipartite graph can be constructed. The detection boxes of the previous and next frames are paired in pairs. As the tracking result, it can be modeled into a bipartite graph. We use this bipartite graph as the input of GCN. The output of GCN is also a graph, but the edge is a number between 0 and 1, indicating the probability of the edge being selected. Since we already know which boxes in the previous and next frames are matched, we can set the output value of the matched edge to 1 and the others to 0. The trained network is called GCN1.
[0044] like Figure 4 For category B, the position information (cx, cy, w, h) of the boxes detected in the previous and next frames is used as the vertex value of GCN, and the EIoU between the vertices is used as the edge value. Since the detection boxes between the previous and next frames are related, a bipartite graph can be constructed to pair the detection boxes of the previous and next frames as the tracking result. This can be modeled as a bipartite graph. This bipartite graph is used as the input of GCN. The output of GCN is also a graph, but the edge is a number between 0 and 1, indicating the probability of the edge being selected. Since we already know which boxes in the previous and next frames are matched, we can set the output values of these edges to 1 and the others to 0. The trained network is called GCN2.
[0045] Next, we use the trained model to track the target.
[0046] Use the pre-trained YOLOX to detect the target in each frame of the video, such as Figure 5 As shown in the figure, the detection results are several boxes containing specific objects, which are represented by vectors; each vector is in the form of (cx, cy, h, w, c), where the category dimension is discarded. Only single-category multi-target tracking (MOT) is considered here, and the same is true for multi-category. Among them, cx, cy are the center pixel coordinates of the box, h, w are the height and width of the box respectively, and c is the confidence of the detection.
[0047] Here, YOLOX uses a pre-trained model and uses it directly without the need for self-training.
[0048] The tracking box of the previous frame is sent to the corresponding LSTM network. The input is a number of vectors, each of which is in the shape of (cx, cy, h, w). The output is the prediction result of the corresponding object in the current frame, which is also a number of vectors, each of which is in the shape of (cx, cy, h, w).
[0049] The prediction boxes output by the LSTM network of the previous frame tracking result are cropped on the current frame and sent to GhostNetV3. Each predicted box is resized uniformly. The input is a matrix with a dimension of (n, h, w, 3), where n represents the number of prediction boxes, h represents height, w represents width, and 3 represents the three channels of the image. The output is a matrix with a dimension of (n, x), where n has the same meaning as above and x represents the dimension of the feature vector.
[0050] By matching the detection frame obtained by YOLOX with the prediction frame obtained by LSTM, we can know which frames of the previous frame the detection frame of the current frame comes from, thus achieving the tracking effect. The target detection results of the current frame are classified, and the detection frames with confidence greater than or equal to 0.5 and less than 0.5 are processed differently.
[0051] Sometimes, the confidence of the detection result decreases due to occlusion, blur, etc. When we use the low-confidence detection result to extract features, it is often difficult to extract the feature vector of the corresponding category target because the target occupies a small proportion in the frame, is covered by other targets, and is blurred. This will cause mismatches when matching the detection frame and the prediction frame, thus affecting the target tracking result. Therefore, we classify the confidence of the target detection result of the current frame, and perform different processing according to 0.5 as the boundary.
[0052] For the detection boxes with confidence greater than or equal to 0.5, these detection boxes are cropped and sent to GhostNetV3 to unify the scale of each detection box. The input is a matrix with a dimension of (n, h, w, 3), where n represents the number of detection boxes with confidence greater than or equal to 0.5, h represents height, w represents width, and 3 represents the three channels of the image. The output is a matrix with a dimension of (n, x), where n has the same meaning as above and x represents the dimension of the feature vector.
[0053] The above information is organized into a graph, where the vertices are the predicted boxes and the detection boxes with a confidence greater than or equal to 0.5 in the current frame, and the vertex values are their feature vectors after GhostNetV3. Edges are established between the predicted boxes and the detection boxes with a confidence greater than or equal to 0.5 in the current frame, and the edge value is the Mahalanobis distance between the two vectors. The matching of the predicted boxes and the detection boxes of the previous and next frames is modeled as a bipartite graph matching problem. The goal is to minimize the edge value. This graph is sent to GCN1, and the output is still a graph with the same graph structure. The edge value is the probability of the edge being selected, that is, the probability of the corresponding two boxes matching.
[0054] First, find the edge with the largest value, confirm their matching relationship, then delete all the edges associated with the two vertices of this edge (set the value to zero), then find the edge with the largest value among the remaining edges, confirm their matching relationship, then delete all the edges associated with the two vertices of this edge (set the value to zero), and repeat the above steps until all edges are deleted (the values of unselected edges are 0). In this way, we have determined all the optimal matching relationships.
[0055] Since the predicted frame and the detection frame of the current frame cannot completely match. For the detection frame matched in the current frame, write the ID of the corresponding prediction frame of the previous frame on the detection frame to indicate that they are the same object. For the detection frames that are not matched in the current frame, treat them as newly detected objects, give them new IDs, and assign them a separate LSTM network. For the unmatched prediction frame, it may be matched with a detection frame with a confidence level less than 0.5.
[0056] For detection frames with confidence less than 0.5, they will build a graph together with the unmatched prediction frames in the previous process. The vertices of the graph are the detection frames with confidence less than 0.5 and the unmatched prediction frames. Since the confidence of these detection frames is too low, they are no longer suitable for feature extraction, so the value of each vertex is their position, which is a vector of the form (cx, cy, h, w). An edge is established between the unmatched prediction frame of the previous frame and the detection frame with confidence less than 0.5 of the current frame. The value of the edge is the EIoU value between the two frames. EIoU is calculated as follows:
[0057] First, the IoU value between the two boxes is calculated. The IoU value is the overlapping area of the two boxes divided by the combined area. Next, the distance d between the center points of the two bounding boxes is calculated. c :
[0058]
[0059] The coordinates of the center points of the two boxes are (x1, y1) and (x2, y2), and then the square of the diagonal length of the two boxes is calculated. 2 max :
[0060]
[0061] Among them, x max , x min ,y max ,y min They are the boundary coordinates of the minimum enclosing rectangle of the two boxes.
[0062] Next the aspect ratio difference ARD is calculated.
[0063]
[0064] Where (w1, h1) and (w2, h2) are the width and height of the two boxes respectively. Finally, the EIoU value is obtained.
[0065]
[0066] In this way, the unmatched tracking results of the previous and next frames and the detection box matches with a confidence level less than 0.5 are modeled into a bipartite graph. The goal is to maximize the edge value. This graph is fed into GCN2, and the output is still a graph. The edge value is the probability of being selected, that is, the probability that the corresponding two boxes match.
[0067] First, find the edge with the largest value, confirm their matching relationship, then delete all the edges associated with the two vertices of this edge (set the value to zero), then find the edge with the largest value among the remaining edges, confirm their matching relationship, then delete all the edges associated with the two vertices of this edge (set the value to zero), and repeat the above steps until all edges are deleted (the values of unselected edges are 0), so that all the optimal matching relationships are determined.
[0068] Similarly, the current round of matching still cannot guarantee a complete match. For the prediction box matched in this round, the prediction box result is drawn on the current frame, and the corresponding ID is written on the top. The confidence is the confidence of the corresponding detection box in the current frame. The detection box is not used as the final tracking result here because the confidence of the detection box is too low (less than 0.5), so the predicted result is more trusted. The detection box and prediction box that are not matched in this round are considered as matching loss and are discarded.
[0069] At this point, the tracking result of the current frame is obtained through the tracking result of the previous frame. Repeat the above operation for all the previous and next frames in sequence to obtain the tracking result of the entire video. The input of target detection and tracking is a video (image sequence), and the output is the target tracking result of each frame. The target tracking will be drawn in the form of a box and marked with the unique ID of the target.
[0070] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A multi-target tracking method based on temporal memory and GNN, characterized in that: The following steps are involved: (1) Train the LSTM model for subsequent prediction; (2) Train the GhostNetV3 model for subsequent feature extraction; (3) Train the GCN model for subsequent matching of detection boxes and prediction boxes; During the training process, the confidence of the detection box in the data set is classified with a boundary of 0.5, and two GCNs are trained, namely GCN1 and GCN2; (4) Use the pre-trained YOLOX to perform target detection and obtain the detection frame; (5) Send the tracking frame of the previous frame to the corresponding LSTM network to predict the tracking result of the current frame and obtain the predicted frame; (6) Match the detection frame obtained by YOLOX and the prediction frame obtained by LSTM, classify the target detection results of the current frame, and perform different processing respectively; (7) For the detection frame of the current frame obtained in step (4) with a confidence greater than or equal to 0.5, match it with the prediction frame of the previous frame tracking frame obtained in step (5) output by LSTM, and construct them into a bipartite graph. The vertices of the bipartite graph are the feature vectors output by GhostNetV3 of these frames trained in step (2), and the edge value is the distance between the feature vectors. The goal is to minimize the edge value. The graph is sent to GCN1, and the output is still a graph to obtain the matching relationship; (8) For the detection boxes with a confidence score less than 0.5 obtained in step (4), construct a bipartite graph together with the prediction boxes that were not matched in step (7). The vertices are the coordinates of these boxes, and the edge values are the intersection and union ratios of the boxes. The goal is to maximize the edge values. The graph is sent to GCN2, and the output is still a graph to obtain the matching relationship. (9) According to the matching relationship obtained in step (7) and step (8), the matched boxes are directly drawn in the current frame, and the unmatched detection boxes with a confidence level greater than or equal to 0.5 are regarded as new targets, and other unmatched boxes are discarded.
2. The multi-target tracking method based on time memory and GNN according to claim 1 is characterized in that: In step (1), the input of the LSTM model is the positions of an object in the video sequence, which are arranged in chronological order. The positions are represented by boxes. The position at each moment is a vector. The output of the LSTM model is the position of the object in the next frame at the corresponding moment.
3. The multi-target tracking method based on time memory and GNN according to claim 1, characterized in that: In step (2), the input of the GhostNetV3 model is the block diagram of each object, and the output of the GhostNetV3 model is a vector indicating which category it belongs to, that is, which object it belongs to; the last layer of the trained GhostNetV3 is removed, and the output of the removed GhostNetV3 is used as the feature vector of the object.
4. The multi-target tracking method based on time memory and GNN according to claim 1 is characterized in that: The specific process of step (3) is as follows: Use the multi-target tracking dataset to create a new dataset and train two GCNs with different weights. For each frame of the multi-target tracking dataset video, consider the two consecutive frames before and after, and classify the detection boxes in the two frames whose confidence is greater than or equal to 0.5 into one category, named category A, and the remaining objects into another category, named category B. For category A, the box detected in the previous frame is intercepted and sent to GhostNetV3 to obtain the corresponding feature vector. The box detected in the next frame is intercepted and sent to GhostNetV3 to obtain the corresponding feature vector. The feature vectors of the boxes detected in the previous and next frames are used as the vertex values of GCN, and the Mahalanobis distance between the vertices is used as the edge value to construct a bipartite graph; this bipartite graph is used as the input of GCN. The output of GCN is also a graph, but the edge is a number between 0 and 1, indicating the probability of the edge being selected. The output value of the matched edge is set to 1, and the others are set to 0. The trained network is called GCN1; For category B, the position information of the boxes detected in the previous and next frames is used as the vertex value of GCN, and the EIoU between the vertices is used as the edge value to construct a bipartite graph; this bipartite graph is used as the input of GCN. The output of GCN is also a graph, but the edge is a number between 0 and 1, indicating the probability of the edge being selected. The output value of the matching edge is set to 1, and the others are set to 0. The trained network is called GCN2.
5. The multi-target tracking method based on time memory and GNN according to claim 1, characterized in that: In step (4), the pre-trained YOLOX is used to perform object detection on each frame of the video to obtain detection boxes containing specific objects. These boxes are represented by vectors.
6. The multi-target tracking method based on time memory and GNN according to claim 1, characterized in that: In step (5), the input tracking box is represented by a number of vectors, and the output is the predicted box of the corresponding object in the current frame, also represented by a number of vectors.
7. The multi-target tracking method based on time memory and GNN according to claim 1, characterized in that: The specific process of step (7) is as follows: A bipartite graph is constructed, in which the vertices are the prediction boxes and the detection boxes with a confidence greater than or equal to 0.5 in the current frame, and the values of the vertices are their feature vectors after GhostNetV3. Edges are established between the prediction boxes and the detection boxes with a confidence greater than or equal to 0.5 in the current frame. The value of the edge is the Mahalanobis distance of the two vectors. The prediction boxes and detection boxes of the previous and next frames are matched and modeled into a bipartite graph. The goal is to minimize the edge value. This graph is sent to GCN1, and the output is still a graph with unchanged graph structure. The value of the edge is the probability of the edge being selected, that is, the probability of the corresponding two boxes matching. First, find the edge with the largest value, confirm their matching relationship, and then delete all the edges associated with the two vertices of this edge. For the remaining edges, repeatedly find the edge with the largest value among the remaining edges, confirm their matching relationship, and then delete all the edges associated with the two vertices of this edge. Until all edges are deleted, all the optimal matching relationships are determined.
8. The multi-target tracking method based on time memory and GNN according to claim 1, characterized in that: The specific process of step (8) is as follows: A bipartite graph is constructed, where the vertices are the detection boxes with confidence less than 0.5 and the unmatched prediction boxes. Since the confidence of these detection boxes is too low and they are no longer suitable for feature extraction, the value of each vertex is their position. An edge is established between the unmatched prediction boxes of the previous frame and the detection boxes with confidence less than 0.5 of the current frame. The value of the edge is the EIoU value between the two boxes. The detection boxes with confidence less than 0.5 and the unmatched prediction boxes are modeled into a bipartite graph. The goal is to minimize the edge value. This graph is sent to GCN2. The output is still a graph with unchanged graph structure. The value of the edge is the probability of the edge being selected, that is, the probability of the corresponding two boxes matching. First, find the edge with the largest value, confirm their matching relationship, and then delete all the edges associated with the two vertices of this edge. For the remaining edges, repeatedly find the edge with the largest value among the remaining edges, confirm their matching relationship, and then delete all the edges associated with the two vertices of this edge. Until all edges are deleted, all the optimal matching relationships are determined.
9. The multi-target tracking method based on time memory and GNN according to claim 1, characterized in that: The specific process of step (9) is as follows: According to the matching relationship obtained in step (7) and step (8), for the detection frame matched in the current frame, write the ID of the prediction frame corresponding to the previous frame on the detection frame. For the detection frame that is not matched in the current frame, treat it as a newly detected object, assign a new ID to each of them, and allocate an independent LSTM network to each of them; the prediction frame and detection frame that are not matched in the end are regarded as tracking loss. At this point, the tracking result of the current frame is obtained through the tracking result of the previous frame. The above operation is performed on all the previous and next frames in sequence to obtain the tracking result of the entire video.
Citation Information
Patent Citations
Multi-target tracking method
CN111127513A
Multi-target tracking method and device, electronic equipment and computer storage medium
CN112163473A
Multiple target tracking method and apparatus, calculating device and storage medium
US20230030496A1
KR20220148053A
