Target tracking method combined with track prediction

Abstract

The invention discloses a target tracking method combined with track prediction, and belongs to the technical field of computer target tracking, and the method comprises the steps: collecting visible light video data of target motion states in different weathers and different time periods, and carrying out the preprocessing of each frame of image of the visible light video data; a YOLOv8-S model is adopted to predict the preprocessed images, target position and type information is obtained, and the target position and type information obtained through detection of multiple frames of images form a multi-frame target position information sequence; constructing space-time diagrams for the multiple frames of target position information sequences, and stacking the space-time diagrams of continuous T frames into a space-time diagram sequence; a track prediction model is constructed, and track prediction is carried out; optimal matching of prediction and detection is carried out through a cost matrix and a Hungary algorithm, mismatching is filtered through a lightweight verification network, identity consistency judgment and data association are carried out, and a final tracking result is output. According to the invention, stable and continuous tracking of the moving target in a complex scene is realized.

Description

Technical Field

[0001] This application belongs to the field of computer target tracking technology, specifically relating to a target tracking method that combines trajectory prediction. Background Technology

[0002] Target tracking focuses on using data such as images to obtain information such as the position, velocity, and heading of targets in order to monitor their dynamics. It plays an important role in reducing target collision accidents, ensuring navigation safety, and improving shipping efficiency. In existing technologies, target tracking techniques mostly rely on a combination of target detection and data association: on the one hand, target detection algorithms, represented by the YOLO series, can efficiently extract the position information of targets in a single frame, but lack the ability to predict the long-term movement trend of targets, and are prone to detection gaps in situations such as rapid target maneuvering, occlusion, or complex weather conditions; on the other hand, classic multi-target tracking algorithms such as DeepSORT rely on frame-by-frame association of detection results. Although they introduce appearance features to assist in matching, they are prone to identity ID switching and tracking loss problems when targets have similar appearances, are densely distributed, or are briefly occluded.

[0003] In the trajectory prediction stage, traditional methods such as Kalman filtering assume that the target motion is a linear model. However, the actual motion of the target is affected by many factors such as the environment and human operation control, and the trajectory has nonlinear and time-varying characteristics, making it difficult to adapt the prediction accuracy to complex scenarios. In addition, existing technologies are insufficient in modeling the spatiotemporal interaction between multiple targets. The motions of multiple targets have mutual influence or correlation patterns. If such correlations are ignored, the tracking robustness in dense target scenarios will be greatly reduced.

[0004] In summary, how to combine accurate trajectory prediction with effective multi-target spatiotemporal correlation to achieve stable and continuous tracking of moving targets in complex scenarios has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the above-mentioned problems by proposing a target tracking method that combines trajectory prediction with a video moving target tracking method that combines target detection, trajectory prediction, and identity consistency determination. This method can better integrate the advantages of different algorithms, thereby achieving multi-target tracking more accurately and quickly.

[0006] This application provides a target tracking method combining trajectory prediction, including the following steps: S1. Collect visible light video data covering target motion status in different weather conditions and time periods from multiple surveillance cameras, preprocess each frame of the visible light video data to obtain the preprocessed image; S2. Using the YOLOv8-S model as the base model, predict the preprocessed image to obtain target location and type information. Combine the target location and type information obtained from multiple frames of images in the order of the frames to form a multi-frame target location information sequence. S3. Construct a spatiotemporal graph for a multi-frame target location information sequence, and stack the spatiotemporal graphs of consecutive T frames into a spatiotemporal graph sequence; S4. Construct the trajectory prediction model STGCN, which includes three spatiotemporal convolutional blocks. Each spatiotemporal convolutional block includes a temporal convolutional layer, batch normalization, ReLU activation, spatial graph convolutional layer, batch normalization, and ReLU activation in sequence. The network is finally connected to a fully connected layer to perform trajectory prediction. S5. Optimal matching of prediction and detection is performed using the cost matrix and Hungarian algorithm. False matches are filtered out using a lightweight verification network. Identity consistency is determined and data is associated. The final tracking result is then output.

[0007] Furthermore, in step S1, the preprocessing of the visible light video data includes: Perform target detection and annotation, and annotate the bounding boxes of the targets using the YOLO format. ;in, These are the coordinates of the top-left corner of the bounding box. These are the coordinates of the bottom right corner of the bounding box; Spatiotemporal trajectory annotation is performed to record the target's motion state in consecutive frames. ;in, These are the center coordinates of the target at time t, calculated using the bounding box coordinates. The calculation formula is as follows: , ; and These are the velocities in the x and y directions, respectively, calculated using differential methods: ,in, Seconds are the time interval between two adjacent frames. It is the target's heading angle, obtained through target attitude estimation or other auxiliary information.

[0008] Furthermore, the specific steps of step S3 are as follows: S31. For frame t, construct a spatiotemporal graph. ,in For a set of nodes, It is an edge set; S32. Stack the spatiotemporal graphs of 5 consecutive frames into a spatiotemporal graph sequence. , as input to the trajectory prediction model.

[0009] Furthermore, the specific steps of step S31 are as follows: S311. For the t-th frame, construct the node set. Among them, the number of nodes , N t Let t be the number of targets detected in the t-th frame, and t be the number of targets detected per node. Corresponding test results Construct node feature vectors that integrate spatial state, appearance features, and motion features. :

[0010] in, 4-dimensional target location information sequence , For the target appearance features, the features from the previous layer of the classification branch of the YOLOv8-S model in step S2 are used. For the target motion characteristics, Initialize to , Calculate the displacement relative to the previous frame:

[0011] Δcx i t Let Δcy be the displacement of the target in the x-direction. i t Let cx be the displacement of the target in the y-direction. i t Let cx be the x-coordinate of the center of the target in the current frame. i t-1 Let cy be the x-coordinate of the center of the target in the previous frame. i t Let cy be the center y-coordinate of the target in the current frame. i t-1 The center y-coordinate of the target in the previous frame; S312. For the t-th frame, construct the edge set using K-NN. The spatial distance between all nodes is calculated by using the Euclidean distance of the normalized center coordinates.

[0012] For each node Select the K nodes with the smallest distance and establish a directed edge. ; The Gaussian kernel function is used to measure the proximity between nodes, and the weights are... As an adjacency matrix Elements:

[0013] in, This is a bandwidth hyperparameter with a value of 0.1; To avoid feature bias caused by differences in node degree, A is symmetrically normalized to obtain a normalized adjacency matrix:

[0014] in, For degree matrix, For the first The sum of the elements in the row.

[0015] Furthermore, the specific steps of step S4 are as follows: S41. Sequence of spatiotemporal graphs As the network input for the trajectory prediction model, the input dimension is... ; S42. Construct a trajectory prediction model, which includes three spatiotemporal convolutional blocks. Each spatiotemporal convolutional block includes a temporal convolutional layer, batch normalization, ReLU activation, spatial graph convolutional layer, batch normalization, and ReLU activation in sequence. The network is finally connected to a fully connected layer. S43. Employ a smoothed L1 loss, averaging it across all samples and nodes in the training set:

[0016] in, The number of training samples is T, and each sample is a T-frame spatiotemporal graph sequence; The number of nodes in a single sample; Let be the true coordinates of the i-th node in the m-th sample; The predicted coordinates for STGCN; S44. Output the prediction image for the next frame. The core output is the predicted state for each node:

[0017]

[0018] Among them, width and height Use the current frame and ignore changes in target size over a short period of time; If the appearance does not change significantly in a short period of time, the appearance features of the current frame will be used. .

[0019] Furthermore, in step S42: Temporal convolutional layers capture the temporal motion patterns of a single target, employing 1D convolutional kernels. Let the input time-series features be... The convolution kernel is , If the output feature dimension is specified, then the output is:

[0020] Spatial graph convolutional layers aggregate neighbor node features to model interactions between targets, using a normalized adjacency matrix. Graph convolution, let the temporal convolution output be... The spatial convolution weights are , If the spatial convolution output dimension is given, then the output is:

[0021] in, It is the ReLU activation function. The normalized adjacency matrix output in step S312 This is a spatial convolution output; The design of the fully connected layer will incorporate the output of the last temporal convolutional layer. Mapping to predicted coordinates, first take the average over the time dimension. Then, the predicted center coordinates of each node are output through a fully connected layer. : .

[0022] Further, in step S43, the smoothing L1 loss is: .

[0023] Furthermore, the specific steps of step S5 are as follows: S51. Define the prediction map output from step S4 as the prediction set P, containing... One goal, each goal correspond , The tracking ID of the previous frame is used; the target position information sequence output in step S2 is defined as the detection set D, containing... One goal, each goal correspond ; S52. Construct a cost matrix by weighted summation of motion cost and appearance cost, where motion cost is based on the intersection-union ratio of predicted and detected bounding boxes, and appearance cost is based on the cosine similarity between predicted and detected appearances; use the Hungarian algorithm to perform data association between the prediction set and the detection set. S53. Construct a 2-layer MLP identity consistency determination network. The network input is a 5-dimensional feature vector for each association pair. The 5-dimensional feature vector includes the center coordinate difference between the predicted box and the detection box, the aspect ratio difference, and the appearance similarity. The network output is obtained by passing the sigmoid activation function to obtain the confidence score. S54. If the confidence level is ≥0.8, the association is considered reliable. The ID of the detected target is updated to the ID of the corresponding predicted target, and the STGCN prediction result is corrected using the detection result. If the confidence level is ≤0.3, the association is unreliable. If the intersection-union ratio (IU) is <0.1 and the cosine similarity is <0.5, it is considered a new target. A new ID is assigned to it, and node features are initialized. If the above conditions are not met, it is considered a sudden change in appearance caused by occlusion. Step S2 is re-executed, and the IU filtering is disabled. If, after re-executing step S2, the matching confidence level between the detection result and the predicted target is still ≤0.3, and the IU <0.1 and cosine similarity <0.5 are not met, the STGCN prediction result is used to maintain tracking. If no reliable detection result is found for the target in 3 consecutive frames, the target is marked as disappeared and removed from the tracking set. If 0.3 < confidence level <0.8, it is considered a temporary occlusion or detection ambiguity. The STGCN prediction result is adopted, and the ID remains unchanged. S55. For unmatched detected targets, if their confidence level is higher than the normal detection threshold and they are detected in 2 consecutive frames, they are determined to be new targets and assigned a new ID to be added to the tracking set; for unmatched predicted targets, if no detected targets are matched in 3 consecutive frames, they are determined to be disappeared targets and removed from the tracking set. S56. Output the target tracking result of the current frame. The tracking result includes four status indicators: detection update, prediction maintenance, new target, and disappearance.

[0024] Furthermore, the specific steps of step S52 are as follows: S521. Construct the cost matrix, which is a weighted sum of motion cost and appearance cost:

[0025] in, As weight, element Represents the elements of the prediction set With detection set elements The lower the matching cost, the higher the matching degree; Cost of exercise Based on the Intersection over Union (IoU) between the predicted bounding box and the detected bounding box, a larger IoU indicates higher motion consistency and lower cost. If IoU = 0, let... This indicates that the movements are completely inconsistent:

[0026]

[0027] Cost of appearance The cost is based on the cosine similarity between the predicted appearance and the detected appearance; the higher the similarity, the lower the cost.

[0028]

[0029] in, Cosine similarity; S522. Perform outlier filtering, if... , directly Set as This indicates that the match is invalid, and is used to avoid interference from extreme outliers. S523. Solve for the minimum weight matching in a bipartite graph using the Hungarian algorithm to obtain the optimal association pairs. Unmatched predicted targets are marked as pending confirmation, and unmatched detected targets are marked as new target candidates.

[0030] Furthermore, the 5-dimensional feature vector of step S53 is:

[0031] in: ,

[0032] , .

[0033] As can be seen from the above technical solutions, the present invention has the following advantages: This application presents a target tracking method combining trajectory prediction. By constructing a spatiotemporal graph that integrates the target's own state and the interaction relationships between targets, discrete detection results are transformed into a structured node-edge model. This allows the model to learn the interaction patterns of a group of targets, improving the continuity and robustness of tracking in complex scenarios such as dense target movement and occlusion, and avoiding the discontinuity problem that easily occurs in single-target detection. The trajectory prediction model, through the synergistic effect of spatiotemporal convolutional blocks, captures both the target's own temporal motion pattern and aggregates the interaction features of neighboring targets, adapting to the time-varying and nonlinear motion characteristics of targets, and improving the trajectory prediction accuracy in complex scenarios such as dense multi-target movement. At the same time, the smoothed L1 loss function reduces the interference of outliers on training, further ensuring prediction stability. The fusion mechanism of motion cost and appearance cost, combined with the Hungarian algorithm, improves the robustness of data association. Furthermore, a lightweight validation network filters out mismatches, effectively solving the ID jump problem when a target reappears after occlusion. This application achieves stable and continuous tracking of moving targets in complex scenarios. Attached Figure Description

[0034] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating the target tracking method that incorporates trajectory prediction in this application. Detailed Implementation

[0036] Various embodiments of the invention will be described more fully in the specific steps of the target tracking method incorporating trajectory prediction, which will be described in detail below. The invention may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the invention to the specific embodiments disclosed herein, but rather the invention should be understood to cover all modifications, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of the invention.

[0037] It should be understood that, when used in this specification, the term "comprising" indicates the presence of the described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0038] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0039] To make the objectives, features, and advantages of this invention more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solutions protected by this invention. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0040] Please see Figure 1 The diagram shown is a flowchart of a target tracking method combining trajectory prediction in a specific embodiment, including the following steps: S1. Collect visible light video data covering target motion status in different weather conditions and time periods from multiple surveillance cameras, preprocess each frame of the visible light video data to obtain the preprocessed image; It should be noted that collecting visible light videos from multiple environments, weather conditions, and time periods can cover various real-world scenarios for target tracking, avoiding the limitations of model adaptation caused by single data. S2. Using the YOLOv8-S model as the base model, predict the preprocessed image to obtain target location and type information. Combine the target location and type information obtained from multiple frames of images in the order of the frames to form a multi-frame target location information sequence. It should be noted that the YOLOv8-S model balances detection speed and accuracy, meets the real-time requirements of target tracking, and avoids affecting tracking efficiency due to detection lag. S3. Construct a spatiotemporal graph for a multi-frame target location information sequence, and stack the spatiotemporal graphs of consecutive T frames into a spatiotemporal graph sequence; It should be noted that converting discrete multi-frame target position information into a spatiotemporal graph with a node-edge structure can simultaneously model the target's own state and the spatial interaction between targets, adapting to multi-target coexistence scenarios; stacking multi-frame spatiotemporal graphs to form a sequence can preserve the target motion temporal information. S4. Construct the trajectory prediction model STGCN, which includes three spatiotemporal convolutional blocks. Each spatiotemporal convolutional block includes a temporal convolutional layer, batch normalization, ReLU activation, spatial graph convolutional layer, batch normalization, and ReLU activation in sequence. The network is finally connected to a fully connected layer to perform trajectory prediction. It should be noted that the temporal convolutional layer can capture temporal motion patterns such as target acceleration and turning, while the spatial convolutional layer can aggregate features of neighboring targets and model multi-target interactions; batch normalization and ReLU ensure network training stability, and the fully connected layer accurately outputs predicted positions, providing reliable trajectory prediction for complex scenarios such as dense take-off and landing of multiple targets. S5. Optimal matching of prediction and detection is performed using the cost matrix and Hungarian algorithm. False matches are filtered out using a lightweight verification network. Identity consistency is determined and data is associated. The final tracking result is then output. It should be noted that the combination of the cost matrix and the Hungarian algorithm can achieve the optimal matching between prediction and detection, adapt to multi-target coexistence scenarios, and improve the accuracy of data association; the lightweight verification network can effectively filter out false matches and solve the ID jump problem when the target reappears after being occluded.

[0041] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, in order to fully illustrate the specific implementation process of this embodiment, another target tracking method combining trajectory prediction is provided, the method including: S1. Collect visible light video data covering target motion states under different weather conditions and time periods from multiple surveillance cameras, and preprocess each frame of the visible light video data to obtain a preprocessed image; Step S1, the preprocessing of the visible light video data includes: Perform target detection and annotation, and annotate the bounding boxes of the targets using the YOLO format. ;in, These are the coordinates of the top-left corner of the bounding box. These are the coordinates of the bottom right corner of the bounding box; Spatiotemporal trajectory annotation is performed to record the target's motion state in consecutive frames. ;in, These are the center coordinates of the target at time t, calculated using the bounding box coordinates. The calculation formula is as follows: , ; and These are the velocities in the x and y directions, respectively, calculated using differential methods: ,in, Seconds are the time interval between two adjacent frames. It is the target's heading angle, obtained through target attitude estimation or other auxiliary information; S2. Using the YOLOv8-S model as the base model, predict the preprocessed image to obtain target location and type information. Combine the target location and type information obtained from multiple frames of images in the order of the frames to form a multi-frame target location information sequence. It should be noted that the YOLOv8-S model's accurate detection capabilities are used to sequentially detect multiple pre-processed images, forming a continuous sequence of target location and type information, providing accurate and continuous initial observations for subsequent tracking.

[0042] S3. Construct a spatiotemporal graph for the multi-frame target location information sequence, and stack the spatiotemporal graphs of consecutive T frames into a spatiotemporal graph sequence; the specific steps of step S3 are as follows: S31. For frame t, construct a spatiotemporal graph. ,in For a set of nodes, For the edge set; the specific steps of step S31 are as follows: S311. For the t-th frame, construct the node set. Among them, the number of nodes , N t Let t be the number of targets detected in the t-th frame, and t be the number of targets detected per node. Corresponding test results Construct node feature vectors that integrate spatial state, appearance features, and motion features. :

[0043] in, 4-dimensional target location information sequence , For the target appearance features, the features from the previous layer of the classification branch of the YOLOv8-S model in step S2 are used. For the target motion characteristics, Initialize to , Calculate the displacement relative to the previous frame:

[0044] Δcx i t Let Δcy be the displacement of the target in the x-direction. i t Let cx be the displacement of the target in the y-direction. i t Let cx be the x-coordinate of the center of the target in the current frame. i t-1 Let cy be the x-coordinate of the center of the target in the previous frame. i t Let cy be the center y-coordinate of the target in the current frame. i t-1 The center y-coordinate of the target in the previous frame; S312. For the t-th frame, construct the edge set using K-NN. The spatial distance between all nodes is calculated by using the Euclidean distance of the normalized center coordinates.

[0045] For each node Select the K nodes with the smallest distance and establish a directed edge. ; The Gaussian kernel function is used to measure the proximity between nodes, and the weights are... As an adjacency matrix Elements:

[0046] in, This is a bandwidth hyperparameter with a value of 0.1; To avoid feature bias caused by differences in node degree, A is symmetrically normalized to obtain a normalized adjacency matrix:

[0047] in, For degree matrix, For the first The sum of the elements in the row.

[0048] It should be noted that K=3 is used to balance the granularity of interaction modeling with the amount of computation. S32. Stack the spatiotemporal graphs of 5 consecutive frames into a spatiotemporal graph sequence. , as input to the trajectory prediction model.

[0049] S4. Construct the STGCN trajectory prediction model, which includes three spatiotemporal convolutional blocks. Each spatiotemporal convolutional block sequentially includes a temporal convolutional layer, batch normalization, ReLU activation, a spatial graph convolutional layer, batch normalization, and ReLU activation. The network is finally connected to a fully connected layer for trajectory prediction. The specific steps of step S4 are as follows: S41. Sequence of spatiotemporal graphs As the network input for the trajectory prediction model, the input dimension is... ; For example, select For time step, For the number of nodes, For node feature dimensions; S42. Construct a trajectory prediction model, including three spatiotemporal convolutional blocks. Each spatiotemporal convolutional block sequentially includes a temporal convolutional layer, batch normalization, ReLU activation, a spatial graph convolutional layer, batch normalization, and ReLU activation. The network is finally connected to a fully connected layer. In step S42: Temporal convolutional layers capture the temporal motion patterns of a single target, employing 1D convolutional kernels. Let the input time-series features be... The convolution kernel is , If the output feature dimension is specified, then the output is:

[0050] Spatial graph convolutional layers aggregate neighbor node features to model interactions between targets, using a normalized adjacency matrix. Graph convolution, let the temporal convolution output be... The spatial convolution weights are , If the spatial convolution output dimension is given, then the output is:

[0051] in, It is the ReLU activation function. The normalized adjacency matrix output in step S312 This is a spatial convolution output; The design of the fully connected layer will incorporate the output of the last temporal convolutional layer. Mapping to predicted coordinates, first take the average over the time dimension. Then, the predicted center coordinates of each node are output through a fully connected layer. : .

[0052] For example, parameter settings: kernel size of the 1D convolution kernel. stride=1, padding=1, used to ensure the time dimension remains unchanged; input Output Number of convolution kernels = 256, batch normalization parameters , ; , Batch normalized parameters are convolved at the same time; Bias term .

[0053] S43. Employ a smoothed L1 loss, averaging it across all samples and nodes in the training set:

[0054] in, The number of training samples is T, and each sample is a T-frame spatiotemporal graph sequence; The number of nodes in a single sample; Let be the true coordinates of the i-th node in the m-th sample; Here are the predicted coordinates for STGCN; in step S43, the smoothing L1 loss is: .

[0055] S44. Output the prediction image for the next frame. The core output is the predicted state for each node:

[0056]

[0057] Among them, width and height Use the current frame and ignore changes in target size over a short period of time; If the appearance does not change significantly in a short period of time, the appearance features of the current frame will be used. .

[0058] S5. Optimal matching of prediction and detection is performed using the cost matrix and the Hungarian algorithm. False matches are filtered out using a lightweight validation network. Identity consistency is determined and data is associated, outputting the final tracking result. The specific steps of step S5 are as follows: S51. Define the prediction map output from step S4 as the prediction set P, containing... One goal, each goal correspond , The tracking ID of the previous frame is used; the target position information sequence output in step S2 is defined as the detection set D, containing... One goal, each goal correspond ; S52. Construct a cost matrix by weighted summation of motion cost and appearance cost, where motion cost is based on the intersection-union ratio of predicted and detected bounding boxes, and appearance cost is based on the cosine similarity between predicted and detected appearances; use the Hungarian algorithm to perform data association between the prediction and detection sets; the specific steps of step S52 are as follows: S521. Construct the cost matrix, which is a weighted sum of motion cost and appearance cost:

[0059] in, As weight, element Represents the elements of the prediction set With detection set elements The lower the matching cost, the higher the matching degree; Cost of exercise Based on the Intersection over Union (IoU) between the predicted bounding box and the detected bounding box, a larger IoU indicates higher motion consistency and lower cost. If IoU = 0, let... This indicates that the movements are completely inconsistent:

[0060]

[0061] Cost of appearance The cost is based on the cosine similarity between the predicted appearance and the detected appearance; the higher the similarity, the lower the cost.

[0062]

[0063] in, Cosine similarity; For example, weight Take 0.6; S522. Perform outlier filtering, if... , directly Set as This indicates that the match is invalid, and is used to avoid interference from extreme outliers. S523. Solve for the minimum weight matching in a bipartite graph using the Hungarian algorithm to obtain the optimal association pairs. Unmatched predicted targets are marked as pending confirmation, and unmatched detected targets are marked as new target candidates.

[0064] S53. Construct a 2-layer MLP identity consistency determination network. The network input is a 5-dimensional feature vector for each association pair. The 5-dimensional feature vector includes the center coordinate difference between the predicted box and the detected box, the aspect ratio difference, and the appearance similarity. The network output is used to obtain the confidence score through the Sigmoid activation function. The 5-dimensional feature vector in step S53 is:

[0065] in: ,

[0066] , .

[0067] S54. If the confidence level is ≥0.8, the association is considered reliable. The ID of the detected target is updated to the ID of the corresponding predicted target, and the STGCN prediction result is corrected using the detection result. If the confidence level is ≤0.3, the association is unreliable. If the intersection-union ratio (IU) is <0.1 and the cosine similarity is <0.5, it is considered a new target. A new ID is assigned to it, and node features are initialized. If the above conditions are not met, it is considered a sudden change in appearance caused by occlusion. Step S2 is re-executed, and the IU filtering is disabled. If, after re-executing step S2, the matching confidence level between the detection result and the predicted target is still ≤0.3, and the IU <0.1 and cosine similarity <0.5 are not met, the STGCN prediction result is used to maintain tracking. If no reliable detection result is found for the target in 3 consecutive frames, the target is marked as disappeared and removed from the tracking set. If 0.3 < confidence level <0.8, it is considered a temporary occlusion or detection ambiguity. The STGCN prediction result is adopted, and the ID remains unchanged. For example, if the confidence level is ≥0.8, update the node features: ; S55. For unmatched detected targets, if their confidence level is higher than the normal detection threshold and they are detected in 2 consecutive frames, they are determined to be new targets and assigned a new ID to be added to the tracking set; for unmatched predicted targets, if no detected targets are matched in 3 consecutive frames, they are determined to be disappeared targets and removed from the tracking set. S56. Output the target tracking result of the current frame. The tracking result includes four status indicators: detection update, prediction maintenance, new target, and disappearance.

[0068] For example, the network structure is a 2-layer MLP, with 5-dimensional input → 32-dimensional hidden layer → ReLU → 1-dimensional output → Sigmoid; Final output Final frame tracking results , For detecting updates, predicting maintenance, new targets, and disappearance; It should be noted that the method described above in this application is also suitable for parallel processing. For example, one computing device can be used to run YOLOv8-S or other high-quality object detection models, while another computing device can be used to run STGCN. The results of the former can then be transmitted via network communication to verify the results of the latter algorithm. When using the method, the results of STGCN can be used as the prediction results. As a graph network, STGCN runs much faster than object detection algorithms, thus its tracking speed can be significantly improved.

[0069] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0070] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method combining trajectory prediction, characterized in that, The method includes the following steps: S1. Collect visible light video data covering target motion status in different weather conditions and time periods from multiple surveillance cameras, preprocess each frame of the visible light video data to obtain the preprocessed image; S2. Using the YOLOv8-S model as the base model, predict the preprocessed image to obtain target location and type information. Combine the target location and type information obtained from multiple frames of images in the order of the frames to form a multi-frame target location information sequence. S3. Construct a spatiotemporal graph for a multi-frame target location information sequence, and stack the spatiotemporal graphs of consecutive T frames into a spatiotemporal graph sequence; S4. Construct the trajectory prediction model STGCN, which includes three spatiotemporal convolutional blocks. Each spatiotemporal convolutional block includes a temporal convolutional layer, batch normalization, ReLU activation, spatial graph convolutional layer, batch normalization, and ReLU activation in sequence. The network is finally connected to a fully connected layer to perform trajectory prediction. S5. Optimal matching of prediction and detection is performed using the cost matrix and Hungarian algorithm. False matches are filtered out using a lightweight verification network. Identity consistency is determined and data is associated. The final tracking result is then output.

2. The target tracking method combining trajectory prediction according to claim 1, characterized in that, Step S1, the preprocessing of the visible light video data includes: Perform target detection and annotation, and annotate the bounding boxes of the targets using the YOLO format. ;in, These are the coordinates of the top-left corner of the bounding box. These are the coordinates of the bottom right corner of the bounding box; Spatiotemporal trajectory annotation is performed to record the target's motion state in consecutive frames. ;in, These are the center coordinates of the target at time t, calculated using the bounding box coordinates. The calculation formula is as follows: , ; and These are the velocities in the x and y directions, respectively, calculated using differential methods: ,in, Seconds are the time interval between two adjacent frames. It is the target's heading angle, obtained through target attitude estimation or other auxiliary information.

3. The target tracking method combining trajectory prediction according to claim 1, characterized in that, The specific steps of step S3 are as follows: S31. For frame t, construct a spatiotemporal graph. ,in For a set of nodes, It is an edge set; S32. Stack the spatiotemporal graphs of 5 consecutive frames into a spatiotemporal graph sequence. , as input to the trajectory prediction model.

4. The target tracking method combining trajectory prediction according to claim 3, characterized in that, The specific steps of step S31 are as follows: S311. For the t-th frame, construct the node set. Among them, the number of nodes , N t Let t be the number of targets detected in the t-th frame, and t be the number of targets detected per node. Corresponding test results Construct node feature vectors that integrate spatial state, appearance features, and motion features. : in, 4-dimensional target location information sequence , For the target appearance features, the features from the previous layer of the classification branch of the YOLOv8-S model in step S2 are used. For the target motion characteristics, Initialize to , Calculate the displacement relative to the previous frame: Δcx i t Let Δcy be the displacement of the target in the x-direction. i t Let cx be the displacement of the target in the y-direction. i t Let cx be the x-coordinate of the center of the target in the current frame. i t-1 Let cy be the x-coordinate of the center of the target in the previous frame. i t Let cy be the center y-coordinate of the target in the current frame. i t-1 The center y-coordinate of the target in the previous frame; S312. For the t-th frame, construct the edge set using K-NN. The spatial distance between all nodes is calculated by using the Euclidean distance of the normalized center coordinates. For each node Select the K nodes with the smallest distance and establish a directed edge. ; The Gaussian kernel function is used to measure the proximity between nodes, and the weights are... As an adjacency matrix Elements: in, This is a bandwidth hyperparameter with a value of 0.1; To avoid feature bias caused by differences in node degree, A is symmetrically normalized to obtain a normalized adjacency matrix: in, For degree matrix, For the first The sum of the elements in the row.

5. The target tracking method combining trajectory prediction according to claim 1, characterized in that, The specific steps of step S4 are as follows: S41. Sequence of spatiotemporal graphs As the network input for the trajectory prediction model, the input dimension is... ; S42. Construct a trajectory prediction model, which includes three spatiotemporal convolutional blocks. Each spatiotemporal convolutional block includes a temporal convolutional layer, batch normalization, ReLU activation, spatial graph convolutional layer, batch normalization, and ReLU activation in sequence. The network is finally connected to a fully connected layer. S43. Employ a smoothed L1 loss, averaging it across all samples and nodes in the training set: in, The number of training samples is T, and each sample is a T-frame spatiotemporal graph sequence; The number of nodes in a single sample; Let be the true coordinates of the i-th node in the m-th sample; The predicted coordinates for STGCN; S44. Output the prediction image for the next frame. The core output is the predicted state for each node: Among them, width and height Use the current frame and ignore changes in target size over a short period of time; If the appearance does not change significantly in a short period of time, the appearance features of the current frame will be used. .

6. The target tracking method combining trajectory prediction according to claim 5, characterized in that, In step S42: Temporal convolutional layers capture the temporal motion patterns of a single target, employing 1D convolutional kernels. Let the input time-series features be... The convolution kernel is , If the output feature dimension is specified, then the output is: Spatial graph convolutional layers aggregate neighbor node features to model interactions between targets, using a normalized adjacency matrix. Graph convolution, let the temporal convolution output be... The spatial convolution weights are , If the spatial convolution output dimension is given, then the output is: in, It is the ReLU activation function. The normalized adjacency matrix output in step S312 This is a spatial convolution output; The design of the fully connected layer will incorporate the output of the last temporal convolutional layer. Mapping to predicted coordinates, first take the average over the time dimension. Then, the predicted center coordinates of each node are output through a fully connected layer. : 。 7. The target tracking method combining trajectory prediction according to claim 5, characterized in that, In step S43, the smoothing L1 loss is: 。 8. The target tracking method combining trajectory prediction according to claim 1, characterized in that, The specific steps of step S5 are as follows: S51. Define the prediction map output from step S4 as the prediction set P, containing... One goal, each goal correspond , The tracking ID of the previous frame is used; the target position information sequence output in step S2 is defined as the detection set D, containing... One goal, each goal correspond ; S52. Construct a cost matrix by weighted summation of motion cost and appearance cost, where motion cost is based on the intersection-union ratio of predicted and detected bounding boxes, and appearance cost is based on the cosine similarity between predicted and detected appearances; use the Hungarian algorithm to perform data association between the prediction set and the detection set. S53. Construct a 2-layer MLP identity consistency determination network. The network input is a 5-dimensional feature vector for each association pair. The 5-dimensional feature vector includes the center coordinate difference between the predicted box and the detection box, the aspect ratio difference, and the appearance similarity. The network output is obtained by passing the sigmoid activation function to obtain the confidence score. S54. If the confidence level is ≥0.8, the association is considered reliable. The ID of the detected target is updated to the ID of the corresponding predicted target, and the STGCN prediction result is corrected using the detection result. If the confidence level is ≤0.3, the association is unreliable. If the intersection-union ratio (IU) is <0.1 and the cosine similarity is <0.5, it is considered a new target. A new ID is assigned to it, and node features are initialized. If the above conditions are not met, it is considered a sudden change in appearance caused by occlusion. Step S2 is re-executed, and the IU filtering is disabled. If, after re-executing step S2, the matching confidence level between the detection result and the predicted target is still ≤0.3, and the IU <0.1 and cosine similarity <0.5 are not met, the STGCN prediction result is used to maintain tracking. If no reliable detection result is found for the target in 3 consecutive frames, the target is marked as disappeared and removed from the tracking set. If 0.3 < confidence level <0.8, it is considered a temporary occlusion or detection ambiguity. The STGCN prediction result is adopted, and the ID remains unchanged. S55. For unmatched detected targets, if their confidence level is higher than the normal detection threshold and they are detected in 2 consecutive frames, they are determined to be new targets and assigned a new ID to be added to the tracking set; for unmatched predicted targets, if no detected targets are matched in 3 consecutive frames, they are determined to be disappeared targets and removed from the tracking set. S56. Output the target tracking result of the current frame. The tracking result includes four status indicators: detection update, prediction maintenance, new target, and disappearance.

9. The target tracking method combining trajectory prediction according to claim 8, characterized in that, The specific steps of step S52 are as follows: S521. Construct the cost matrix, which is a weighted sum of motion cost and appearance cost: in, As weight, element Represents the elements of the prediction set With detection set elements The lower the matching cost, the higher the matching degree; Cost of exercise Based on the Intersection over Union (IoU) between the predicted bounding box and the detected bounding box, a larger IoU indicates higher motion consistency and lower cost. If IoU = 0, let... This indicates that the movements are completely inconsistent: Cost of appearance The cost is based on the cosine similarity between the predicted appearance and the detected appearance; the higher the similarity, the lower the cost. in, Cosine similarity; S522. Perform outlier filtering, if... , directly Set as This indicates that the match is invalid, and is used to avoid interference from extreme outliers. S523. Solve for the minimum weight matching in a bipartite graph using the Hungarian algorithm to obtain the optimal association pairs. Unmatched predicted targets are marked as pending confirmation, and unmatched detected targets are marked as new target candidates.

10. The target tracking method combining trajectory prediction according to claim 1, characterized in that, The 5-dimensional feature vector of step S53 is: in: ， ， 。

Images