A multi-target tracking system for highway scenes based on UAV visual perception

Abstract

The present invention relates to the field of traffic management technology, and in particular to a highway scene multi-target tracking system based on unmanned aerial vehicle (UAV) visual perception. Through high-definition cameras and 5G communication technology, the system achieves efficient acquisition and low-latency return of highway scene videos, providing a reliable data basis for subsequent processing. Data preprocessing and flight control are used to improve video quality and flight efficiency, and the UAV path is dynamically adjusted based on the motion trajectory analysis results of key dynamic targets. Key dynamic targets are acquired through a one-shot detector model and optical flow method, and effective tracking is achieved by combining a deep learning algorithm with multi-target tracking technology, thereby improving target detection accuracy and tracking stability. The motion trajectories of key dynamic targets are analyzed and the motion trajectory analysis results of the key dynamic targets are visualized to intuitively display the motion trajectories of the key dynamic targets.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic management, and in particular to a highway scene multi-target tracking system based on unmanned aerial vehicle (UAV) visual perception. Background Art

[0002] With the acceleration of urbanization and the increase in traffic volume, traditional traffic management methods are unable to adapt to the new situation. UAV technology provides an efficient and intelligent solution for traffic management by collecting data in real time and performing processing and analysis. The current application of UAV technology in the field of traffic management does face many challenges: because UAVs need to collect video data in real time, and conventional target detection algorithms have limited processing speed, it is difficult to efficiently process massive video frames in a short period of time, which greatly limits its application effectiveness. At the same time, in complex and changing highway scenes, the definition and identification standards of abnormal behavior are still vague and cannot provide convenience for traffic management. In addition, there is a lack of effective integration and analysis methods for the massive amount of video data and motion trajectory information collected, making it difficult to achieve in-depth data mining and application, resulting in a waste of resources. To this end, the present invention proposes a highway scene multi-target tracking system based on UAV visual perception. Summary of the Invention

[0003] The purpose of the present invention is to solve the problems in the background technology and propose a highway scene multi-target tracking system based on drone visual perception.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A multi-target tracking system for highway scenes based on UAV visual perception, including: a UAV platform, a ground control station, a target detection and tracking module, and a motion trajectory analysis and visualization module;

[0006] UAV platform: collects highway scene videos in real time and transmits the collected video data back. The UAV platform includes an image acquisition unit and a data transmission unit.

[0007] Ground control station: processes the video data transmitted by the drone and generates flight control instructions to dynamically adjust the path. The ground control station includes a data pre-processing unit and a flight control unit;

[0008] Target detection and tracking module: This module uses a one-shot detector model to extract key dynamic target features in highway scenes and perform cross-frame matching statistics. It also combines the optical flow method to analyze the motion vectors of static objects to offset drone interference and obtain the true motion information of key dynamic targets. It uses a deep learning algorithm to detect targets in video frames captured by drones. It uses a multi-target tracking algorithm to track the detected key dynamic targets and obtain their motion trajectories. The target detection and tracking module includes a key dynamic target acquisition unit, a key dynamic target association unit, a motion compensation unit, a key dynamic target detection unit, and a key dynamic target tracking unit.

[0009] Motion trajectory analysis and visualization module: Analyzes the motion trajectory of key dynamic targets and visualizes the motion trajectory analysis results of key dynamic targets; the motion trajectory analysis and visualization module includes a motion trajectory analysis unit and a visualization unit.

[0010] Furthermore, the image acquisition unit is equipped with a 4K high-definition camera and a gimbal stabilization system to collect real-time video data of road scenes; the data return unit uses 5G communication technology for low-latency transmission of video data.

[0011] Furthermore, the data preprocessing unit is used to perform denoising, illumination balance and geometric correction on the received video data; the flight control unit is used to control the flight of the UAV and dynamically adjust the flight path according to the motion trajectory analysis results of key dynamic targets.

[0012] Furthermore, the key dynamic target acquisition unit is used to obtain preprocessed video data, use one-shot detector model training to extract key features, and identify and count key dynamic targets in highway scenes.

[0013] Furthermore, the key dynamic target association unit is used to use the structural similarity index to determine whether the key dynamic targets of consecutive frames are the same target, and calculate the similarity of the key dynamic targets between the key dynamic targets of the current frame and the key dynamic targets of the next frame; if the value of the structural similarity index of the key dynamic targets in the consecutive frames exceeds the preset cross-frame structural similarity matching threshold, they are determined to be the same target.

[0014] Furthermore, the motion compensation unit is used to calculate the motion vector of pixels in the image between consecutive frames for static objects in the highway scene using the optical flow method, and fuse the average and mode weighted average of the moving direction and speed of key pixels to obtain the true pixel motion vector of the key dynamic target.

[0015] Furthermore, the key dynamic target detection unit is used to extract features from the input video frames using the optimized YOLO-X network structure. Combining temporal feature fusion and dynamic threshold adjustment strategy, it performs fine-grained detection on the identified key dynamic targets. The process of identifying the key dynamic target types includes:

[0016] After the YOLO backbone network performs preliminary feature extraction, a dynamic feature selection mechanism is constructed based on target scale difference analysis. This mechanism is used to adaptively assign feature weights of different levels according to the scale of key dynamic targets to generate a preliminary target feature map. Based on the generated preliminary target feature map, it is combined with the temporal characteristics of the video sequence. Specifically, the target feature map of each frame is regarded as a temporal node in the temporal feature map. At the same time, the target feature relationship between adjacent frames is defined as a temporal edge in the temporal feature map. The feature vectors of key dynamic targets are used as the attributes of the temporal nodes in the temporal feature map, and the relative position relationship between key dynamic targets is used as the weight of the temporal edge in the temporal feature map to capture the feature information and spatial relationship of key feature targets in the temporal sequence.

[0017] After the construction of the temporal feature graph is completed, it is input into the graph neural network for processing, where the graph neural network operates on the temporal nodes and temporal edges in the temporal feature graph; the graph neural network learns the temporal feature associations between key dynamic targets in different frames, and encodes the dynamic change information of the key dynamic targets in the temporal sequence into the feature representation; through multi-layer graph neural network processing, the feature representation of the fused temporal information is obtained, where the feature representation contains the dynamic change information of the key dynamic targets in the entire temporal sequence; the splicing operation is used to fuse the feature representation of the fused temporal information output by the graph neural network with the target feature graph of the current frame to generate a target feature graph after the fusion of the temporal information; based on the target feature graph after the fusion of the temporal information, the type of key dynamic target is identified.

[0018] Furthermore, the key dynamic target tracking unit is used to receive the output of the key dynamic target detection unit, construct an incremental dynamic feature memory, dynamically adjust the threshold based on the scene density, implement trajectory association and initialization through the memory library, and manage short-term and long-term motion trajectories. The process of generating a unique ID and motion trajectory of the key dynamic target includes:

[0019] For each identified key dynamic target, the acceleration change rate and detection confidence of consecutive A frames are calculated and combined into a dynamic attribute vector. A frame is the length of the dynamic attribute calculation window, which is used to calculate the number of consecutive frames of dynamic attributes.

[0020] Extract the appearance feature descriptors of key dynamic targets and store them in association with the dynamic attribute vector. Create a memory pool that only stores the dynamic attributes and appearance feature descriptors of the most recent G motion trajectory segments. When a new key dynamic target enters the memory pool, if the memory pool is full, remove the oldest dynamic attributes according to the FIFO rule. G is the maximum number of trajectory segments that can be stored in the memory pool.

[0021] Count the number of key dynamic targets in the current highway scene and estimate the density of their distribution. That is, calculate the highway scene density factor based on the number of key dynamic targets in the current frame and the area of ​​the highway scene.

[0022] Based on the road scene density factor, adjust the basic association matching threshold and the basic detection confidence threshold;

[0023] Motion trajectory association and initialization based on the memory pool: Each key dynamic target in the current frame is compared with the motion trajectory segments in the memory pool, and the association cost is calculated. That is, the Mahalanobis distance and appearance distance are weightedly fused into the association cost;

[0024] If the key dynamic target satisfies the association cost less than the basic association matching threshold and the detection confidence is greater than the basic detection confidence threshold for N consecutive frames, then the key dynamic target is considered to belong to this motion trajectory, a unique ID is assigned to it, and the motion trajectory is formally initialized to generate an initial motion trajectory with a unique ID; where N frames is the length of the motion trajectory initialization window, which is the number of consecutive matching frames used for motion trajectory initialization;

[0025] Perform motion trajectory lifecycle management for the initial motion trajectory, including short-term and long-term trajectory management. Detection results that are not associated with any motion trajectory fragment in the memory pool are managed as short-term motion trajectories; key dynamic targets that are successfully associated with any motion trajectory fragment in the memory pool are managed as long-term motion trajectories.

[0026] After completing the motion trajectory lifecycle management, the unique ID and motion trajectory route of the key dynamic target are output.

[0027] Furthermore, the motion trajectory analysis unit is used to perform spatial and temporal enhancement processing on the motion trajectory data output by multi-target tracking, fuse appearance features and dynamic attributes to generate multimodal motion trajectory features, and extract initial behavior pattern labels based on the motion trajectory point sequence; initialize the behavior graph based on traffic rules, combine historical abnormal motion trajectory clustering to mine new abnormal behavior nodes, and dynamically update the graph structure to form a dynamic behavior graph containing normal and abnormal patterns; use the spatiotemporal attention Transformer to encode motion trajectory features, combine the dynamic behavior graph to calculate the anomaly score and path deviation, comprehensively judge the trajectory abnormality and map it into a natural language description, and finally classify it according to the degree of abnormality; the specific process is as follows:

[0028] Obtain the motion trajectory data output by the multi-target tracking unit, including the unique ID, motion trajectory point sequence, appearance feature identifier, and dynamic attribute vector; enhance the motion trajectory points, appearance feature identifier, and dynamic attribute vector of key dynamic targets and fuse them into multimodal motion trajectory features;

[0029] The method involves spatially interpolating the motion trajectory points to generate a smooth motion trajectory curve and obtain a spatially enhanced motion trajectory point sequence. A timestamp sequence is introduced to calculate the instantaneous velocity, acceleration, azimuth angle, and acceleration change rate of the motion trajectory points for temporal enhancement. A time-series weighted average is performed on the appearance feature vectors of the same target in the motion trajectory to generate an enhanced appearance feature representation. Based on the motion trajectory point sequence, behavioral pattern labels are extracted as initial semantic annotations.

[0030] An initial graph is constructed based on traffic rules. In this initial graph, behavior state nodes are the core components, representing various traffic behavior states, while state transition edges are used to represent the transition relationship between traffic behavior states.

[0031] Obtain a historical motion trajectory database, cluster abnormal motion trajectories in the historical motion trajectory database, and extract abnormal behavior patterns; add the mined abnormal patterns as new behavior state nodes to the initial graph and initialize the transition probability; receive new motion trajectory data in real time, dynamically update the graph structure, and thus construct a dynamic behavior graph;

[0032] A spatiotemporal attention transformer is introduced to encode multimodal motion trajectory features. The encoded multimodal motion trajectory features are compared with the characteristic distribution of normal behavior patterns in the dynamic behavior map to calculate the anomaly score. The path deviation of the multimodal motion trajectory on the dynamic behavior map is calculated.

[0033] The anomaly score and the graph deviation are combined to determine whether the motion trajectory of the key dynamic target is abnormal; the path of the abnormal motion trajectory on the dynamic behavior graph is mapped into a natural language description; and a weighted calculation is performed based on the anomaly score and the graph deviation to obtain the anomaly level, including minor anomaly, severe anomaly, and dangerous anomaly.

[0034] Furthermore, the visualization unit is used to overlay and display the motion trajectory analysis results and abnormality level classification of key dynamic targets on the electronic map in real time.

[0035] Compared with the existing technology, the beneficial effects of the present invention are: by real-time collection of highway scene videos and low-latency return transmission, the timeliness and integrity of monitoring data are ensured, providing a reliable basis for subsequent analysis; by preprocessing the video data, the data quality is effectively improved, and the drone path is dynamically adjusted according to the target motion trajectory, significantly improving the accuracy and efficiency of monitoring; by combining the one-shot detector model and the optical flow method, the key dynamic target features are quickly and accurately extracted and the drone interference is offset, the real motion information of the target is obtained, and the robustness of target detection is enhanced; by combining the deep learning algorithm with multi-target tracking technology, the motion trajectory of the key dynamic target is obtained, providing reliable data for subsequent trajectory analysis; by analyzing the motion trajectory of the key dynamic target, the abnormality of the motion trajectory is judged, and graded processing is performed, which helps to timely discover and respond to potential risks. At the same time, the motion trajectory analysis results are visualized, so that monitoring personnel can intuitively understand the target motion pattern, thereby improving the intelligence level of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a module diagram of a highway scene multi-target tracking system based on UAV visual perception proposed by the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the implementation regulations described are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] Reference Figure 1 , a multi-target tracking system for highway scenes based on UAV visual perception, the system includes a UAV platform, a ground control station, a target detection and tracking module, and a motion trajectory analysis and visualization module;

[0039] Drone platform: collects real-time video of highway scenes and transmits the collected video data back. The drone platform includes an image acquisition unit and a data return unit. The image acquisition unit is equipped with a 4K high-definition camera and a gimbal stabilization system to collect real-time video data of highway scenes. The data return unit uses 5G communication technology for low-latency transmission of video data.

[0040] Ground control station: This processes the video data transmitted by the drone and generates flight control commands to dynamically adjust the flight path. The ground control station includes a data preprocessing unit and a flight control unit. The data preprocessing unit is used to perform noise reduction, illumination balance, and geometric correction on the received video data. The flight control unit is used to control the drone's flight and dynamically adjust the flight path based on the motion trajectory analysis results of key dynamic targets.

[0041] Target detection and tracking module: This module uses a one-shot detector model to extract key dynamic target features in highway scenes and perform cross-frame matching statistics. It also combines the optical flow method to analyze the motion vectors of static objects to offset drone interference and obtain the true motion information of key dynamic targets. It uses a deep learning algorithm to detect targets in video frames captured by drones. It uses a multi-target tracking algorithm to track the detected key dynamic targets and obtain their motion trajectories. The target detection and tracking module includes a key dynamic target acquisition unit, a key dynamic target association unit, a motion compensation unit, a key dynamic target detection unit, and a key dynamic target tracking unit.

[0042] Motion trajectory analysis and visualization module: Analyzes the motion trajectory of key dynamic targets and visualizes the motion trajectory analysis results of key dynamic targets; the motion trajectory analysis and visualization module includes a motion trajectory analysis unit and a visualization unit.

[0043] It should be further explained that, in the specific implementation process, the key dynamic target acquisition unit is used to obtain pre-processed video data, use the one-shot detector model training to extract key features, and identify and count the key dynamic targets in the highway scene. ,in represents the key dynamic target, represents the number of key dynamic targets; the specific process is: input the preprocessed video frame by frame into the one-shot detector model, and use the model to quickly adapt to the target detection task in the road scene; directly generate candidate target areas through the model; combine with a small number of target annotation samples to quickly identify potential key dynamic targets in the candidate target area; it can be understood that the one-shot detector model is a deep learning model that can quickly learn and identify target features with only a small number of samples (even a single sample), and is used to efficiently extract the features of key dynamic targets in video data.

[0044] It should be further explained that, in the specific implementation process, the key dynamic target association unit is used to use the structural similarity index to determine whether the key dynamic targets of consecutive frames are the same target. The key dynamic target of the next frame ( are all key dynamic target indexes), calculate the similarity of key dynamic targets , and its calculation formula is:

[0045] , where Represents brightness, contrast and structural similarity respectively; are preset weight coefficients corresponding to brightness, contrast, and structure respectively. The structural similarity index is used to measure image similarity from the perspectives of brightness, contrast, and structure.

[0046] If the value of the structural similarity index of the key dynamic targets in consecutive frames exceeds the preset cross-frame structural similarity matching threshold, they are judged to be the same target; among which, the cross-frame structural similarity matching threshold is used to measure the judgment criteria of whether the key dynamic targets in consecutive frames are the same entity.

[0047] It should be further explained that, in the specific implementation process, the motion compensation unit is used to calculate the motion vector of the pixels in the image between consecutive frames for the static objects in the highway scene using the optical flow method, and fuse the average and mode weighted average of the moving direction and speed of the key pixels to offset the impact of the UAV's own motion on the target tracking, and obtain the true pixel motion vector of the key dynamic target, that is, the offset value of the moving direction and speed of the key dynamic target in the highway scene. The two are combined to obtain the true moving direction and speed of the key dynamic target in the scene:

[0048] , where is the true pixel motion vector of the key dynamic target, is the pixel motion vector calculated by the optical flow method (including the influence of the drone's own motion), is the motion vector set of key pixel points; is the average value of the key pixel motion vector (used to offset global drift), is the mode of the key pixel motion vector (used to suppress outlier interference), is the weight coefficient of UAV motion compensation, is the weighting coefficient of the mean and mode.

[0049] It should be further explained that, in the specific implementation process, the key dynamic target detection unit is used to extract features from the input video frames using the optimized YOLO-X network structure. Combined with the temporal feature fusion and dynamic threshold adjustment strategy, it performs fine-grained detection on the identified key dynamic targets. The process of identifying the key dynamic target type includes:

[0050] After the YOLO backbone network performs preliminary feature extraction, a dynamic feature selection mechanism is constructed based on target scale difference analysis. This mechanism is used to adaptively assign feature weights of different levels according to the scale of key dynamic targets and generate a preliminary target feature map. Specifically, target scale difference analysis refers to the analysis of feature differences between targets of different scales (such as small-scale pedestrians in the distance and large-scale vehicles nearby) so that feature weights of different levels can be adaptively assigned according to the target scale during feature extraction. That is, for small-scale targets (such as pedestrians in the distance), more weights are assigned to deep semantic features to obtain richer semantic information; for large-scale targets (such as nearby vehicles), the weights of shallow spatial features are appropriately increased to retain more detailed information, thereby obtaining a more target-specific feature representation; based on the generation of a preliminary target feature map, it is combined with the temporal characteristics of the video sequence, specifically: the target feature map of each frame is regarded as a temporal feature map. A temporal node in a temporal feature graph. These temporal nodes are arranged in sequence in the time dimension, representing the state of target features at different moments. At the same time, the target feature relationship between adjacent frames is defined as a temporal edge in the temporal feature graph. This relationship is quantified by calculating the difference index between the target features of adjacent frames. The difference index can be quantified by calculating the Euclidean distance or cosine similarity between the target feature vectors of adjacent frames to measure the amplitude of feature changes. In a video sequence, there are continuous motion and state changes between different frames. By constructing a temporal feature graph, the association relationship of key dynamic targets in the time dimension, that is, the temporal association information between different frames, is represented. The feature vectors of key dynamic targets are used as the attributes of the temporal nodes in the temporal feature graph, and the relative position relationship between key dynamic targets is used as the weight of the temporal edge in the temporal feature graph to capture the feature information and spatial relationship of key feature targets in the temporal sequence.

[0051] After the construction of the temporal feature graph is completed, it is input into the graph neural network for processing, where the graph neural network operates on the temporal nodes and temporal edges in the temporal feature graph: during the information transmission process, the temporal nodes will receive information from adjacent temporal nodes and transmit their own information to adjacent temporal nodes; in the information aggregation stage, the temporal nodes will integrate the received information to update their own feature representations; the graph neural network learns the temporal feature associations between key dynamic targets in different frames, and encodes the dynamic change information of key dynamic targets in the temporal sequence into the feature representation; through multi-layer graph neural network processing, the feature representation of the fused temporal information is obtained, where the feature representation contains the dynamic change information of the key dynamic targets in the entire temporal sequence; the splicing operation is used to fuse the feature representation of the fused temporal information output by the graph neural network with the target feature graph of the current frame to generate a target feature graph after the fusion of the temporal information; based on the target feature graph after the fusion of the temporal information, the types of key dynamic targets are identified, including vehicles and pedestrians.

[0052] It should be further explained that, in the specific implementation process, the key dynamic target tracking unit is used to receive the output of the key dynamic target detection unit, build an incremental dynamic feature memory, dynamically adjust the threshold based on the scene density, implement trajectory association and initialization through the memory library, and manage short-term and long-term motion trajectories. The process of generating the unique ID and motion trajectory of the key dynamic target includes the following:

[0053] For each identified key dynamic target, calculate the acceleration change rate and detection confidence of consecutive A frames, and combine the acceleration change rate and detection confidence into a dynamic attribute vector;

[0054] Among them, the acceleration change rate of consecutive A frames for:

[0055] , where is the instantaneous acceleration at frame t; is the target velocity in the tth frame (calculated by the position difference between adjacent frames); is the frame time interval (default is 1 frame); A frame is the dynamic attribute calculation window length, which is the number of consecutive frames used to calculate dynamic attributes. This window length determines the sensitivity of dynamic attributes to short-term motion changes of key dynamic targets (usually short, such as 3-5 frames);

[0056] Dynamic property vector for: , where To test confidence;

[0057] Extract the appearance feature descriptors of key dynamic targets (i.e., ReID feature vectors) and store them in association with the dynamic attribute vectors. Build a memory pool that only stores the dynamic attributes and appearance feature descriptors of the most recent G motion trajectory segments. When a new key dynamic target enters the memory pool, if the memory pool is full, remove the oldest dynamic attributes according to the FIFO rule (first in, first out). G is the maximum number of trajectory segments that can be stored in the memory pool.

[0058] Count the number of key dynamic targets in the current highway scene and estimate the density of their distribution: Calculate the highway scene density factor based on the number of key dynamic targets in the current frame and the area of ​​the highway scene.

[0059] Among them, the highway scene density factor The calculation formula is:

[0060] , where is the number of key dynamic targets in the current frame, is the area of ​​the highway scene;

[0061] Based on the road scene density factor, adjust the basic association matching threshold and the basic detection confidence threshold:

[0062] , where is the adjusted basic association matching threshold, is the basic association matching threshold, which is used to judge the degree of association between the key dynamic target and the motion trajectory fragment in the memory pool. When the association cost is less than this threshold, the association is considered successful; is the adjusted basic detection confidence threshold, The basic detection confidence threshold is used to filter low-confidence detection results. Only key dynamic targets with a detection confidence greater than this threshold will be further processed. They are respectively the positive adjustment coefficient of the highway scene density factor on the basic association matching threshold and the negative adjustment coefficient on the basic detection confidence threshold;

[0063] For each key dynamic target in the current frame, compare it with the motion trajectory fragments in the memory pool and calculate the association cost: the Mahalanobis distance and appearance distance are weightedly fused into the association cost;

[0064] Among them, the associated cost The calculation formula is:

[0065] , where is the fusion weight of Mahalanobis distance and appearance distance; is the Mahalanobis distance, , The trajectory predicted by the Kalman filter Location, Key dynamic targets The detection position, is the matrix transpose symbol, is the Kalman filter covariance matrix; is the appearance distance, , For the motion trajectory With key dynamic targets The Mahalanobis distance is based on the difference between the motion trajectory position predicted by the Kalman filter and the key dynamic target detection position, and the appearance distance is based on the Euclidean distance between the key dynamic target and the appearance feature descriptor.

[0066] If the key dynamic target satisfies the requirement that the association cost is less than the basic association matching threshold and the detection confidence is greater than the basic detection confidence threshold for N consecutive frames, then the key dynamic target is considered to belong to this motion trajectory, a unique ID is assigned to it, and the motion trajectory is formally initialized to generate an initial motion trajectory with a unique ID:

[0067] , where The starting frame number (i.e., the first frame from which the current key dynamic target first meets the association condition) is used to mark the starting point of successful matching to ensure the stability of N consecutive frames. For example, assuming N=5 (5 consecutive frames must be matched successfully), if the key dynamic target is at frame number When the condition is met for the first time, it is necessary to check whether frames 10 to 14 all meet the condition; if frames 10 to 14 all pass, then Initialize the trajectory at the same time; if a frame in the middle fails (such as frame 12 is not matched), the count is reset; N frames is the motion trajectory initialization window length, which is the number of consecutive matching frames used for motion trajectory initialization. The window length determines the reliability requirement of motion trajectory initialization (usually long, such as 5~10 frames));

[0068] The initial motion trajectory is managed through its lifecycle, including short-term and long-term trajectory management. Detection results that are not associated with any trajectory fragment in the memory pool (i.e., marked as new targets) are managed as short-term motion trajectories. After a short-term motion trajectory is maintained for Z frames, if it is still not associated with any trajectory fragment in the memory pool, it is determined to contain a false detection or a briefly appearing key dynamic target and is destroyed. Z frames are the length of the short-term trajectory maintenance window, which controls the maximum number of frames that a trajectory not associated with the memory pool can survive. This window length determines the tolerance of lifecycle management for briefly appearing key dynamic targets (usually set to 10-20 frames, but the specific value needs to be adjusted based on the dynamics of the scene). Key dynamic targets that are successfully associated with any trajectory fragment in the memory pool are managed as long-term motion trajectories, and their dynamic attribute vectors (acceleration rate, detection confidence) and appearance feature identifiers are continuously updated to reflect the latest state of the key dynamic target during motion. The formula for updating dynamic attributes and appearance features using sliding average is:

[0069] , where is the forgetting factor, that is, the proportion of historical information retained during long-term trajectory updates, is the updated dynamic attribute vector, is the appearance feature identifier, is the updated appearance feature identifier;

[0070] After completing the motion trajectory lifecycle management, the unique ID and motion trajectory route of the key dynamic target are output (wherein the motion trajectory route consists of a series of motion trajectory points, which are used to reflect the motion trajectory of the key dynamic target in the video sequence).

[0071] It should be further explained that, in the specific implementation process, the motion trajectory analysis unit is used to perform spatial and temporal enhancement processing on the motion trajectory data output by multi-target tracking, fuse appearance features and dynamic attributes to generate multimodal motion trajectory features, and extract initial behavior pattern labels (such as going straight, changing lanes) based on the motion trajectory point sequence; initialize the behavior graph through traffic rules, combine historical abnormal motion trajectory clustering to mine new abnormal behavior nodes, and dynamically update the graph structure (such as transition probability) to form a dynamic behavior graph containing normal and abnormal patterns; use the spatiotemporal attention Transformer to encode motion trajectory features, combine the dynamic behavior graph to calculate the anomaly score and path deviation, comprehensively judge the trajectory abnormality and map it into a natural language description, and finally classify it according to the degree of abnormality (minor, severe, dangerous); the specific process is as follows:

[0072] Obtain motion trajectory data output by the multi-target tracking unit , including unique ID, motion trajectory point sequence, appearance feature identifier and dynamic attribute vector:

[0073] ,in, It is the key dynamic target index output by the multi-target tracking unit; The number of key dynamic targets output by the multi-target tracking unit; A unique identifier for the key dynamic target; is the motion trajectory point sequence, is the motion trajectory point, is the length of the motion trajectory, m is the motion trajectory index; is a set of appearance feature identifiers; is a dynamic attribute vector sequence, is the dynamic attribute vector value, including instantaneous velocity, acceleration, direction angle and acceleration change rate, that is, ;

[0074] The motion trajectory points, appearance feature identifiers and dynamic attribute vectors of key dynamic targets are enhanced and fused into multimodal motion trajectory features. :

[0075] , where is the enhanced dynamic attribute vector, It is a vector concatenation operation;

[0076] Among them, the motion trajectory points are spatially interpolated (such as cubic spline interpolation) to generate a smooth motion trajectory curve and obtain a spatially enhanced motion trajectory point sequence and appearance feature identifier sets ;

[0077] Introduce a timestamp sequence to calculate the instantaneous velocity, acceleration, direction angle, and acceleration change rate of the motion trajectory points for time enhancement:

[0078] ;

[0079] Perform time-series weighted averaging on the appearance feature vectors of the same target in the motion trajectory to generate enhanced appearance feature representation :

[0080] , where To test confidence, is an appearance feature identifier;

[0081] Based on the motion trajectory point sequence, behavior pattern labels (such as going straight, changing lanes, and staying) are extracted as initial semantic annotations.

[0082] Construct an initial map based on traffic rules. In the graph, the behavior state node is the initial graph The core component of represents various traffic behavior states (such as normal driving, speeding, and wrong-way driving). The state transition edge is used to represent the transition relationship between traffic behavior states. Initially, it is assumed that the probabilities of all state transitions are uniformly distributed:

[0083] , where are the initial behavior state node set, the initial state transfer edge, is the behavior state node, Behavioral state The initial transition probability, All are behavior state node indexes, is the number of behavior state nodes;

[0084] Obtain a historical motion trajectory database (including normal behavior patterns and known abnormal patterns), cluster abnormal motion trajectories in the historical motion trajectory database (such as DBSCAN), and extract abnormal behavior patterns (such as frequent lane changes and abnormal dwelling). Add the mined abnormal patterns as new behavior state nodes to the initial graph and initialize the transition probabilities. Receive new motion trajectory data in real time and dynamically update the graph structure (including adding abnormal behavior state nodes and adjusting transition probabilities), thereby constructing a dynamic behavior graph (including normal behavior patterns and known abnormal patterns):

[0085] , where Behavior state after initial graph update The transition probability, is the smoothing factor, Behavioral state The actual number of transfers, It is a temporary index variable used for summing and traversing all behavior state nodes; is the actual number of transitions to all behavioral states;

[0086] The spatiotemporal attention Transformer is introduced to encode multimodal motion trajectory features. Spatiotemporal attention includes temporal attention, spatial attention, and graph attention. Temporal attention is used to capture the temporal dependencies between motion trajectory points. Spatial attention is used to enhance the spatial correlation of motion trajectory points (such as the mutual influence of adjacent motion trajectory points). Graph attention integrates the behavioral state node features of the dynamic behavior graph with the motion trajectory features to enhance behavioral semantic understanding.

[0087] Compare the encoded multimodal motion trajectory features with the characteristic distribution of normal behavior patterns in the dynamic behavior map and calculate the anomaly score :

[0088] , where is the multimodal motion trajectory feature, are the mean and covariance of normal motion trajectory features, is the matrix transpose symbol;

[0089] Calculate the path deviation of multimodal motion trajectories on dynamic behavior maps (i.e. the difference between the current trajectory path and the reference shortest path):

[0090] , where is the current trajectory path, is the reference shortest path;

[0091] The anomaly score and graph deviation are combined to determine whether the movement trajectory of the key dynamic target is abnormal. The path of the abnormal movement trajectory on the dynamic behavior graph is mapped into a natural language description (for example, the key dynamic target reverses from lane A to lane B for 5 seconds). For example, a pre-trained language model (such as BERT) is used to perform grammatical and logical verification on the generated semantics to ensure consistency with traffic regulations.

[0092] A weighted calculation is performed based on the anomaly score and the map deviation to obtain the anomaly level, including minor anomalies, major anomalies, and dangerous anomalies. For example, a minor anomaly is a short-term speeding, lasting <3 seconds, and the speed is >10% of the speed limit; a major anomaly is a wrong direction, lasting >3 seconds, and the direction is opposite to the mainstream direction; a dangerous anomaly is a collision risk, and the distance to the adjacent motion trajectory is < the preset safety threshold.

[0093] It should be further explained that, in the specific implementation process, the visualization unit is used to overlay the motion trajectory analysis results and abnormality level classification of key dynamic targets on the electronic map in real time, so that users can intuitively understand the road traffic conditions.

[0094] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0095] It should be understood that determining B based on A does not mean determining B based solely on A. B can also be determined based on A and / or other information.

[0096] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0097] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-target tracking system for highway scenes based on UAV visual perception, characterized by: It includes UAV platform, ground control station, target detection and tracking module, and motion trajectory analysis and visualization module; UAV platform: collects highway scene videos in real time and transmits the collected video data back. The UAV platform includes an image acquisition unit and a data transmission unit. Ground control station: processes the video data transmitted by the drone and generates flight control instructions to dynamically adjust the path. The ground control station includes a data pre-processing unit and a flight control unit; Target Detection and Tracking Module: This module uses a one-shot detector model to extract key dynamic target features in highway scenes and perform cross-frame matching statistics. It also uses optical flow to analyze the motion vectors of static objects to offset drone interference and obtain the true motion information of key dynamic targets. The module includes a key dynamic target acquisition unit, a key dynamic target association unit, a motion compensation unit, a key dynamic target detection unit, and a key dynamic target tracking unit. The key dynamic target tracking unit receives the output of the key dynamic target detection unit, constructs an incremental dynamic feature memory, dynamically adjusts the threshold based on the scene density, and implements trajectory association and initialization through the memory library. The process includes: For each identified key dynamic target, the acceleration change rate and detection confidence of consecutive A frames are calculated and combined into a dynamic attribute vector. A frame is the length of the dynamic attribute calculation window, which is used to calculate the number of consecutive frames of dynamic attributes. Extract the appearance feature descriptors of key dynamic targets and store them in association with the dynamic attribute vector; establish a memory pool that only stores the dynamic attributes and appearance feature descriptors of the most recent G motion trajectory segments, where G is the maximum number of trajectory segments that can be stored in the memory pool; Count the number of key dynamic targets in the current highway scene and estimate the density of their distribution. That is, calculate the highway scene density factor based on the number of key dynamic targets in the current frame and the area of ​​the highway scene. Based on the road scene density factor, adjust the basic association matching threshold and the basic detection confidence threshold; Motion trajectory association and initialization based on the memory pool: Each key dynamic target in the current frame is compared with the motion trajectory segments in the memory pool, and the association cost is calculated. That is, the Mahalanobis distance and appearance distance are weightedly fused into the association cost; If the key dynamic target satisfies the association cost less than the basic association matching threshold and the detection confidence is greater than the basic detection confidence threshold for N consecutive frames, then the key dynamic target is considered to belong to this motion trajectory, a unique ID is assigned to it, and the motion trajectory is formally initialized to generate an initial motion trajectory with a unique ID; where N frames is the length of the motion trajectory initialization window, which is the number of consecutive matching frames used for motion trajectory initialization; Motion trajectory analysis and visualization module: Analyzes the motion trajectory of key dynamic targets and visualizes the motion trajectory analysis results of key dynamic targets; the motion trajectory analysis and visualization module includes a motion trajectory analysis unit and a visualization unit.

2. The highway scene multi-target tracking system based on UAV visual perception according to claim 1 is characterized by: The image acquisition unit is equipped with a 4K high-definition camera and a gimbal stabilization system to collect real-time video data of road scenes; the data return unit uses 5G communication technology for low-latency transmission of video data.

3. The highway scene multi-target tracking system based on UAV visual perception according to claim 1 is characterized by: The data preprocessing unit is used to perform denoising, lighting balance and geometric correction on the received video data; the flight control unit is used to control the flight of the UAV and dynamically adjust the flight path based on the motion trajectory analysis results of key dynamic targets.

4. The highway scene multi-target tracking system based on UAV visual perception according to claim 1 is characterized by: The key dynamic target acquisition unit is used to obtain preprocessed video data, extract key features using one-shot detector model training, and identify and count key dynamic targets in highway scenes.

5. The highway scene multi-target tracking system based on UAV visual perception according to claim 1 is characterized by: The key dynamic target association unit is used to use the structural similarity index to determine whether the key dynamic targets of consecutive frames are the same target. The similarity of the key dynamic targets of the current frame and the key dynamic targets of the next frame is calculated; if the value of the structural similarity index of the key dynamic targets in consecutive frames exceeds the preset cross-frame structural similarity matching threshold, they are determined to be the same target.

6. The highway scene multi-target tracking system based on UAV visual perception according to claim 1 is characterized by: The motion compensation unit is used to calculate the motion vector of pixels in the image between consecutive frames for static objects in the highway scene using the optical flow method, and fuse the average and mode weighted average of the moving direction and speed of key pixels to obtain the true pixel motion vector of the key dynamic target.

7. A highway scene multi-target tracking system based on UAV visual perception according to any one of claims 1 to 6, characterized in that: The key dynamic target detection unit is used to extract features from input video frames using the optimized YOLO-X network structure. It then combines temporal feature fusion with a dynamic threshold adjustment strategy to perform fine-grained detection on identified key dynamic targets. The process of identifying key dynamic target types includes the following: After the YOLO backbone network performs preliminary feature extraction, a dynamic feature selection mechanism is constructed based on target scale difference analysis. This mechanism is used to adaptively assign feature weights of different levels according to the scale of key dynamic targets to generate a preliminary target feature map. Based on the generated preliminary target feature map, it is combined with the temporal characteristics of the video sequence. Specifically, the target feature map of each frame is regarded as a temporal node in the temporal feature map. At the same time, the target feature relationship between adjacent frames is defined as a temporal edge in the temporal feature map. The feature vectors of key dynamic targets are used as the attributes of the temporal nodes in the temporal feature map, and the relative position relationship between key dynamic targets is used as the weight of the temporal edge in the temporal feature map to capture the feature information and spatial relationship of key feature targets in the temporal sequence. Among them, the relative position relationship is obtained by quantitatively analyzing the difference index between the feature vectors of adjacent frames, and the quantization result is used as the weight of the temporal edge in the temporal feature map. After the construction of the temporal feature graph is completed, it is input into the graph neural network for processing, where the graph neural network operates on the temporal nodes and temporal edges in the temporal feature graph; the graph neural network learns the temporal feature associations between key dynamic targets in different frames, and encodes the dynamic change information of the key dynamic targets in the temporal sequence into the feature representation; through multi-layer graph neural network processing, the feature representation of the fused temporal information is obtained, where the feature representation contains the dynamic change information of the key dynamic targets in the entire temporal sequence; the splicing operation is used to fuse the feature representation of the fused temporal information output by the graph neural network with the target feature graph of the current frame to generate a target feature graph after the fusion of the temporal information; based on the target feature graph after the fusion of the temporal information, the type of key dynamic target is identified.

8. The highway scene multi-target tracking system based on UAV visual perception according to claim 1 is characterized by: The key dynamic target tracking unit is used to manage short-term and long-term motion trajectories. The process of generating the unique ID and motion trajectory of the key dynamic target includes: When a new key dynamic target enters the memory pool, if the memory pool is full, the oldest dynamic attribute is removed according to the FIFO rule; Perform motion trajectory lifecycle management for the initial motion trajectory, including short-term and long-term trajectory management. Detection results that are not associated with any motion trajectory fragment in the memory pool are managed as short-term motion trajectories; key dynamic targets that are successfully associated with any motion trajectory fragment in the memory pool are managed as long-term motion trajectories. After completing the motion trajectory lifecycle management, the unique ID and motion trajectory route of the key dynamic target are output.

9. The highway scene multi-target tracking system based on UAV visual perception according to claim 8, characterized in that: The motion trajectory analysis unit is used to perform spatial and temporal enhancement processing on the motion trajectory data output by multi-target tracking, fuse appearance features and dynamic attributes to generate multimodal motion trajectory features, and extract initial behavior pattern labels based on the motion trajectory point sequence; The behavior graph is initialized by traffic rules, and new abnormal behavior nodes are mined by clustering historical abnormal motion trajectories. The graph structure is dynamically updated to form a dynamic behavior graph that contains normal and abnormal patterns. The spatiotemporal attention transformer is used to encode motion trajectory features. The anomaly score and path deviation are calculated based on the dynamic behavior graph. The trajectory abnormality is comprehensively judged and mapped into a natural language description, and finally classified according to the degree of abnormality. The specific process is as follows: Obtain the motion trajectory data output by the multi-target tracking unit, including the unique ID, motion trajectory point sequence, appearance feature identifier, and dynamic attribute vector; enhance the motion trajectory points, appearance feature identifier, and dynamic attribute vector of key dynamic targets and fuse them into multimodal motion trajectory features; The method involves spatially interpolating the motion trajectory points to generate a smooth motion trajectory curve and obtain a spatially enhanced motion trajectory point sequence. A timestamp sequence is introduced to calculate the instantaneous velocity, acceleration, azimuth angle, and acceleration change rate of the motion trajectory points for temporal enhancement. A time-series weighted average is performed on the appearance feature vectors of the same target in the motion trajectory to generate an enhanced appearance feature representation. Based on the motion trajectory point sequence, behavioral pattern labels are extracted as initial semantic annotations. An initial graph is constructed based on traffic rules. In this initial graph, behavior state nodes are the core components, representing various traffic behavior states, while state transition edges are used to represent the transition relationship between traffic behavior states. Obtain a historical motion trajectory database, cluster abnormal motion trajectories in the historical motion trajectory database, and extract abnormal behavior patterns; add the mined abnormal patterns as new behavior state nodes to the initial graph and initialize the transition probability; receive new motion trajectory data in real time, dynamically update the graph structure, and thus construct a dynamic behavior graph; A spatiotemporal attention transformer is introduced to encode multimodal motion trajectory features. The encoded multimodal motion trajectory features are compared with the characteristic distribution of normal behavior patterns in the dynamic behavior map to calculate the anomaly score. The path deviation of the multimodal motion trajectory on the dynamic behavior map is calculated. The anomaly score and the graph deviation are combined to determine whether the motion trajectory of the key dynamic target is abnormal; the path of the abnormal motion trajectory on the dynamic behavior graph is mapped into a natural language description; and a weighted calculation is performed based on the anomaly score and the graph deviation to obtain the anomaly level, including minor anomaly, severe anomaly, and dangerous anomaly.

10. The highway scene multi-target tracking system based on UAV visual perception according to claim 9, characterized in that: The visualization unit is used to overlay and display the motion trajectory analysis results and abnormality level classification of key dynamic targets on the electronic map in real time.

Images