A method for detecting, tracking and measuring speed of unmanned aerial vehicle based on deep learning and multi-sensor fusion

Abstract

The application discloses a kind of unmanned vehicle detection, tracking and speed measurement method based on deep learning and multi-sensor fusion, video image stream of road vehicle is collected using gimbal camera carried by unmanned aerial vehicle, and simultaneously using laser ranging module carried by unmanned aerial vehicle to collect height data with preset frequency;Synchronous recording each frame image corresponding unmanned aerial vehicle flight state parameter;YOLOv8-based instance segmentation model is constructed, and the class, confidence, bounding box coordinates and instance segmentation mask of vehicle target are obtained using instance segmentation model;The detected vehicle target is tracked using HMA-SORT algorithm, and the continuous trajectory point of vehicle in image pixel coordinate system is obtained;According to laser ranging data and camera parameters, calculate ground sampling distance;Combined with vehicle continuous trajectory point and unmanned aerial vehicle flight state, judge vehicle driving road condition as straight line, curve or slope, calculate vehicle relative displacement using corresponding geometric model for different road conditions, and solve vehicle absolute ground speed combined with unmanned aerial vehicle speed vector.The application effectively solves the problem of speed measurement error caused by complex terrain and unmanned aerial vehicle movement, and has the advantages of high detection accuracy, strong environmental adaptability and the like.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, intelligent transportation systems and unmanned aerial vehicle (UAV) application technology, specifically a method for UAV vehicle detection, tracking and speed measurement based on deep learning and multi-sensor fusion. Background Technology

[0002] With the rapid development of unmanned aerial vehicle (UAV) technology, using UAVs for traffic inspection has become an important part of intelligent transportation systems (ITS). Compared with fixed roadside surveillance cameras, UAVs have advantages such as mobility, wide field of view, and no blind spots, enabling rapid response and comprehensive monitoring of congested road sections, traffic accident scenes, and violations.

[0003] However, vehicle detection, tracking, and speed measurement from the perspective of drone aerial photography faces numerous technical challenges. Firstly, in terms of vehicle detection, drones fly at relatively high altitudes, typically 50 to 100 meters or more, resulting in ground vehicles appearing as very small targets in the image (the "small target" problem), with blurred texture features. Simultaneously, the aerial background is extremely complex, including tree occlusion, building shadows, and road markings, making it highly susceptible to missed detections. Existing general-purpose target detection algorithms (such as the YOLO series) often suffer from insufficient feature extraction and high false positive / false negative rates when dealing with such densely packed small targets.

[0004] Secondly, in multi-target tracking, the six degrees of freedom motion (translation, rotation, and jitter) of the drone itself causes drastic displacement of the video background. Traditional Kalman filter-based tracking algorithms (such as DeepSORT) typically assume that the camera is stationary or that the motion is smooth. In high-dynamic drone scenarios, the overall motion of the background may be misinterpreted by the algorithm as the motion of the target, resulting in a large deviation between the predicted position of the Kalman filter and the actual detection position. This leads to frequent jumps in the target's identity ID, making it impossible to form a continuous and stable trajectory, which seriously affects subsequent traffic flow analysis.

[0005] Furthermore, in vehicle speed measurement, traditional visual speed measurement methods typically assume the road is an ideal horizontal straight line, using a simple perspective transformation scale to map pixel coordinates to physical coordinates. However, in practical applications, drones often operate in complex terrains such as mountainous areas, overpasses, and ramps, where roads often have significant curvature and slope. Directly using a planar projection model ignores terrain undulations and path curvature, leading to significant speed measurement errors. For example, on a slope, the horizontal projection distance is smaller than the actual slope distance, resulting in an underestimation of the speed.

[0006] Finally, existing high-precision UAV speed measurement solutions mostly rely on expensive millimeter-wave radar or RTK differential positioning equipment, which are costly and have high system integration complexity, making them difficult to promote in large-scale inspections. Although low-cost solutions based on pure vision and single-point laser ranging have simple hardware, they currently lack a unified geometric solution model that can adapt to various road conditions such as straight lines, curves, and slopes. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method for unmanned aerial vehicle (UAV) vehicle detection, tracking, and speed measurement based on deep learning and multi-sensor fusion. This method improves the detection accuracy of small targets by refining the YOLOv8 model structure, proposes the HMA-SORT algorithm to address tracking instability under camera motion, and combines laser ranging to construct a multi-road-condition intelligent geometric speed measurement model, achieving high-precision and robust traffic parameter extraction.

[0008] To achieve the above objectives, this invention provides a method for detecting, tracking, and measuring the speed of unmanned aerial vehicles (UAVs) based on deep learning and multi-sensor fusion, comprising the following steps:

[0009] Step S1: Data Acquisition and Spatiotemporal Synchronization: The video image stream of road vehicles is acquired using the gimbal camera on the UAV, and the ground altitude data is acquired at a preset frequency using the laser ranging module on the UAV. The flight status parameters of the UAV corresponding to each frame of the image are recorded synchronously. The flight status parameters of the UAV include the UAV's three-dimensional coordinates, flight speed vector, and gimbal pitch angle.

[0010] Step S2: Construct an instance segmentation model based on YOLOv8. Input the acquired video images into the instance segmentation model, extract features through the backbone network, perform multi-scale feature fusion using the neck network, and finally output the vehicle target category, confidence score, bounding box coordinates, and instance segmentation mask through the decoupled detection head.

[0011] Step S3: Use the HMA-SORT algorithm to perform correlation tracking on the detected vehicle targets to obtain the continuous trajectory points of the vehicle in the image pixel coordinate system;

[0012] Step S4: Calculate the ground sampling distance based on the laser ranging data and camera parameters; combine the vehicle's continuous trajectory points and the UAV's flight status to determine whether the vehicle is traveling on a straight road, a curve, or a slope; use the corresponding geometric model to calculate the vehicle's relative displacement for different road conditions; and combine the UAV's velocity vector to calculate the vehicle's absolute ground speed.

[0013] Compared with the prior art, the significant advantages of this invention are:

[0014] (1) High detection accuracy and strong computational efficiency. By introducing the PConv partial convolution operator into the backbone network, the computational redundancy is greatly reduced while retaining the feature extraction capability, and the inference speed on the airborne embedded device is significantly improved; the SimAM parameterless attention mechanism is embedded in the neck network, which can adaptively enhance the features of small targets without additional training parameters, effectively solving the problem of missed detection caused by small vehicle targets and complex backgrounds under aerial photography perspective. (2) Strong tracking stability and excellent anti-interference capability. The HMA-SORT algorithm eliminates the background displacement interference caused by the UAV's own motion through GMC global motion compensation, and NSA-Kalman adaptively adjusts the filtering parameters according to the detection confidence, effectively suppressing detection jitter; the hierarchical hybrid association strategy improves the recall rate of occluded and blurred targets; the OCM momentum correction mechanism solves the problem of trajectory drift and ID jump after long-term occlusion, and realizes continuous and stable tracking under complex dynamic backgrounds. (3) High speed measurement accuracy and strong adaptability to multiple road conditions. An adaptive geometric solution model for three road conditions—straight road, curve, and slope—was constructed. The curvature error of the curve was corrected by least squares circular arc fitting, and the projection error caused by the slope was compensated by local plane fitting. The vehicle's absolute ground speed was calculated by integrating the UAV flight speed vector, which effectively overcame the speed measurement distortion problem of the traditional planar projection model in complex terrain. (4) The system has low cost and high integration. It only relies on ordinary gimbal cameras and single-point laser rangefinders, without the need for expensive millimeter-wave radar or RTK differential positioning equipment. The hardware cost is low, the system integration is simple, and it is suitable for the promotion and application of large-scale UAV traffic inspection scenarios. Attached Figure Description

[0015] Figure 1 This is a general flowchart of the method according to an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the improved YOLOv8-Faster-SimAM model network structure in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the structure of the C2f_Faster module and the PConv operator in an embodiment of the present invention.

[0018] Figure 4 This is a schematic diagram of the SimAM attention mechanism structure in an embodiment of the present invention.

[0019] Figure 5 This is a logical architecture diagram of the HMA-SORT tracking algorithm in an embodiment of the present invention.

[0020] Figure 6 This is a schematic diagram of the straight track geometry in an embodiment of the present invention.

[0021] Figure 7 This is a modified diagram of the geometric principle of a curved road in an embodiment of the present invention.

[0022] Figure 8 This is a modified diagram of the slope geometry principle in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0024] A method for detecting, tracking, and measuring the speed of unmanned aerial vehicles (UAVs) based on deep learning and multi-sensor fusion is proposed. This method utilizes deep learning algorithms to process visual information and geometric algorithms to fuse ranging information, enabling high-precision perception of vehicles in complex dynamic environments.

[0025] Step S1: Data Acquisition and Spatiotemporal Synchronization: Use the gimbal camera mounted on the UAV to acquire video image streams of road vehicles, and simultaneously use the laser ranging module to acquire ground altitude data at a preset frequency. ,in The vertical height is measured by a laser rangefinder; the UAV flight status parameters corresponding to each frame of the image are recorded synchronously, including the UAV's three-dimensional coordinates. Flight speed vector and gimbal tilt angle ,in This represents the three-axis velocity components of the UAV in the world coordinate system;

[0026] In this embodiment, the physical carrier is the DJI M400RTK quadcopter drone, which has high-precision RTK positioning capabilities and stable flight control performance.

[0027] For payload, it is equipped with a Zenmuse H30 multi-lens pod. This pod integrates a high-resolution wide-angle camera, a high-zoom camera, and a single-point laser rangefinder. The image acquisition resolution is set to [value missing]. The frame rate is 10 FPS.

[0028] In terms of computing platform, it is equipped with the Manifold3 airborne computer. This computer communicates with the drone's flight control system and payload in full-duplex mode via the PSDK (PayloadSDK) interface, and is responsible for acquiring sensor data in real time and running AI algorithms.

[0029] To address the multi-sensor time synchronization issue, this embodiment employs a strategy combining hard triggering and soft synchronization. The UAV's PPS (pulses per second) signal is used as the master clock source to uniformly mark the camera exposure time and the laser ranging sampling time.

[0030] During data acquisition, the drone patrols above the road segment under test. Cameras capture real-time video streams of the road. A laser ranging module emits a laser beam vertically downwards at a frequency of 50Hz to obtain the drone's vertical altitude data relative to the ground. .

[0031] Simultaneously, the flight status parameters corresponding to each frame of the image are recorded with microsecond-level precision via the OSDK interface. These parameters include the UAV's three-dimensional GPS coordinates in the world coordinate system. .

[0032] The recorded parameters also include the drone's three-axis flight velocity vector. This is used to subsequently calculate the vehicle's absolute ground speed.

[0033] In addition, the attitude angles of the gimbal camera, especially the pitch angle, also need to be recorded. This angle directly affects the geometric projection relationship of the imaging plane.

[0034] Step S2: Instance Segmentation and Object Detection. A YOLOv8-Faster-SimAM instance segmentation model is constructed. Acquired video images are input into the model. Features are extracted through the backbone network, and multi-scale feature fusion is performed using the neck network. Finally, the decoupled detection head outputs the vehicle target category and confidence level. , bounding box coordinates and instance segmentation mask, where This represents the probability that the target belongs to a specific category.

[0035] To address the two major pain points of "small targets" and "complex backgrounds" in drone aerial photography, this embodiment deeply reconstructs the YOLOv8n-seg instance segmentation model and designs the YOLOv8-Faster-SimAM model.

[0036] Improvement 1: Lightweight reconstruction of the backbone network. The original YOLOv8 backbone network heavily used C2f modules, resulting in extremely high computational redundancy when processing shallow features (P2, P3 layers) of high-resolution images.

[0037] In this embodiment, the C2f modules used in Backbone to process features at scales P2, P3, P4, and P5 are all replaced with the C2f_Faster module. The design of the C2f_Faster module is inspired by FasterNet, and its core component is partial convolution (PConv).

[0038] The PConv operator is designed to utilize the redundancy between feature map channels. For the input feature map... (in For the number of channels, For height, (where the width is 1), PConv does not perform convolution on all channels.

[0039] Specifically, PConv divides the input channel into two parts: the first part is the front One channel (in this embodiment, we take...) The first part is called the representative channel; the second part is the remaining part. One channel.

[0040] PConv only applies to this previous point. Each channel Spatial convolution operations are performed to extract spatial features. For the remaining channels, the identity mapping is maintained, meaning no computation is performed, and they are directly passed.

[0041] The computational advantage (FLOPs) of this design is extremely significant. Traditional convolution has a computational cost of... PConv's computational cost is only ,in This represents the kernel size.

[0042] When taking At that time, the computational cost of PConv is only one-third that of traditional convolution. This not only reduces the amount of floating-point operations, but more importantly, it significantly reduces memory access latency and substantially improves inference speed on airborne embedded devices.

[0043] In the C2f_Faster module, the PConv layer is immediately followed by a... The point convolution is used. The function of this layer is to linearly combine the spatial features extracted by PConv with the information of the preserved channels to restore the interaction between all channels.

[0044] Improvement point 2: Feature enhancement of the neck network. In the PANet feature fusion stage of the neck part, in order to solve the problem that small target features are easily lost during downsampling, this embodiment introduces the SimAM parameterless attention mechanism.

[0045] The specific approach is to replace the C2f module in Neck with the C2f_SimAM module. This module reweights the feature map after the upsampling and concatenation operations of feature fusion.

[0046] SimAM is based on the energy function theory of neuroscience. Its core idea is that neurons with distinctive characteristics (targets) have firing patterns that differ significantly from those of surrounding neurons (background).

[0047] This embodiment describes each neuron in the feature map. An energy function is defined. This is used to measure its linear separability from surrounding neurons:

[0048]

[0049] in, For neurons Energy value, The value of the target neuron. The predicted value after linear transformation. It is the total number of neurons within the channel. Except Other neurons besides and These are the weights and biases of the linear transformation, used to capture the correlation between neurons.

[0050] To find the minimum energy for each neuron, the above equation needs to be minimized. This embodiment derives an analytical solution (closed-form solution) for this energy function, eliminating the need for parameter learning through SGD training.

[0051] Minimum Energy The calculation formula is as follows:

[0052]

[0053] in, and These are the global mean and variance of the input feature map in the current channel, respectively. The value of the current target neuron. This is the regularization coefficient, used to prevent overfitting. The formula shows that neurons... vs. background mean The farther the Euclidean distance, the greater its energy. The lower the value, the higher the importance.

[0054] Based on minimum energy, this embodiment calculates attention weights. The weights are the reciprocals of the energy, and are normalized using the Sigmoid function.

[0055]

[0056] Finally, the generated weights Compared with the original feature map Perform element-wise multiplication This process can adaptively enhance the texture features of vehicle targets while suppressing background noise such as trees and road surfaces.

[0057] Step S3: Use the HMA-SORT algorithm for multi-target correlation tracking.

[0058] This step employs an improved HMA-SORT algorithm to correlate and track detected vehicle targets. This process is a closed-loop system designed to address issues such as background displacement, small target blurring, and occlusion interruptions from the UAV's high-altitude perspective. Specifically, the following four core modules are executed cyclically in logical order:

[0059] Step 3.1: Visual Image Stabilization and Global Motion Compensation (GMC): Process the posterior trajectory state of two consecutive frames and the previous frame to complete visual image stabilization and global motion compensation, obtaining the prior trajectory state after compensating for camera motion; Kalman Filter Prediction Correction Stage: This module serves as the preprocessing stage of the tracking system, mainly acting in the "prediction" stage of the Kalman filter. The specific steps are as follows:

[0060] First, extract the current frame. Compared to the previous frame Image feature points are analyzed using the rotation-invariant ORB (Oriented Fast Rotation Binary Robust Independent Fundamental Features) algorithm, which calculates the gray-level moments of the feature point's neighborhood. (in For pixel grayscale values, , The principal direction of a feature point is determined by the order of the moment. ,in , These are the first-order longitudinal moment and the transverse moment, respectively. It is the arctangent function in the four quadrants;

[0061] A rotation matrix is ​​constructed using the main direction, and the image feature points and feature point descriptors are corrected to the horizontal direction using the rotation matrix, thereby ensuring that features can still be accurately extracted when the UAV yaws or rotates.

[0062] The feature points of the two corrected image frames are paired and mismatches are removed using the Random Sample Consensus (RANSAC) algorithm. The specific loop process is as follows:

[0063] In each round of the "sampling-verification" cycle, firstly, random sampling is performed from all matching feature point pairs. For each pair of points (the minimum number of samples required for the affine transformation), a candidate affine transformation matrix is ​​estimated using these three pairs of points. Then, the candidate affine transformation matrix is ​​used to validate all other feature point pairs, the transformation error is calculated, and point pairs with errors below a preset threshold are counted as inliers, and the current inlier ratio is updated. ;

[0064] According to the updated Substitute into the formula Minimum number of iterations required to recalculate ,in For pre-set reliability;

[0065] The process of "random sampling → model building → interior point statistics → update" is continuously repeated. The process continues until the number of executed loops is no less than the current number. The value is determined by the algorithm. Throughout the iteration, the estimation result with the most interior points is always retained as the optimal candidate. After the loop ends, using all the selected static background interior points (i.e., feature point pairs located in static areas such as roads, buildings, and ground, whose inter-frame motion conforms to the overall background motion caused by the UAV), the affine transformation matrix describing the background motion is calculated using the least squares method. (Including translation, rotation, and scale / shearing parameters), while removing feature points corresponding to dynamic objects such as moving vehicles as outliers.

[0066] Read the posterior state estimates of all trajectories in the previous frame The prior state is obtained through physical prediction using the state transition equation. :

[0067]

[0068] in The state transition matrix is ​​established based on the assumption of uniform motion. The state vector of each trajectory is defined as follows:

[0069]

[0070] in The coordinates of the center point of the target bounding box. The area of ​​the bounding box. Aspect ratio, The horizontal and vertical velocities at the center point The rate of area change; in the inter-frame time interval Under the assumption that the position and scale-related components of the previous frame are extrapolated to the current frame according to the motion model, that is:

[0071]

[0072] velocity components and aspect ratio By keeping it unchanged, a pure kinematic prediction of the target at the current moment can be completed.

[0073] The transformation is a two-dimensional affine transformation obtained from pairs of static background feature points in two adjacent frames through random sampling consistency filtering and least squares estimation; written in homogeneous coordinates as:

[0074]

[0075] In this embodiment Pick Formal and final behavior ,Right now:

[0076]

[0077] in Characterizing the rotation and scaling / shearing of the background within the image plane. The translation is represented. The components of the state vector related to geometric position and scale are transformed according to the affine relationship described above (velocity and other components are synchronously corrected or maintain the corresponding update rules according to the GMC implementation consistent with the tracking framework selected in this embodiment), resulting in the corrected prior state, uniformly denoted as:

[0078]

[0079] This is the corrected prior state after background motion compensation. The physical meaning of this step is to force the prediction result in the coordinate system of the previous frame to be mapped to the background (image) coordinate system of the current frame, so that the subsequent processing of the filter is mainly the real motion of the vehicle relative to the ground, thereby completing "visual image stabilization".

[0080] Step 3.2: Uncertainty-Aware Filtering (NSA-Kalman) – The update preparation stage of Kalman filtering after obtaining the corrected prediction state. Subsequently, this module operates in the "update" phase of the Kalman filter, adaptively adjusting the observation noise covariance matrix to determine the filter's level of confidence in the current detection result, ultimately obtaining the adaptive observation noise covariance matrix and the Kalman gain; the observation noise covariance matrix of the Kalman filter. It is defined as a diagonal matrix, whose diagonal elements correspond to the observation vectors. The measurement variance.

[0081] Two baseline diagonal matrices are preset: the minimum noise covariance matrix. (Corresponding to high-confidence detection) and maximum noise covariance matrix (Corresponding to low-confidence detection). Combined with the detection box confidence output by YOLOv8. (That is, the model's certainty in determining that "a vehicle target does exist within the detection box" is jointly determined by the target's existence probability and the class probability, with a value range of...) The current observation noise covariance matrix is ​​constructed in real time using a linear interpolation formula. ,Right now:

[0082]

[0083] This step is only complete. The calculation of this value, which serves as a quantitative description of observation uncertainty, will be used as an input in the subsequent solution of the Kalman gain.

[0084] Calculate the Kalman gain based on the observed noise covariance matrix:

[0085]

[0086] in, The prior error covariance matrix. Kalman gain. The physical meaning of is a weighting coefficient that determines the relative confidence of the filter in the predicted and observed values ​​during state updates: when When the value is small (high detection confidence). When the value is large, the filter adopts more observations from the current detection box; when When the value is large (low detection confidence, blurry target), Smaller filters retain more predictions based on motion models. When the detected target is blurred, it leads to... At lower levels, An increase in the diagonal element values ​​(i.e., the measurement variance of each observation component) indicates that the system determines the measurement result of the current detection box to be unreliable, which in turn leads to an increase in the Kalman gain. This mechanism allows the filter to automatically reduce the weight of the current unstable observation (detection box) when updating the state in subsequent updates, and instead retain more of the predicted state based on the motion model from the previous time step. This effectively smooths out trajectory jumps caused by detector fluctuations.

[0087] Step 3.3: Hierarchical Hybrid Association Strategy and Spatial Metric Augmentation – Based on Modified Prior State (GMC module output) and the calculated observation noise covariance matrix (Output from NSA-Kalman module) Performs two-layer cascaded matching on all detection boxes in the current frame and existing trajectories to obtain the optimal matching relationship between each trajectory and detection box, thereby maintaining a unique and stable identity ID for each vehicle. This embodiment uses a two-layer cascaded matching mechanism to improve the recall rate for low-confidence targets.

[0088] First, the confidence level of each detection box output by YOLOv8 is calculated. Compared with the preset high score threshold (e.g., 0.6) and low score threshold (e.g., 0.1) for comparison: If Then it will be included in the high-scoring set. ;like Then it is classified into the low-scoring set. ;like If the bounding box is high-scoring, it is discarded and not included in any matching. The first level of association targets the set of high-scoring bounding boxes. Compared with the existing trajectory pool, a comprehensive cost matrix that integrates appearance features and motion state is calculated. .in The appearance feature cost is obtained by calculating the cosine distance between the depth feature vector of the detection box image and the trajectory history feature vector; The Mahalanobis distance cost is calculated using the following formula: In the formula The information vector physically represents the deviation between the actual observed location and the location predicted by the algorithm. It is the inverse of the prediction error covariance matrix, used to normalize the bias to accommodate the uncertainty of motion.

[0089] Subsequently, the Hungarian algorithm is used to globally optimize the matrix and output the optimal matching index list, which clearly indicates which trajectories and which high-resolution detection boxes belong to the same target.

[0090] For successfully matched trajectories, the corresponding detection box coordinates are used. Using the current observation value, combined with the Kalman gain calculated in step 3.2 The final state update after performing Kalman filtering:

[0091]

[0092]

[0093] in, For the updated posterior state estimation, This is the updated posterior error covariance matrix. This is the observation matrix. The above update is performed after the trajectory achieves a valid match at the current level: trajectories that successfully match the high-resolution detection boxes in the first level are adopted after the first-level matching result is determined. Complete a Kalman update; no match in the first layer, but matches the low-scoring set in the second layer. The successfully matched trajectory will also be used after the second-level matching result is determined. Complete the Kalman update described above. If no matching trajectory is obtained after two layers of association, the observation update described above will not be performed (the process will switch to maintenance logic such as trajectory loss counting). Therefore, after a successful match at the second layer, the same form of state update as at the first layer needs to be performed, rather than updating only at the first layer; whether to update depends on whether the matching at each layer is successful, rather than being determined in advance before matching.

[0094] For the remaining trajectories not in the matching list (i.e., not matched in the first layer), proceed to the second layer of association and attempt to match them with the low-scoring set. Matching is then performed. At this stage, considering that blurry low-frame images lead to unreliable appearance features, ReID features are forcibly disabled, and DIoU is introduced to replace the standard IoU as the spatial geometric distance metric. The formula for calculating DIoU is:

[0095]

[0096] in This represents the Euclidean distance between the center of the predicted bounding box and the center of the detected bounding box. This represents the diagonal length of the minimum closure region that can cover both boxes. Since a smaller value in the cost matrix indicates a better match (to adapt to the minimum cost solution mechanism of the Hungarian algorithm), inverting DIoU yields the final matching cost. For all "remaining trajectories" The combination of "low-scoring detection boxes" is calculated one by one. The cost matrix is ​​then filled in, and the second-layer matching result is obtained using the Hungarian algorithm. For trajectories that are successfully matched in the second layer, the corresponding detection boxes are used according to the formula above. Perform a Kalman state update.

[0097] After completing the Kalman state update for all successfully matched trajectories in the current frame, proceed to step 3.4 to perform OCM momentum correction.

[0098] Step 3.4: Observation-Centered Momentum Correction (OCM) and Iterative Process – Trajectory State Correction and Maintenance Phase. Finally, to address the trajectory drift problem caused by the recurrence of vehicle obstruction after prolonged periods, this embodiment monitors the lifecycle state of each trajectory in real time. The trajectory state determination rules are as follows: when a trajectory is newly created, its initial state is active; when the trajectory is continuously... Frame (in this embodiment, the frame is taken) When a trajectory fails to match any detection box, its state changes to "lost." When a lost trajectory re-matches a detection box in the current frame, its state immediately returns to "active." If an active trajectory fails to match any detection box for several consecutive frames, its state changes to "lost." Once the lost trajectory successfully matches a detection box in the current frame (i.e., the reconnected frame, the frame in which the trajectory returns to active from "lost"), its state immediately returns to "active," and the momentum correction mechanism is used to correct the target velocity at the current moment. Specifically:

[0099] Extract the coordinates of the last frame before the trajectory was lost. Coordinates of the current reconnected frame Calculate the average virtual velocity vector during the loss period. .

[0100] For trajectories that have been lost and recovered to be active, after the correspondence between the reconnected frame and the detection box has been established, the calculated virtual velocity vector will be... Write the prior state currently used for Kalman updates to this trajectory. velocity components This covers velocity estimation biases that occur during periods of prolonged lack of observation; subsequently, the coordinates of the detection frame are used in reconnected frames. As observation Perform Kalman filter state update:

[0101]

[0102] Obtain the posterior state This ensures that the position and velocity after reconnection are within a reasonable range, and the trajectory ID remains continuous. Momentum correction, as a priori velocity injection before this observation update, is performed in conjunction with the Kalman update of the matched trajectory in this frame in step 3.3: the reconnected trajectory is completed before being substituted into the above formula. Upon writing, the trajectory within this frame still uses the reconnected detection box as the sole observation source for a single state update. In other implementations, an AND operation can be applied to the velocity posterior after the observation update. Equivalent constraints are equivalent to prior injection.

[0103] Used for filter state compensation to suppress velocity divergence caused by prolonged periods without observation; it is not used as an independent measurement of vehicle physical velocity. During periods when the trajectory is in a lost-track state, an observation-only update (OOS) strategy is implemented, and updates to the filter error covariance matrix are stopped. (Or freeze the corresponding item as selected) to avoid unreasonable expansion of the error range when there is a lack of observations. After completing the above processing, output the current frame. The vehicle tracking results are used to read the next frame. Return to step 3.1 to begin the next iteration.

[0104] Step S4: Multi-road condition adaptive speed measurement

[0105] After obtaining a stable vehicle pixel trajectory, this step does not directly apply a single planar projection formula, but instead employs an adaptive processing flow of "first determining road conditions, then matching the model." This flow combines laser point clouds and visual trajectories to intelligently identify straight roads, curves, and slopes, and calls upon the corresponding geometric model to calculate the vehicle's absolute ground velocity. Specifically, the ground sampling distance (GSD) is first calculated. The current vertical altitude of the drone above the ground is then obtained using a laser rangefinder. And read the pitch angle of the gimbal camera. The actual slant distance from the camera's optical center to the road surface imaging center is calculated based on trigonometric relationships. Combined with sensor pixel size and lens focal length The ground sampling distance was calculated. This parameter is used as a unified conversion factor from pixel coordinates to physical coordinates in the subsequent speed measurement models for straight roads, curves, and slopes, i.e., the actual physical displacement. Pixel displacement GSD.

[0106] Next, intelligent road condition assessment is performed. The historical trajectory queue of vehicles and the sparse point cloud of the ground are maintained in real time, and logical judgments are executed according to priority. To avoid misjudgments caused by vehicle lane changes or vibrations, this embodiment employs a curve recognition method based on macroscopic motion vectors. Specifically, the current frame is selected... vehicle centroid coordinates And backtrack to select the first Frame coordinates Passing the exam Frame coordinates (in (For a preset step size, e.g., 10 frames). Based on these three time points, construct two continuous macroscopic vectors: from points... point to Preorder macroscopic vector , and from point point to Current macroscopic vector Calculate the angle between these two vectors. If the included angle continues to exceed the threshold within a preset time window. (like If a significant and continuous change in the vehicle's direction of motion occurs, it indicates that the vehicle is in a curve. If a curve is not detected, the slope is further identified based on the absolute elevation difference. The current moment is selected. The vehicle's ground projection point is used as the rear endpoint. and the backtracking time window The following frame (i.e. The ground projection point corresponding to the frame is used as the front-end point. Use laser point cloud data to query the corresponding altitude of these two points. and If the absolute elevation difference between two points Exceeding the preset threshold If the slope is 1.5 meters, the vehicle is determined to be traveling on a slope. If none of the above conditions are met, the road is determined to be straight.

[0107] Finally, the corresponding geometric solution model is automatically activated based on the judgment result. If the vehicle is traveling on a curve, the least squares circle fitting model is activated. It is assumed that the trajectory points within the judgment interval are distributed along the circle equation. Above, among which Let be the coordinates of the center of the circle to be determined. Let be the radius. To solve using the least squares method, the equation is linearized as follows: In this formula, the parameter to be determined is defined as... , constant term Construct an overdetermined system of equations by selecting the coordinates of all trajectory points within the decision interval, and then obtain the optimal solution vector. Then, the coordinates of the center of the circle were solved. and radius Subsequently, the leading edge point of the vehicle within the judgment interval is calculated. and the end point Relative to the center of the circle central angle Using the arc length formula Calculate the actual curved distance traveled by the vehicle to obtain the relative speed after curvature correction. .

[0108] If the vehicle is on a slope, activate the local plane fitting model. To obtain the actual slope distance, the road gradient needs to be calculated. Select several ground laser points around the vehicle target to fit the local spatial plane equation. To avoid confusion with the center symbol, let the plane equation be... ,in For height, The horizontal coordinate is... Representing the plane in shaft and Gradient coefficients in the axial direction, The intercept is used. Construct the error function. The optimal plane parameters are obtained by solving for the partial derivatives. In this model, the most critical parameters are determined by... Normal vector of a given plane Slope angle Defined as the normal vector with the vertical vector The angle between them is calculated using the following formula: Using this angle to measure the horizontal projection speed Compensation is performed to obtain the actual driving speed along the slope. If the path is determined to be a straight section, the relative velocity is calculated directly using Euclidean distance. Regardless of the model used, the final calculation incorporates the UAV's three-axis flight velocity vector. Through vector composition formula The absolute ground speed of the vehicle is calculated. Through the above complete process, the present invention achieves a balance between accuracy and speed at the detection end using PConv and SimAM, solves the interference of camera motion and occlusion at the tracking end using HMA-SORT, and achieves high-precision measurement at the speed measurement end using a multi-road condition geometric model.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. Any modifications, equivalent substitutions, or improvements made in accordance with the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting, tracking, and measuring the speed of unmanned aerial vehicles (UAVs) based on deep learning and multi-sensor fusion, characterized in that, Includes the following steps: Step S1: Data Acquisition and Spatiotemporal Synchronization: The video image stream of road vehicles is acquired using the gimbal camera on the UAV, and the ground altitude data is acquired at a preset frequency using the laser ranging module on the UAV. The flight status parameters of the UAV corresponding to each frame of the image are recorded synchronously. The flight status parameters of the UAV include the UAV's three-dimensional coordinates, flight speed vector, and gimbal pitch angle. Step S2: Construct an instance segmentation model based on YOLOv8. Input the acquired video images into the instance segmentation model, extract features through the backbone network, perform multi-scale feature fusion using the neck network, and finally output the vehicle target category, confidence score, bounding box coordinates, and instance segmentation mask through the decoupled detection head. Step S3: Use the HMA-SORT algorithm to perform correlation tracking on the detected vehicle targets to obtain the continuous trajectory points of the vehicle in the image pixel coordinate system; Step S4: Calculate the ground sampling distance based on the laser ranging data and camera parameters; combine the vehicle's continuous trajectory points and the UAV's flight status to determine whether the vehicle is traveling on a straight road, a curve, or a slope; use the corresponding geometric model to calculate the vehicle's relative displacement for different road conditions; and combine the UAV's velocity vector to calculate the vehicle's absolute ground speed.

2. The method for detecting, tracking, and measuring the speed of unmanned aerial vehicles based on deep learning and multi-sensor fusion according to claim 1, characterized in that, The instance segmentation model uses an improved YOLOv8 network, specifically improved as follows: Backbone Network: The C2f module used to process feature maps at scales P2, P3, P4, and P5 in the YOLOv8 network is replaced with the C2f_Faster module; the C2f_Faster module utilizes partial convolution operators to convert the channels of the input feature map. It is divided into two parts: the first part is the channel. conduct Convolution is used to extract spatial features, and the remaining channels Preserving the identity mapping without performing convolution calculations, where The total number of channels in the input feature map. The number of channels participating in the convolution calculation; then through Point convolution fuses features across all channels; Neck network: The C2f module in the feature fusion stage is replaced with the C2f_SimAM module; after feature upsampling and concatenation operations, the C2f_SimAM module embeds a parameterless attention mechanism; the parameterless attention mechanism calculates the importance weight of each neuron in the feature map by defining an energy function, and performs element-wise multiplication of the importance weight with the original feature map to enhance the target features.

3. The method for detecting, tracking, and measuring the speed of unmanned aerial vehicles based on deep learning and multi-sensor fusion according to claim 2, characterized in that, The specific method by which the parameterless attention mechanism calculates the importance weight of each neuron in the feature map by defining an energy function is as follows: Define the energy function To measure neurons Linear separability from surrounding neurons: in For neurons Energy value, The value of the target neuron. The linear transformation estimate of the target neuron. The values ​​of the surrounding neurons, These are the corresponding labels or reference values ​​for the surrounding neurons. This represents the total number of neurons in that channel. and These are the weights and biases of the linear transformation, respectively; The minimum energy for each neuron is calculated by minimizing the energy function. : in For the minimum energy value, and These are the mean and variance of the input feature map along the channel dimension, respectively. The value of the current target neuron. The regularization coefficient is used. Calculate attention weights It is the reciprocal of energy, that is The importance weights of each neuron are obtained by normalizing the weights using the Sigmoid function.

4. The method for detecting, tracking, and measuring the speed of unmanned aerial vehicles based on deep learning and multi-sensor fusion according to claim 1, characterized in that, The specific steps for using the HMA-SORT algorithm to correlate and track detected vehicle targets and obtain continuous trajectory points of the vehicle in the image pixel coordinate system are as follows: Feature point pairing and mismatch deletion are performed on two consecutive frames of images to obtain the affine transformation matrix. Based on the posterior state of the trajectory in the previous frame, physical prediction is performed to obtain the prior state, and the prior state is corrected to obtain the corrected prior state. The current observation noise covariance matrix is ​​constructed in real time using a linear interpolation formula, and the Kalman gain is calculated based on the observation noise covariance matrix. Based on the corrected prior state and observation noise covariance moment, a two-level cascaded matching is performed on all detection boxes in the current frame and existing trajectories to obtain the optimal matching relationship between each trajectory and the detection box. Monitor the lifecycle status of the track and perform momentum correction on the track that recovers from the lost state to the active state.

5. The method for detecting, tracking, and measuring the speed of unmanned aerial vehicles based on deep learning and multi-sensor fusion according to claim 4, characterized in that, The method involves performing feature point pairing and mismatch removal on two consecutive image frames to obtain the affine transformation matrix. Based on the posterior trajectory state of the previous frame, a physical prediction is performed to obtain the prior state, which is then corrected. The specific method for obtaining the corrected prior state is as follows: S3.1.1: Extract image feature points from the current frame t and the previous frame t−1, and use the ORB algorithm to calculate the gray-level moments of the feature point neighborhood. Determine the principal direction of the feature points ,in , These are the first-order longitudinal moment and the transverse moment, respectively. It is the arctangent function in the fourth quadrant. For pixel grayscale values, , Let be the order of the moment; S3.1.2: Construct a rotation matrix using the principal direction to correct the image feature points and feature point descriptors to the horizontal direction; S3.1.3: Pair the feature points of the two corrected frames, use the RANSAC algorithm to remove mismatches, and filter points within the static background; S3.1.4: Using the selected static background in-points, calculate the affine transformation matrix describing the background motion using the least squares method. ; S3.1.5: Read the posterior state estimates of all trajectories from the previous frame. The prior state is obtained through physical prediction using the state transition equation. : in This is the state transition matrix; The state vector for each trajectory is defined as: in The coordinates of the center point of the target bounding box. The area of ​​the bounding box. Aspect ratio, The horizontal and vertical velocities at the center point The rate of area change; in the inter-frame time interval Under the assumption that the position and scale-related components of the previous frame are extrapolated to the current frame according to the motion model, that is: Apply an affine transformation matrix to the components of the state vector that are related to geometric position and scale. The modified prior state is obtained by transformation and is uniformly denoted as: This is the corrected prior state after background motion compensation.

6. The method for detecting, tracking, and measuring the speed of unmanned aerial vehicles based on deep learning and multi-sensor fusion according to claim 4, characterized in that, The current observation noise covariance matrix is ​​constructed in real time using a linear interpolation formula. The specific method for calculating the Kalman gain based on the observation noise covariance matrix is ​​as follows: S3.2.1: Combining the detection box confidence score with YOLOv8 output The current observation noise covariance matrix is ​​constructed in real time using a linear interpolation formula. Specifically: In the formula, It is the minimum noise covariance matrix. The maximum noise covariance matrix, S3.2.2: Calculate the Kalman gain based on the observed noise covariance matrix. : in, Let be the prior error covariance matrix.

7. The method for detecting, tracking, and measuring the speed of unmanned aerial vehicles based on deep learning and multi-sensor fusion according to claim 4, characterized in that, Based on the corrected prior state and observation noise covariance moments, the specific method for performing a two-level cascaded matching between all detection boxes in the current frame and existing trajectories to obtain the optimal matching relationship between each trajectory and the detection box is as follows: S3.3.1: Divide the detection box confidence scores into a high-score set and a low-score set, specifically as follows: like Then it will be included in the high-scoring set. ;like Then it is classified into the low-scoring set. ;like If the match is not found, it will be discarded and will not be included in any matching process. S3.3.2: First-level association: Calculate the comprehensive cost matrix for Dhigh and the existing trajectory pool: in, For the sake of appearance features, This is the cost of the Mahalanobis distance; Weighting ratio; The optimal matching index is solved using the Hungarian algorithm. For the successfully matched trajectory, Kalman filtering is performed to update the state by combining the Kalman gain K. S3.3.3: Second-level association: For trajectories and low-scoring sets that did not match in the first-level association. Using DIoU as the spatial geometric distance metric, the matching cost is obtained by inverting DIoU: in This represents the Euclidean distance between the center of the predicted bounding box and the center of the detected bounding box. This represents the diagonal length of the smallest closure region that can cover two boxes; The matching results are obtained using the Hungarian algorithm, and Kalman filtering is performed on the successfully matched trajectories to update their states.

8. The method for detecting, tracking, and measuring the speed of unmanned aerial vehicles based on deep learning and multi-sensor fusion according to claim 4, characterized in that, The specific method for momentum correction of a trajectory recovering from a lost state to an active state is as follows: Track status determination rules: Newly created tracks are initially in an active state; if no detection box is matched for n consecutive frames, the track is switched to a lost state. If the lost state trajectory matches the detection box, it will immediately return to the active state, where n is a preset threshold. Extract the coordinates of the last frame before the lost active trajectory. Coordinates of the current reconnected frame Calculate the average virtual velocity vector Write the average virtual velocity vector into the current prior state of the trajectory. velocity components Reconnection frame detection box coordinates As observation Perform Kalman filtering to update the state and obtain the posterior state. : After processing is complete, output the vehicle tracking result of the current frame, read the next frame and return to step 3.1 to execute the loop.

9. The method for detecting, tracking, and measuring the speed of unmanned aerial vehicles based on deep learning and multi-sensor fusion according to claim 1, characterized in that, The specific method for determining whether a vehicle is traveling on a straight line, a curve, or a slope by combining continuous vehicle trajectory points with the drone's flight status is as follows: S4.2.1: Curve detection: Select the current frame Vehicle centroid coordinates And backtrack to select the first Frame coordinates Passing the exam Frame coordinates ,in Preset step size; From point point to Preorder macroscopic vector , and from point point to Current macroscopic vector ; Calculate the angle between these two vectors. If the included angle The threshold is continuously exceeded within the preset time window. If so, it is determined to be a curve driving state; S4.2.2: Slope Determination: Select the current time. The vehicle's ground projection point is used as the rear endpoint. and the backtracking time window The ground projection point corresponding to the subsequent frame is used as the front-end point. ; Use laser point cloud data to query the corresponding altitude of these two points. and If the absolute elevation difference between two points Exceeding the preset threshold If so, the vehicle is determined to be traveling on a slope; S4.2.3: Straight Road Judgment: If the curve or slope judgment is not triggered, the vehicle is judged to be in a straight road driving state.

10. The method for detecting, tracking, and measuring the speed of unmanned aerial vehicles based on deep learning and multi-sensor fusion according to claim 1, characterized in that, The specific method for calculating the relative displacement of vehicles using corresponding geometric models for different road conditions, and combining this with the UAV velocity vector to calculate the absolute ground velocity of the vehicles, is as follows: S4.3.1: Speed ​​Measurement on Curves: Assuming the trajectory points within the judgment interval are distributed along the circular equation... Above, among which Let be the coordinates of the center of the circle to be determined. Using the radius, linearize the circle equation to: In the form of, , , Construct an overdetermined system of equations to solve for the parameters (a,b,c), inversely solve for the center (A,B) and radius R, and calculate the vehicle's leading edge point within the judgment interval. and the end point Relative to the center of the circle central angle Using the arc length formula Calculate the actual curved distance traveled by the vehicle to obtain the relative speed after curvature correction. ; S4.3.2: Slope Speed ​​Measurement: Activate the local plane fitting model, select several ground laser points around the vehicle, and fit the spatial plane equation. ,in For height, The horizontal coordinate is... Representing the plane in shaft and Gradient coefficients in the axial direction, The intercept; Constructing the error function Solve for the optimal plane parameters; Calculate the plane normal vector with the vertical vector The included angle β: ; Using the included angle β to project the velocity horizontally Compensation is performed to obtain the actual driving speed along the slope. ; S4.3.3: Straight-line speed measurement: The relative speed Vrel of vehicles is calculated using Euclidean distance; S4.3.4: Absolute ground velocity calculation: combining the UAV's three-axis flight velocity vector Through vector composition formula Calculate the vehicle's absolute ground speed.

Images