Vehicle position and speed joint estimation method based on mobile unmanned aerial vehicle aerial video

Abstract

The application discloses a vehicle position and speed joint estimation method based on a mobile unmanned aerial vehicle aerial video, and the method comprises the following steps: acquiring a video, a flight control log and camera parameters, performing time synchronization, obtaining the unmanned aerial vehicle position, the camera posture and the center coordinates of each frame, extracting a vehicle observation area, determining a ground projection point, converting the projection point into a pixel ray, combining the flight control posture and the camera external parameter, and converting to a unified geographic coordinate system; the ray and a road geometric model are intersected to obtain a vehicle position observation; target correlation and trajectory maintenance are performed to construct a real coordinate trajectory; a state space model is used to jointly estimate the vehicle position and speed, abnormal speed is inhibited through motion constraint, and a trajectory ID, a coordinate position, a speed size and a direction are output; and the vehicle image observation is converted into a real coordinate in the unified geographic coordinate system through the "image observation-pixel ray-road geometric intersection" method, so that the vehicle image observation is adapted to complex working conditions such as unmanned aerial vehicle translation, turning, lifting and changes of a cloud head posture.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent transportation perception technology, and in particular to a method for joint estimation of vehicle position and speed based on aerial video captured by a mobile drone. Background Technology

[0002] With the development of drone aerial photography technology, acquiring road vehicle trajectories and speeds using drone video has gradually become an important means of intelligent traffic perception. However, during actual flight, drones typically experience translation, turning, ascent and descent, attitude disturbances, and gimbal angle changes, causing the vehicle's displacement in the image plane to be affected by both the vehicle's own motion and the camera's motion. Therefore, accurately reconstructing the vehicle's true position and speed from mobile drone aerial video under complex flight conditions is a significant technical challenge in the field of drone-based traffic perception.

[0003] In existing technologies, vehicle speed estimation methods based on UAV aerial video typically rely on stationary hovering UAVs and can be mainly categorized into fixed mapping relationship methods, image domain detection and tracking methods, and moving camera motion compensation methods. Among these, methods based on fixed mapping relationships usually convert vehicle image displacement into ground displacement based on camera field of view, flight altitude, ground sampling distance, road control points, or homography matrix. However, these methods depend on road geometric calibration or stable image-to-ground mapping relationships, making them susceptible to homography errors when dealing with distant vehicles, non-planar roads, and significant changes in viewing angle.

[0004] Image-domain detection and tracking methods typically employ YOLO-like detectors, ByteTrack, DeepSORT, or other multi-object tracking algorithms. They first obtain vehicle detection boxes and pixel trajectories, then estimate vehicle speed using prior knowledge of fixed scale, fixed GSD, road width, lane length, or vehicle size. However, since the core trajectory resides in the image coordinate system, for moving drone scenarios, the target displacement in the image domain is not equivalent to the vehicle's actual road displacement, making it difficult to distinguish between the vehicle's actual motion and camera motion. Methods based on moving camera motion compensation primarily focus on target tracking stability under drone motion conditions. They reduce the impact of moving cameras on target association through camera motion compensation, homogeneous transformation, ground plane mapping, or unified viewpoint representation. However, their main objective remains improving multi-object tracking performance, without systematic research on road vehicle trajectory extraction and speed measurement tasks.

[0005] Therefore, there is a need for a method that can fully utilize the frame-by-frame changing flight control pose, camera parameters, and road geometry information of mobile UAVs to convert vehicle image observations into real coordinate observations and perform joint estimation of vehicle position and speed in a unified real geographic coordinate system, so as to achieve stable estimation of road vehicle position and speed under complex flight conditions of mobile UAVs. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide a method for joint estimation of vehicle position and speed based on aerial video from mobile drones, so as to solve one or more problems in the prior art.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] A method for joint estimation of vehicle position and velocity based on aerial video from mobile drones includes the following steps:

[0009] S1. Acquire aerial video data, flight control log data and camera parameters of the mobile drone, synchronize the video frames of the aerial video with the flight control log in time and perform interpolation alignment to obtain the drone position, camera attitude and camera center coordinates corresponding to each video frame.

[0010] S2. Perform image observation and extraction on road vehicles in video frames to obtain vehicle observation areas, and determine ground projection representative points from vehicle observation areas;

[0011] S3. Based on the camera intrinsic parameters in the camera parameters, the ground projection representative points are converted into pixel rays in the camera coordinate system. Combined with the UAV flight control pose and the camera extrinsic parameters in the camera parameters, the pixel rays are converted to a unified real geographic coordinate system.

[0012] S4. Under a unified real geographic coordinate system, the transformed pixel ray is intersected with the prior known road geometry model to obtain the vehicle's position observation under the real geographic coordinate system; the motion pattern of the ground vehicle is modeled using a state space model, and the motion state of the vehicle in the next frame is predicted based on the vehicle trajectory of the previous frame. Then, the vehicle observation results and vehicle motion state prediction results of the frame are associated with the target and the trajectory is maintained in the real geographic coordinate system.

[0013] S5. Based on the data association results of S4, perform the posterior update step of the state space model to obtain the joint recursive estimation results of vehicle position and speed, and suppress abnormal speeds through motion constraints. Output the vehicle's trajectory ID, true coordinate position, speed magnitude and speed direction in a unified real geographic coordinate system.

[0014] Furthermore, in step S1:

[0015] The aerial video data includes consecutive video frames, frame number, frame rate, or frame timestamp.

[0016] The flight control log data includes the UAV's attitude, altitude, flight speed, and log timestamps;

[0017] The camera parameters include camera intrinsic parameters and camera extrinsic parameters.

[0018] Furthermore, in step S1:

[0019] For each video frame, the corresponding UAV position and flight control attitude angle are obtained by time alignment and interpolation based on the timestamp of that frame in the flight control log, as shown in the following formula:

[0020]

[0021] In the formula: Indicates the first The drone position corresponding to the frame. Indicates the flight control attitude angle. This represents flight control log data. Indicates the first Frame corresponds to timestamp. This indicates operations such as time synchronization, interpolation, or maintaining adjacent frames.

[0022] Based on the synchronized UAV position, flight control attitude, and camera parameters, calculate the camera center coordinates and camera attitude corresponding to the frame to form a frame-by-frame pose sequence. The calculation formula is shown below:

[0023]

[0024] In the formula: Indicates the first Frame image, Indicates the corresponding timestamp. Represents the camera intrinsic parameter matrix. Indicates the first The location of the frame camera center in a unified real geographic coordinate system. Indicates the first The extrinsic parameter matrix from the frame camera coordinate system to the unified real geographic coordinate system.

[0025] Furthermore, the specific steps in step S2 are as follows:

[0026] S21. Perform road vehicle image observation and extraction on each frame of aerial image to obtain vehicle observation results. The vehicle observation results include vehicle observation area, vehicle category and detection confidence. The vehicle observation area is a horizontal detection box, a rotated detection box, an instance segmentation mask or a set of key points.

[0027] S22. Generate a set of candidate ground projection representative points based on the vehicle observation area;

[0028] S23. Based on the camera's viewing angle, the shape of the vehicle's observation area, road area constraints, and the predicted position from historical trajectories, determine the final ground projection representative point from the set of candidate ground projection representative points.

[0029] Furthermore, the vehicle observation area generates a set of candidate ground projection representative points, including:

[0030] If the vehicle observation area is a horizontal detection box, the candidate points include the center point of the box, the midpoint of the bottom edge, or the midpoint of the side edge near the camera.

[0031] If the vehicle observation area is a rotating detection frame, then the candidate points include the center point of the rotating frame or the midpoint of the edge;

[0032] If the vehicle observation area is an instance segmentation mask, then the candidate points include the mask centroid, mask outline points, or approximate points of the vehicle grounding point.

[0033] Furthermore, step S3 specifically includes the following steps:

[0034] S31. Extract the pixel coordinates of the representative points of the ground projection;

[0035] S32, Based on camera intrinsic parameter matrix The ground projection representative point is converted from pixel coordinates to normalized pixel rays in the camera coordinate system, and the calculation formula is as follows:

[0036]

[0037] In the formula: Indicates the first The first frame The ray in the camera coordinate system corresponding to each vehicle representative point. This represents the vector normalization operation;

[0038] S33. Based on the camera pose corresponding to each frame, the pixel ray in the camera coordinate system is transformed to a unified real geographic coordinate system using the camera extrinsic parameters to obtain the vehicle observation ray in the real geographic coordinate system. The calculation formula is as follows:

[0039]

[0040] In the formula: Represents the vehicle observation ray in a unified real geographic coordinate system. Indicates the first The extrinsic parameter matrix from the frame camera coordinate system to the unified real geographic coordinate system.

[0041] Furthermore, step S4 specifically includes the following steps:

[0042] S41. Construct a road geometry model, wherein the road geometry model is a fixed road plane, a segmented road plane, a local road plane fitted by road control points, a lane constraint plane, or a digital elevation model;

[0043] S42. Find the intersection between the vehicle observation ray and the road geometry model, and calculate the intersection ratio. The calculation formula is as follows:

[0044]

[0045] In the formula: Indicates the first Frame Camera Center Down direction coordinates Represents rays Down direction component, This represents the intersection ratio of the ray from the camera center to the road plane;

[0046] S43. When the intersection ratio is positive and the ray direction satisfies the valid condition, the vehicle's position observation in a unified real geographic coordinate system is obtained; otherwise, the vehicle observation is marked as an invalid projection. The calculation formula for the vehicle's position observation in a unified real geographic coordinate system is as follows:

[0047]

[0048] In the formula: Indicates the first The first frame The position of each vehicle is observed in a unified real geographic coordinate system;

[0049] S44. Perform target association and trajectory maintenance under a unified real geographic coordinate system for valid vehicle position observations. Associate the valid real coordinate observations of vehicles in the current frame with the existing vehicle trajectories in the previous frame, and update the vehicle trajectory set. The calculation formula is as follows:

[0050]

[0051] In the formula: Indicates the first The frame contains all valid vehicle real coordinate observations. This indicates that the previous frame already contained a set of vehicle trajectories. This represents the updated set of vehicle trajectories. This indicates the data association process.

[0052] Furthermore, step S5 specifically includes the following steps:

[0053] S51. Establish a state vector for each vehicle trajectory. The state vector shall include at least the vehicle's northward position, eastward position, northward velocity component, and eastward velocity component in the real coordinate system. The calculation formula is shown below:

[0054]

[0055] In the formula: and Representing vehicle targets respectively In the The north and east positions of the frame, and These represent the northward velocity component and the eastward velocity component, respectively.

[0056] S52. A state-space model is used to jointly estimate the vehicle position and velocity. The vehicle motion state estimation result is obtained through vehicle state prediction and recursive updating of ground vehicle position observations. The calculation formula is shown in the following formula:

[0057]

[0058] In the formula: Represents the state transition matrix. Represents the observation matrix. This indicates the actual coordinates of the vehicle. Indicates process noise. Indicates observation noise;

[0059] S53. Calculate the vehicle speed based on the vehicle motion state estimation results, and determine the vehicle speed direction from the speed components. The calculation formula is as follows:

[0060]

[0061] In the formula: Indicates vehicle target In the Frame speed;

[0062] S54. Perform abnormal velocity suppression on the estimated velocity results. The abnormal velocity suppression includes: combining the maximum reasonable velocity, the maximum reasonable acceleration and the observation residual, and performing abnormal marking, confidence reduction, delayed output or rejection processing on results with velocity jumps, exceeding the reasonable velocity range or excessive residuals.

[0063] S55. Output the vehicle trajectory ID, real coordinate position, speed magnitude and speed direction in a unified real geographic coordinate system.

[0064] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0065] (i) This invention utilizes the dynamic change of flight control pose and camera parameters frame by frame by mobile UAV. Through the real coordinate projection method of "image observation-pixel ray-road geometry intersection", vehicle image observation is converted into vehicle position observation under a unified real geographic coordinate system. It does not rely on a fixed scale or a single road homography matrix, and can better adapt to complex flight conditions such as UAV translation, turning, climbing and gimbal attitude changes.

[0066] (ii) The present invention performs vehicle target association and joint estimation of position and speed in a unified real geographic coordinate system, avoiding the errors caused by camera movement, detection box jitter and target ID switching in the image domain pixel trajectory, and the trajectory extraction and speed estimation results are more stable and reliable.

[0067] (III) The present invention directly outputs the vehicle speed state from the state space model. Compared with post-processing methods such as sliding window averaging or moving average, it can reduce speed response lag while suppressing observation noise and avoid weakening the real acceleration and deceleration process of the vehicle due to excessive smoothing. It is more suitable for application scenarios that are sensitive to speed at any time, such as highway speed measurement and abnormal driving behavior recognition.

[0068] (iv) This invention integrates an abnormal speed suppression mechanism based on motion constraints, which can eliminate or correct abnormal speed output caused by detection noise, miscorrelation, etc., and improve engineering applicability. Attached Figure Description

[0069] Figure 1 The diagram illustrates a flowchart of a method for jointly estimating vehicle position and speed based on aerial video from a mobile drone, according to an embodiment of the present invention.

[0070] Figure 2 This diagram illustrates the intersection of pixel rays and road geometric model in the vehicle position and speed joint estimation method based on mobile drone aerial video according to an embodiment of the present invention.

[0071] Figure 3 This diagram illustrates the vehicle target association and state estimation in the NED coordinate system of the vehicle position and velocity joint estimation method based on mobile drone aerial video according to an embodiment of the present invention.

[0072] Figure 4 The diagram shows the lane width scale verification results of the vehicle position and speed joint estimation method based on mobile drone aerial video according to an embodiment of the present invention.

[0073] Figure 5 The diagram shows a comparison of vehicle trajectory and velocity curves for the vehicle position and velocity joint estimation method based on aerial video from a mobile drone, according to an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description of the vehicle position and speed joint estimation method based on mobile UAV aerial video proposed in this invention, in conjunction with the accompanying drawings and specific embodiments, will provide a more detailed explanation. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, used only to facilitate and clearly illustrate the purpose of the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for illustrative purposes to aid those skilled in the art and are not intended to limit the implementation conditions of this invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to the size, without affecting the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0075] Please refer to the following: Figure 1 A method for joint estimation of vehicle position and velocity based on aerial video from a mobile drone includes the following steps:

[0076] S1. Data Acquisition and Synchronization. Acquire aerial video data, flight control log data, and camera parameters from the mobile drone. Perform time synchronization and interpolation alignment between the video frames and flight control logs to obtain the drone's position, camera attitude, and camera center coordinates for each video frame. Specifically:

[0077] The aerial video data includes consecutive video frames, frame number, frame rate, or frame timestamp.

[0078] The flight control log data includes the UAV's attitude, altitude, flight speed, and log timestamps;

[0079] The camera parameters include camera intrinsic parameters and camera extrinsic parameters.

[0080] Therefore, the above input data can be uniformly represented as shown in Equation 1 below:

[0081] (1)

[0082] In the formula: This represents aerial video data. This represents flight control log data. This represents the set of camera parameters.

[0083] Furthermore, the aerial video frames are synchronized with the flight control log. Specifically, for each video frame, the corresponding UAV position and flight control attitude angle are obtained by time alignment and interpolation based on the frame's timestamp in the flight control log, as shown in Equation 2 below:

[0084] (2)

[0085] In the formula: Indicates the first The drone position corresponding to the frame. Indicates the flight control attitude angle. Indicates the first Frame corresponds to timestamp. This indicates operations such as time synchronization, interpolation, or maintaining adjacent frames.

[0086] Furthermore, based on the synchronized UAV position, flight control attitude, and camera parameters, the camera center coordinates and camera attitude corresponding to that frame are calculated to form a frame-by-frame pose sequence, as shown in Equation 3 below:

[0087] (3)

[0088] In the formula: Indicates the first Frame image, Indicates the corresponding timestamp. Represents the camera intrinsic parameter matrix. Indicates the first The location of the frame camera center in a unified real geographic coordinate system. Indicates the first The extrinsic parameter matrix is ​​used to transform the camera coordinate system to a unified real geographic coordinate system. This frame-by-frame pose sequence is used for the transformation of subsequent vehicle image observations to real vehicle ground position observations.

[0089] S2. Image Observation and Extraction. Image observation and extraction are performed on roads and vehicles in the video frames to obtain vehicle observation areas. Then, representative ground projection points are determined from these vehicle observation areas. Specifically:

[0090] S21. Perform road vehicle image observation and extraction on each frame of aerial image to obtain vehicle observation results. The vehicle observation results include vehicle observation area, vehicle category, and detection confidence. The calculation formula is shown in Equation 4 below:

[0091] (4)

[0092] In the formula: Indicates the first The first frame Individual vehicle observation results Indicates the vehicle observation area. Indicates vehicle category, This indicates the confidence level of the detection.

[0093] S22. Generate a candidate ground projection representative point set based on the vehicle observation area. The ground projection representative points are used to characterize the vehicle's position on the road geometry model, and the calculation formula for the candidate set is shown in Equation 5 below:

[0094] (5)

[0095] In the formula: Indicates the first A candidate ground projection representative point This indicates the number of candidate points.

[0096] The vehicle observation area can be a horizontal detection box, a rotated detection box, an instance segmentation mask, or a set of key points. If the vehicle observation area is a horizontal detection box, candidate points include the center point of the box, the midpoint of the bottom edge, or the midpoint of the side edge near the camera; if the vehicle observation area is a rotated detection box, candidate points include the center point of the rotated box or the midpoint of the edge; if the vehicle observation area is an instance segmentation mask, candidate points include the mask centroid, mask contour points, or approximate points of the vehicle grounding point.

[0097] S23. Based on the camera's viewing angle, the shape of the vehicle's observation area, road area constraints, and the predicted position from historical trajectories, the final ground projection representative points are determined from the candidate ground projection representative point set, and the vehicle image observation results are formed. The calculation formula is shown in Equation 6 below:

[0098] (6)

[0099] In the formula: This represents the final selected representative point for the vehicle's ground projection. This vehicle image observation result serves as input for subsequent pixel ray construction and true coordinate projection.

[0100] S3. Pixel Ray Construction and Coordinate Transformation. Based on the camera intrinsic parameters in the camera parameters, the ground projection representative point is converted into a pixel ray in the camera coordinate system. Then, combined with the UAV flight control pose and the camera extrinsic parameters in the camera parameters, the pixel ray is transformed to the NED coordinate system or the local geographic coordinate system. Specifically:

[0101] S31. Extract the pixel coordinates of the representative points of the ground projection, and define the first... The first frame The formula for calculating the ground projection representative point of a vehicle target is shown in Equation 7 below:

[0102] (7)

[0103] In the formula: and These represent the horizontal and vertical pixel coordinates of the representative point in the image, respectively.

[0104] S32, Based on camera intrinsic parameter matrix The ground projection representative point is converted from pixel coordinates to normalized pixel rays in the camera coordinate system, and the calculation formula is shown in Equation 8 below:

[0105] (8)

[0106] In the formula: Indicates the first The first frame The ray in the camera coordinate system corresponding to each vehicle representative point. This represents the vector normalization operation.

[0107] S33. Based on the camera pose corresponding to each frame, the pixel ray in the camera coordinate system is transformed to a unified real geographic coordinate system using the camera extrinsic parameters to obtain the vehicle observation ray in the real geographic coordinate system. The calculation formula is shown in Equation 9 below:

[0108] (9)

[0109] In the formula: This represents the vehicle observation ray in the true geographic coordinate system. Indicates the first The extrinsic parameter matrix from the frame camera coordinate system to the unified real geographic coordinate system.

[0110] This step transforms the vehicle image observation from two-dimensional pixels into an observation ray in a unified real geographic coordinate system, preparing for subsequent intersection with the road geometry model.

[0111] S4. Ray Intersection and Realistic Coordinate Tracking. Under a unified real geographic coordinate system, the transformed pixel rays are intersected with the road geometry model to obtain the vehicle's position observation in the real geographic coordinate system. In the real geographic coordinate system, the ground observation results and the vehicle state prediction results based on the vehicle trajectory of the previous frame are used for target association and trajectory maintenance to construct the vehicle's realistic coordinate trajectory. For details, please refer to [reference needed]. Figure 2 and Figure 3 :

[0112] S41. Construct a road geometry model, which describes the spatial position of the vehicle on the road surface. The road geometry model can be a fixed road plane, a segmented road plane, a local road plane fitted by road control points, a lane-constrained plane, or a digital elevation model. In this embodiment, the NED coordinate system is used as an example. In the fixed road plane implementation, the road plane can be represented by its coordinates in the Down direction. express.

[0113] S42. Find the intersection between the vehicle observation ray and the road geometry model, and calculate the intersection ratio. Taking the NED coordinate system as an example, the calculation formula is shown in Equation 10 below:

[0114] (10)

[0115] In the formula: Indicates the first Frame Camera Center Down direction coordinates Represents rays Down direction component, This represents the intersection ratio of the ray from the camera center to the road plane.

[0116] S43. When the intersection ratio is positive and the ray direction meets the valid conditions, the vehicle's position observation in the real geographic coordinate system is obtained. If the ray is approximately parallel to the road surface, the intersection ratio is non-positive, the intersection point is located behind the camera, the intersection point exceeds the valid road area, or the road geometry model is missing, then the vehicle observation is marked as an invalid projection. The calculation formula for the vehicle's position observation in the real geographic coordinate system is shown in Equation 11 below:

[0117] (11)

[0118] In the formula: Indicates the first The first frame The location of each vehicle is observed in a unified real geographic coordinate system.

[0119] S44. Perform target association and trajectory maintenance in the real geographic coordinate system for valid vehicle position observations. Associate the valid real coordinate observations of vehicles in the current frame with the existing vehicle trajectories in the previous frame to update the vehicle trajectory set. The calculation formula for the updated trajectory set is shown in Equation 12 below:

[0120] (12)

[0121] In the formula: Indicates the first The frame contains all valid vehicle real coordinate observations. This indicates that the previous frame already contained a set of vehicle trajectories. This represents the updated set of vehicle trajectories. This represents the data association process. This association can be modeled using well-known techniques such as Euclidean distance and Mahalanobis distance.

[0122] S5. State Estimation and Output. Based on the data association results from S4, a posterior update step of the state-space model is performed to obtain the joint recursive estimation results of the vehicle's position and velocity. Abnormal velocities are suppressed through motion constraints. The output includes the vehicle's trajectory ID, true coordinate position, velocity magnitude, and velocity direction in the real geographic coordinate system. For details, please refer to the relevant documentation. Figure 3 :

[0123] S51. Establish a state vector for each observed vehicle under a unified real geographic coordinate system. Taking the NED coordinate system as an example, the state vector shall include at least the vehicle's northward position, eastward position, northward velocity vector component, and eastward velocity vector component in the real coordinate system. The calculation formula is shown in Equation 13 below:

[0124] (13)

[0125] In the formula: and Representing vehicle targets respectively In the The north and east positions of the frame, and These represent the northward velocity component and the eastward velocity component, respectively.

[0126] S52. A state-space model is used to jointly estimate the vehicle's position and velocity. The state-space model can be a Kalman filter model, an extended Kalman filter model, an unscented Kalman filter model, a particle filter model, a fixed-window smoothing model, or a trajectory optimization model. The vehicle motion state estimation result is obtained through recursive updates of vehicle state prediction and vehicle ground position observations, as shown in Equation 14 below.

[0127] (14)

[0128] In the formula: Represents the state transition matrix. Represents the observation matrix. This indicates the actual coordinates of the vehicle. Indicates process noise. This indicates observation noise.

[0129] This model allows for the synchronous estimation of vehicle position and velocity in the same state space.

[0130] S53. Calculate the vehicle speed based on the estimated vehicle motion state, and determine the vehicle speed direction from the speed components. The calculation formula is shown in Equation 15 below:

[0131] (15)

[0132] In the formula: Indicates vehicle target In the The frame speed magnitude and vehicle speed direction are determined by... and To be determined jointly.

[0133] S54. Perform abnormal velocity suppression on the estimated velocity results. The abnormal velocity suppression includes: combining the maximum reasonable velocity, the maximum reasonable acceleration and the observation residual, and performing abnormal marking, confidence reduction, delayed output or rejection processing on results with velocity jumps, exceeding the reasonable velocity range or excessive residuals.

[0134] S55. Output the vehicle's trajectory ID, actual coordinate position, speed magnitude, and speed direction in a unified real geographic coordinate system.

[0135] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] For example:

[0137] A drone aerial video clip captured from a ring road in a certain area was selected as a case study to verify the proposed method in engineering. The aerial video was shot from a frontal perspective, and the case data included the mobile drone aerial video, frame-by-frame aligned flight control data, and camera parameters. Using the method described in this invention, the vehicle image observation points are transformed to real coordinate observation points in a unified real geographic coordinate system. Furthermore, Mahalanobis distance is used for vehicle target tracking, and Kalman filtering is used to achieve joint estimation of vehicle position and velocity.

[0138] To verify the rationality of the true coordinate scale obtained after intersecting pixel rays with the road geometry model, this embodiment selects road lane width as the scale verification object. Specifically, 25 lane width segments are marked in 5 frames of aerial images, with the prior single lane width of the highway taken as 3.75 m. After being projected onto the true coordinate system using the method of this invention, the mean lane width is 3.96 m, the median is 3.94 m, the standard deviation is 0.14 m, the mean absolute error is 0.21 m, and the mean relative error is 5.73%. These results demonstrate that this invention, by using flight control data, camera parameters, and the road geometry model for true coordinate projection, can obtain a relatively reasonable ground scale representation. The results are as follows... Figure 4 As shown.

[0139] To further verify the stability of the velocity estimation results of this invention, this case study selected 10 representative trajectories from long car-type trajectories for analysis. The comparison methods included frame-by-frame differential velocity, sliding window velocity, and smoothing post-processing, as well as the Kalman posterior velocity directly output by the state-space model of this invention. Table 1 shows the stability and velocity change preservation capabilities of different velocity estimation methods:

[0140] Table 1 Comparison of different velocity estimation methods

[0141] method Average velocity change between adjacent frames (km / h / frame) First-order difference standard deviation (km / h / frame) Robust speed range (km / h) Relative frame-by-frame differential range retention rate Frame-by-frame difference 0.716 0.937 8.22 1.00 Kalman posterior 0.076 0.085 6.97 0.85 Sliding window 0.066 0.072 5.84 0.71

[0142] Here, the average adjacent frame velocity change represents the average magnitude of the velocity difference between two adjacent frames. The first-order difference standard deviation represents the dispersion of the velocity change; the smaller both values ​​are, the smoother the velocity curve. The robust velocity range represents the magnitude of the main velocity change after removing extreme outliers. The relative frame-by-frame difference range retention rate represents how much velocity change a certain method retains relative to the frame-by-frame difference velocity.

[0143] As shown in Table 1, the average adjacent frame velocity variation and first-order difference standard deviation of the frame-by-frame differential velocity are 0.716 km / h / frame and 0.937 km / h / frame, respectively, indicating significant inter-frame fluctuations. The corresponding Kalman posterior velocity indices decrease to 0.076 km / h / frame and 0.085 km / h / frame, respectively, demonstrating that the state-space model of this invention can significantly suppress velocity jitter. The sliding window method has even lower two stationarity indices, indicating stronger smoothing; however, its robust velocity range and range retention rate are only 5.84 km / h and 0.71, respectively, lower than the Kalman posterior velocity's 6.97 km / h and 0.85. Moving average methods are often used to smooth short-term fluctuations and highlight long-term trends, and therefore may also compress short-term velocity changes.

[0144] Therefore, this case demonstrates that even without using sliding window averaging as the final post-processing, this invention can still output relatively stable speed results. Furthermore, compared to the sliding window method, this invention retains more information about vehicle speed changes, making it more suitable for highway speed measurement scenarios that require a balance between stability and accurate speed changes.

[0145] Furthermore, such as Figure 5 The figure shows the speed variation curves of the vehicle under different speed estimation methods. Figure 5 It can be seen that the frame-by-frame differential velocity fluctuates significantly; the sliding window velocity curve is relatively smooth, but it compresses the velocity change amplitude to a certain extent; the Kalman posterior velocity output by the state space model of this invention suppresses velocity jitter while still retaining the non-zero change trend of vehicle velocity.

[0146] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for joint estimation of vehicle position and velocity based on aerial video captured by a mobile unmanned aerial vehicle (UAV), characterized in that, The steps include the following: S1. Acquire aerial video data, flight control log data and camera parameters of the mobile drone, synchronize the video frames of the aerial video with the flight control log in time and perform interpolation alignment to obtain the drone position, camera attitude and camera center coordinates corresponding to each video frame. S2. Perform image observation and extraction on road vehicles in video frames to obtain vehicle observation areas, and determine ground projection representative points from vehicle observation areas; S3. Based on the camera intrinsic parameters in the camera parameters, the ground projection representative points are converted into pixel rays in the camera coordinate system. Combined with the UAV flight control pose and the camera extrinsic parameters in the camera parameters, the pixel rays are converted to a unified real geographic coordinate system. S4. Under a unified real geographic coordinate system, the transformed pixel ray is intersected with the prior known road geometry model to obtain the vehicle's position observation under the unified real geographic coordinate system. The motion patterns of ground vehicles are modeled using a state-space model, and the motion state of the vehicle in the next frame is predicted based on the vehicle trajectory of the previous frame. Then, the vehicle observation results and vehicle motion state prediction results of the frame are correlated with targets and the trajectory is maintained in the real geographic coordinate system. S5. Based on the data association results of S4, perform the posterior update step of the state space model to obtain the joint recursive estimation results of vehicle position and speed, and suppress abnormal speeds through motion constraints. Output the vehicle's trajectory ID, real coordinate position, speed magnitude and speed direction in the real geographic coordinate system.

2. The method for joint estimation of vehicle position and speed based on aerial video from a mobile drone as described in claim 1, characterized in that, In step S1: The aerial video data includes consecutive video frames, frame number, frame rate, or frame timestamp. The flight control log data includes the UAV's attitude, altitude, flight speed, and log timestamps; The camera parameters include camera intrinsic parameters and camera extrinsic parameters.

3. The method for joint estimation of vehicle position and speed based on aerial video from a mobile UAV as described in claim 2, characterized in that, In step S1: For each video frame, the corresponding UAV position and flight control attitude angle are obtained by time alignment and interpolation based on the timestamp of that frame in the flight control log, as shown in the following formula: In the formula: Indicates the first The drone position corresponding to the frame. Indicates the flight control attitude angle. This represents flight control log data. Indicates the first Frame corresponds to timestamp. This indicates operations such as time synchronization, interpolation, or maintaining adjacent frames. Based on the synchronized UAV position, flight control attitude, and camera parameters, the camera center coordinates and camera attitude corresponding to this frame are calculated to form a frame-by-frame pose sequence. The calculation formula is shown below: In the formula: Indicates the first Frame image, Indicates the corresponding timestamp. Represents the camera intrinsic parameter matrix. Indicates the first The location of the frame camera center in a unified real geographic coordinate system. Indicates the first The extrinsic parameter matrix from the frame camera coordinate system to the unified real geographic coordinate system.

4. The method for joint estimation of vehicle position and speed based on aerial video from a mobile UAV as described in claim 1, characterized in that, The specific steps in step S2 are as follows: S21. Perform road vehicle image observation and extraction on each frame of aerial image to obtain vehicle observation results. The vehicle observation results include vehicle observation area, vehicle category and detection confidence. The vehicle observation area is a horizontal detection box, a rotated detection box, an instance segmentation mask or a set of key points. S22. Generate a set of candidate ground projection representative points based on the vehicle observation area; S23. Based on the camera's viewing angle, the shape of the vehicle's observation area, road area constraints, and the predicted position from historical trajectories, determine the final ground projection representative points from the set of candidate ground projection representative points.

5. The method for joint estimation of vehicle position and speed based on aerial video from a mobile UAV as described in claim 4, characterized in that: The vehicle observation area generates a set of candidate ground projection representative points, including: If the vehicle observation area is a horizontal detection box, the candidate points include the center point of the box, the midpoint of the bottom edge, or the midpoint of the side edge near the camera. If the vehicle observation area is a rotating detection frame, then the candidate points include the center point of the rotating frame or the midpoint of the edge; If the vehicle observation area is an instance segmentation mask, then the candidate points include the mask centroid, mask outline points, or approximate points of the vehicle grounding point.

6. The method for joint estimation of vehicle position and speed based on aerial video from a mobile drone as described in claim 1, characterized in that, Step S3 specifically includes the following steps: S31. Extract the pixel coordinates of the representative points of the ground projection; S32, Based on camera intrinsic parameter matrix The ground projection representative point is converted from pixel coordinates to normalized pixel rays in the camera coordinate system, and the calculation formula is as follows: In the formula: Indicates the first The first frame The ray in the camera coordinate system corresponding to each vehicle representative point. This represents the vector normalization operation; S33. Based on the camera pose corresponding to each frame, the pixel ray in the camera coordinate system is transformed to a unified local geographic coordinate system using the camera extrinsic parameters to obtain the vehicle observation ray in the real geographic coordinate system. The calculation formula is as follows: In the formula: This represents the vehicle observation ray in the true coordinate system. Indicates the first The extrinsic parameter matrix from the frame camera coordinate system to the unified real geographic coordinate system.

7. The method for joint estimation of vehicle position and speed based on aerial video from a mobile UAV as described in claim 1, characterized in that, Step S4 specifically includes the following steps: S41. Construct a road geometry model, wherein the road geometry model is a fixed road plane, a segmented road plane, a local road plane fitted by road control points, a lane constraint plane, or a digital elevation model; S42. Find the intersection between the vehicle observation ray and the road geometry model, and calculate the intersection ratio. The calculation formula is as follows: In the formula: Indicates the first Frame Camera Center Down direction coordinates Represents rays Down direction component, This represents the intersection ratio of the ray from the camera center to the road plane; S43. When the intersection ratio is positive and the ray direction satisfies the valid condition, the vehicle's position observation in a unified real geographic coordinate system is obtained; otherwise, the vehicle observation is marked as an invalid projection. The calculation formula for the vehicle's position observation in the real geographic coordinate system is as follows: In the formula: Indicates the first The first frame The position of each vehicle is observed in a unified real geographic coordinate system; S44. Perform target association and trajectory maintenance under a unified real geographic coordinate system for valid vehicle position observations. Associate the valid real coordinate observations of vehicles in the current frame with the existing vehicle trajectories in the previous frame, and update the vehicle trajectory set. The calculation formula is as follows: In the formula: Indicates the first The frame contains all valid real vehicle coordinate observations. This indicates that the previous frame already contained a set of vehicle trajectories. This represents the updated set of vehicle trajectories. This indicates the target association process.

8. The method for joint estimation of vehicle position and speed based on aerial video from a mobile UAV as described in claim 1, characterized in that, Step S5 specifically includes the following steps: S51. Establish a state vector for each observed vehicle under a unified real geographic coordinate system. The state vector includes at least the vehicle's northward position, eastward position, northward velocity vector component, and eastward velocity vector component in the real coordinate system. The calculation formula is shown below: In the formula: and Representing vehicle targets respectively In the The north and east positions of the frame, and These represent the northward velocity component and the eastward velocity component, respectively. S52. A state-space model is used to jointly estimate the vehicle's position and velocity. The vehicle motion state estimation result is obtained through vehicle state prediction and recursive updating of vehicle ground position observation. The calculation formula is shown in the following formula: In the formula: Represents the state transition matrix. Represents the observation matrix. This indicates the actual coordinates of the vehicle. Indicates process noise. Indicates observation noise; S53. Calculate the vehicle speed based on the vehicle motion state estimation results, and determine the vehicle speed direction from the speed components. The calculation formula is as follows: In the formula: Indicates vehicle target In the Frame speed; S54. Perform abnormal velocity suppression on the estimated velocity results. The abnormal velocity suppression includes: combining the maximum reasonable velocity, the maximum reasonable acceleration and the observation residual, and performing abnormal marking, confidence reduction, delayed output or rejection processing on results with velocity jumps, exceeding the reasonable velocity range or excessive residuals. S55 outputs the vehicle trajectory ID, actual coordinates, speed magnitude, and speed direction.

Images