An unmanned aerial vehicle dynamic target speed measurement method based on optical flow self-motion compensation

Abstract

The application discloses a kind of unmanned aerial vehicle dynamic target speed measurement method based on optical flow self-motion compensation, solve the problem that it is difficult to accurately measure dynamic target real speed under the motion state of unmanned aerial vehicle;Including: the dense optical flow field of continuous frame is calculated using RAFT optical flow model, the dynamic grid and fixed pixel coordinate system are established by global average estimation;Second, YOLOv8 is used for target detection and tracking, and the pixel speed of the target is obtained;The real pixel motion speed of the target relative to the stationary reference system is separated out by vector subtracting the unmanned aerial vehicle self-motion compensation from the target pixel speed;Finally, through the hybrid model combined with physics and machine learning, the compensated pixel speed is converted into the real physical speed of the target in combination with height and other parameters;The application fuses optical flow, dynamic grid and motion compensation, effectively eliminates self-motion interference, significantly improves dynamic speed measurement accuracy and reliability, and is suitable for traffic monitoring, sports analysis and security patrol and other fields.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and UAV application technology, specifically a dynamic target velocity measurement method for UAVs based on optical flow self-motion compensation. Background Technology

[0002] In recent years, with the rapid development of drone technology, its applications in traffic monitoring, smart city management, public safety, logistics and distribution, and sports event analysis have become increasingly widespread. In these application scenarios, real-time and accurate measurement of the speed of moving targets on the ground or in the air is a crucial and challenging task. Traditional speed measurement techniques, such as radar and laser speed measurement, while highly accurate, suffer from high equipment costs, complex deployment, and limited adaptability to complex environments. In contrast, computer vision-based speed measurement methods, with their low cost, non-invasiveness, and adaptability to complex scenes, have gradually become a research hotspot. In particular, motion estimation techniques based on optical flow can extract the speed information of target objects by analyzing the motion patterns of pixels in video sequences, providing an efficient solution for real-time traffic monitoring and drone remote sensing.

[0003] In recent years, deep learning-driven target detection and tracking technologies have significantly improved the robustness and accuracy of visual speed measurement systems. By combining models such as neural networks, the system can achieve precise localization and continuous tracking of vehicles or moving targets. However, existing methods still face challenges in complex lighting, occlusion, and highly dynamic scenes. Optical flow field analysis, due to its sensitivity to motion information and adaptability to scene changes, has become an important tool for solving these problems. Optical flow technology constructs a motion vector field by calculating the spatiotemporal changes in pixel intensity in an image sequence, thereby achieving accurate estimation of target velocity. Especially on UAV platforms, optical flow analysis combined with high-resolution image acquisition can achieve real-time velocity measurement of ground targets from a dynamic perspective.

[0004] However, optical flow-based velocimetry methods still face challenges in practical applications, including real-time performance issues in optical flow calculations, noise interference, and robustness in complex scenarios. Therefore, developing an efficient and accurate UAV dynamic target velocimetry method based on optical flow self-motion compensation can not only improve speed detection performance but also further promote the deep integration of computer vision and UAV technology in intelligent traffic monitoring. This invention addresses these challenges by proposing a novel method that effectively compensates for the self-motion of the observation platform, improves the accuracy of dynamic target velocimetry, and possesses high robustness and flexibility. Summary of the Invention

[0005] To address the challenge of accurately measuring the true velocity of dynamic targets on mobile platforms such as drones, this invention aims to provide a drone dynamic target velocity measurement method based on optical flow self-motion compensation. By utilizing the RAFT optical flow model to obtain a dense optical flow field across the entire image, and by constructing a fixed pixel coordinate system and a dynamic grid coordinate system, the calculated optical flow results are used to update the global optical flow of the grid system. This accurately estimates and outputs the self-motion vector representing the drone on the image plane, precisely eliminating the interference of the drone's own motion on the apparent motion of the target, thereby measuring the target's true and independent velocity.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A dynamic target velocity measurement method for UAVs based on optical flow self-motion compensation includes the following steps:

[0008] (1) Read the UAV video data frame by frame, use the RAFT model to calculate the dense optical flow field of the whole image of the continuous frames, and then calculate the average optical flow of all effective pixels to obtain the global optical flow vector representing the motion of the UAV itself. Based on the global optical flow vector of the UAV's own motion, establish a dynamic grid system and a fixed pixel coordinate system.

[0009] (2) Combine YOLOv8 with Kalman filter to perform real-time detection and tracking of target objects, and obtain the bounding box, ID, category and pixel coordinates of each object;

[0010] (3) Under the dynamic grid system and fixed pixel coordinate system, the apparent pixel velocity of the target object is calculated based on the pixel coordinates of the target object at the start and end of the grid line crossing and the number of frames of motion. The actual pixel velocity of the target object is obtained by removing the UAV’s own motion velocity from the apparent pixel velocity of the target object.

[0011] (4) When the actual pixel velocity of the target object is less than a small threshold, it is recorded as an instantaneous still frame. If the probability of an instantaneous still frame exceeds the threshold, the target object is determined to be in a stationary state; otherwise, it is in a moving state.

[0012] (5) Establish a mapping model that maps the optical flow velocity of the image to the real physical velocity, and convert the real pixel velocity of the moving target object into the real physical velocity.

[0013] In this invention, the RAFT model calculates the dense optical flow field across the entire image through the following steps:

[0014] ① Feature extraction: The model first extracts features from the two input frames, that is, it uses a feature encoder to generate feature maps:

[0015] ;

[0016] in, This represents a feature extraction network. I Indicates the input image;

[0017] ② Correlation Calculation: RAFT calculates the correlation between all pixel pairs, forming a 4D correlation volume for each pixel location. and Calculate the dot product of eigenvectors :

[0018] ;

[0019] in, Indicates the feature channel dimension. Represents the first frame image After feature extraction network In the obtained feature map, located in spatial position First The characteristic values ​​of each channel Represents the first frame image After feature extraction network In the obtained feature map, located in spatial position First The characteristic values ​​of each channel;

[0020] ③ Initialize optical flow: Initial optical flow field Set to zero:

[0021] ;

[0022] ④ Iterative Update: Iterative updates are performed using GRU units. In each iteration, the model estimates the optical flow based on the current optical flow. Relevance features are retrieved from the correlation volume, and the optical flow is updated by combining them with contextual features:

[0023] ;

[0024] in, Indicates the first The updated optical flow field after the next iteration. It is the amount of optical flow update predicted by the GPU unit;

[0025] ;

[0026] in, This indicates the correlation features found from the correlation volume based on the current optical flow estimate;

[0027] ⑤ Upsampling: The final optical flow field is restored to the original image resolution through upsampling. RAFT uses convex upsampling to improve the smoothness of the optical flow field.

[0028] ;

[0029] in, This represents the final optical flow field restored to its original resolution. This indicates convex upsampling, where k is the number of iterations.

[0030] In this invention, a dynamic grid system is established:

[0031] By using the dense optical flow field of the entire image calculated by RAFT, the mean optical flow of all effective pixels is calculated to obtain the global optical flow vector representing the motion of the UAV itself. Maintain a global grid offset. Each frame is updated cumulatively based on global optical flow:

[0032] ;

[0033] in, This represents the global grid offset at the current time t, used to record the cumulative distance (in pixels) that the entire grid has moved across the image plane. This represents the global grid offset at the previous time t-1. The global optical flow vector (u,v) at the current time t represents the overall image translation caused by the UAV's own motion.

[0034] Modular operations are used to implement conveyor belt-style grid line numbering, and the grid line drawing positions are determined by... OK, S represents the grid size. This indicates the reference position of the initial grid line, usually the starting coordinates of the grid (such as 0 or a specific offset).

[0035] In this invention, the calculation process of the actual pixel velocity of the target object is as follows:

[0036] Horizontal apparent pixel speed The calculation formula is:

[0037] ;

[0038] in, , These represent the horizontal pixel coordinates of the center point of the target object across the grid's starting and ending frames, respectively. , These represent the video frame sequence numbers that span the start and end frames of the grid, respectively, with the difference representing the time interval.

[0039] Vertical apparent pixel speed The calculation formula is:

[0040] ;

[0041] in, , These represent the vertical pixel coordinates of the target object at the center point of the starting and ending frames across the grid, respectively.

[0042] The pixel motion velocity of the UAV is estimated by using the global optical flow field, and the target velocity is compensated by the following calculation formula:

[0043] ;

[0044] ;

[0045] in, , These represent the actual pixel velocities of the target object in the horizontal and vertical directions, respectively, after motion compensation. , These represent the global average pixel motion speeds of the drone itself in the horizontal and vertical directions, respectively, calculated by the optical flow module.

[0046] Suppose that the pixel velocity of the target object is sampled and calculated n1 times, resulting in n1 sets of true pixel velocities of the target object in the horizontal and vertical directions, where the nth pixel velocity is... i The true pixel velocities of the target object in the horizontal and vertical directions are denoted as follows: and , ;but

[0047] ;

[0048] ;

[0049] in, and These represent the average true pixel velocity of the target object in the horizontal and vertical directions, respectively.

[0050] The total amplitude of the true pixel velocity of the target object is calculated using the Euclidean norm formula as follows:

[0051] ;

[0052] in, This represents the total amplitude of the actual pixel velocity of the target object relative to the ground.

[0053] In this invention, the formula for determining that the target object is in a stationary state is:

[0054] ;

[0055] in, It is the number of still frames. It is the total number of frames; by setting a minimum number of tracking frames, the total number of frames is limited to ensure the reliability and stability of the static judgment.

[0056] In this invention, a physical model or machine learning model is used to convert the actual pixel velocity of the target object into the actual physical velocity.

[0057] In this invention, when there is a lack of high-quality, well-labeled training data or limited computer resources, a physical model is used for calculation; when there is sufficient data and higher accuracy is desired, a machine learning model is used for calculation.

[0058] In this invention, the physical model is:

[0059] ①The physical model aims to establish a mapping relationship between the image plane and the real world ground. Using the principle of similar triangles, it combines all the parts related to the camera's internal parameters, as well as some environmental correction factors, into a single, configurable conversion coefficient.

[0060] The comprehensive conversion coefficient is calculated to improve the efficiency of subsequent forecasts. The calculation formula is as follows:

[0061] ;

[0062] in, In the physical principles Partial engineering simulation; This represents the fundamental factor determined based on the camera sensor size and the base focal length. Indicates the drone's flight altitude relative to the ground; This represents the calibration coefficient used for fine-tuning environmental errors;

[0063] The formula for converting pixel velocity into mapped real velocity, and finally calculating the real velocity, is as follows:

[0064] ;

[0065] in, This represents the calculated actual physical velocity of the target object; This represents the actual pixel velocity of the target object; This indicates the video's frame rate.

[0066] In this invention, a machine learning model is used to convert the real pixel velocity of a target object into its real physical velocity, including:

[0067] (5-1) Clean the UAV flight data, divide it into features and labels, and split it into training and test sets;

[0068] (5-2) Construct various models that map the relationship between optical flow velocity in the image and real physical velocity, including physical models, linear regression, random forest, gradient boosting tree, multilayer perceptron and XGboost;

[0069] (5-3) Train multiple models separately, evaluate the performance of different models by combining mean absolute error and coefficient of determination, systematically compare the performance of different models on the "UAV horizontal speed" regression task, and then select the optimal model.

[0070] In this invention, the formula for calculating the mean absolute error is:

[0071] ;

[0072] in, Indicates the total number of test samples; Indicates the first The true value of each sample; Indicates the first Predicted values ​​for each sample;

[0073] The formula for calculating the coefficient of determination is:

[0074] ;

[0075] in, Represents the sum of squares of regression; This represents the total sum of squares of deviations; This represents the mean of the test set labels.

[0076] Compared with the prior art, the beneficial effects of the present invention are:

[0077] 1. The purpose of this invention is to provide a dynamic target velocity measurement method for UAVs based on optical flow self-motion compensation, to solve the problem in existing technologies where it is difficult to accurately measure the true velocity of dynamic targets on mobile platforms such as UAVs, especially when the UAV itself is in motion, enabling precise separation and measurement of the target's true velocity. The method involves acquiring continuous video frames captured by the UAV and using the RAFT optical flow model to calculate the dense optical flow field across the entire image between adjacent frames. By performing a global analysis of this optical flow field, the average optical flow vector is calculated, and the self-motion vector representing the UAV on the image plane is accurately estimated and output.

[0078] 2. This invention achieves a visualized mathematical coordinate transformation in algorithmic logic by constructing a dynamic grid coordinate system. This grid system utilizes global optical flow for reverse compensation, essentially "extracting" the relatively static real ground background from the dynamically changing UAV imagery and constructing a stable mathematical reference frame. This mechanism transforms the abstract motion compensation algorithm into intuitive geometric coordinate operations (white-boxing): in this coordinate system, the target object's trajectory is no longer affected by the UAV's attitude; its velocity and direction are transformed into intuitive grid coordinate changes. This not only makes the process of "eliminating interference from the UAV's own motion" transparent and verifiable, but also provides a coordinate-based, intuitive, and precise mathematical calculation path for calculating the target's true motion vector. (This dynamic grid is actually an "intermediate layer": it transforms the original video stream mixed with UAV motion into a clear, ground-based mathematical coordinate system, turning the complex motion compensation algorithm into a visually visible and logically derivable geometric problem.)

[0079] 3. This invention converts optical flow information into a reliable estimate of the true physical velocity of a target object, providing crucial support for UAV monitoring, tracking, and measurement applications, and enabling more accurate vehicle speed estimation. It liberates UAVs from the limitations of traditional fixed observation platforms, allowing them to serve as flexible and efficient mobile speed measurement tools, greatly expanding their application scenarios. It can accurately eliminate the interference of the UAV's own motion on the apparent motion of the target, thereby measuring the target's true and independent velocity. Attached Figure Description

[0080] Figure 1 This is a flowchart of the method of the present invention.

[0081] Figure 2 This is a real-time motion trajectory diagram of the present invention.

[0082] Figure 3 This is the matrix heatmap of the present invention.

[0083] Figure 4 This is a comparison chart of the actual and predicted values ​​of the present invention.

[0084] Figure 5 This is a diagram illustrating the speed measurement effect of the drone according to the present invention. Detailed Implementation

[0085] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0086] The present invention proposes a dynamic target velocity measurement method for UAVs based on optical flow self-motion compensation, the flowchart of which is shown below. Figure 1 As shown, it includes the following steps:

[0087] (1) Combining optical flow with a dynamic mesh system enables robust motion analysis in dynamic environments. Optical flow provides motion information, while the dynamic mesh provides a structured and efficient way to represent and process this information. During UAV movement, this integration is used to detect and track moving objects while compensating for the UAV's own motion, ensuring accurate perception of the surrounding environment.

[0088] The observed optical flow, as the drone itself moves, includes both the drone's motion and the motion of objects in the scene. Optical flow describes the movement of pixels in an image between consecutive frames. Optical flow calculation is based on the following two fundamental assumptions:

[0089] (1-1) Constant Brightness Assumption: The brightness of a pixel remains constant between consecutive frames. That is:

[0090] ;

[0091] in, It is time t at position The pixel brightness is denoted by u, and the optical flow velocities in the horizontal and vertical directions are denoted by v.

[0092] (1-2) Small motion assumption: It is assumed that the movement of the object between consecutive frames is very small, which can be approximated by Taylor expansion:

[0093] ;

[0094] Based on the assumption of constant brightness, the optical flow constraint equation is obtained:

[0095] ;

[0096] Or abbreviated as:

[0097] ;

[0098] in, These are the gradients of the image in the x and y directions and over time, respectively.

[0099] The drone video data is read frame by frame, processed, and converted into grayscale images for optical flow calculation. Frame data is fed into an optical flow calculation queue using a multi-threaded approach, and the RAFT model is used to calculate inter-frame optical flow. The RAFT model is used to calculate optical flow to estimate the movement of pixels in an image between consecutive frames. The RAFT model calculates optical flow through the following steps:

[0100] (1-3) Feature Extraction: The model first extracts features from the two input frames. A feature encoder (typically a ResNet-based network) is used to generate feature maps.

[0101] ;

[0102] in, This represents a feature extraction network. I This represents the input image.

[0103] (1-4) Correlation Calculation: RAFT calculates the correlation between all pixel pairs, forming a 4D correlation volume. For each pixel location... and Calculate the dot product of eigenvectors :

[0104] ;

[0105] in, Indicates the feature channel dimension. Represents the first frame image After feature extraction network In the obtained feature map, located in spatial position First The characteristic values ​​of each channel Represents the first frame image After feature extraction network In the obtained feature map, located in spatial position First The characteristic values ​​of each channel.

[0106] (1-5) Initializing optical flow: Initial optical flow field Set to zero:

[0107] ;

[0108] (1-6) Iterative Update: Iterative updates are performed using GRU units. In each iteration, the model is updated based on the current optical flow estimate. Relevance features are retrieved from the correlation volume, and the optical flow is updated by combining them with contextual features:

[0109] ;

[0110] in, Indicates the first The updated optical flow field after the next iteration. It is the amount of optical flow update predicted by the GPU unit;

[0111] ;

[0112] in, This indicates the correlation features found from the correlation volume based on the current optical flow estimate.

[0113] (1-7) Upsampling: The final optical flow field is restored to the original image resolution through upsampling. RAFT uses convex upsampling to improve the smoothness of the optical flow field:

[0114] ;

[0115] in, This represents the final optical flow field restored to its original resolution. This indicates convex upsampling, where k is the number of iterations.

[0116] (1-8) Loss Function: During the training phase, RAFT uses supervised learning, calculating the loss by comparing the predicted optical flow with the actual optical flow. The loss function is typically a weighted combination of L-losses:

[0117] ;

[0118] in, K Indicates the total number of iterations. Indicates the first The loss weight of the next iteration ( (for attenuation factor) It is Ground Truth.

[0119] Optical flow velocity can be calculated by applying optical flow algorithms to consecutive video frames. Furthermore, optical flow information is also used to display average optical flow magnitude and performance statistics.

[0120] (1-9) Construct a dynamic mesh system based on video resolution and configuration parameters. The core of this system lies in using "white-box thinking" to transform complex motion compensation algorithms into a visualized mathematical coordinate system. The specific construction method is as follows: using the dense optical flow field of the entire image calculated by RAFT, calculate the average optical flow of all effective pixels to obtain the global optical flow vector representing the motion of the UAV itself. Maintain a global grid offset. Each frame is updated cumulatively based on global optical flow:

[0121] ;

[0122] in, This represents the global grid offset at the current time t, used to record the cumulative distance (in pixels) that the entire grid has moved across the image plane. This represents the global grid offset at the previous time t-1. The global optical flow vector (u,v) at the current time t represents the overall image translation caused by the UAV's own motion.

[0123] Modular arithmetic is used to implement conveyor belt-style grid line numbering. The grid lines within the field of view are numbered horizontally as follows: to Vertical numbering is to The grid lines are drawn at positions determined by... Confirm (S is the grid size). The reference position of the initial grid line is usually the starting coordinate of the grid (such as 0 or a specific offset). This not only achieves the effect of the grid moving with the ground background, but more importantly, it establishes a "line-crossing detection" mechanism: due to the high-frequency slight jitter of the drone, directly using frame-by-frame pixel difference will often cause drastic jumps in speed data. This invention establishes a pixel coordinate system with a fixed origin and records the grid number where the target center point is located. Only when the target object crosses the grid line (i.e., enters from one number interval to another) is a valid displacement calculation triggered. This "integral" line-crossing detection naturally filters out the high-frequency jitter of the drone and achieves robust speed measurement based on "white box thinking". The specific pixel coordinates and grid position of the target object's center point in each frame are recorded. For ease of understanding, an example is given here. When frame_number=710, the closest grid position of the vehicle with track_id=985 in the grid system is (V19, H23), and the pixel position is (955.679443, 13.568682). Over the next 5 frames, the closest grid position remained (V19, H23). At frame_number=716, the pixel position changed to (955.045166, 37.638729). At frame_number=717, its position changed to (V19, H0), and the pixel position became (954.897522, 44.174908), achieving the first boundary crossing. At frame_number=730, its position changed to (V19, H1), and the pixel position became (951.935059, 139.43277), achieving the second boundary crossing. At frame_number=743, its position changed to (V19, H2), and the pixel position became (950.812744, 233.952179), achieving the third boundary crossing. This allows for a direct observation of the target object's movement with the dynamic grid, showing the change in the object's pixel displacement compared to the drone's movement.

[0124] (2) Using YOLOv8 for real-time object detection and tracking, combined with a Kalman filter, a high-efficiency and accurate object tracking system can be achieved. YOLOv8 detects objects in the current frame and outputs bounding boxes, categories, and pixel coordinates. For each detected object, the Kalman filter is initialized or updated. In consecutive frames, the Kalman filter is used to predict the object position, and the detection results are correlated using IOU or other metrics. The bounding box, ID, category, and pixel coordinates of each object are output, while the trajectory is smoothed using a Kalman filter.

[0125] (3) Based on the real-time pixel position and the number of frames of motion of the target object, calculate the apparent pixel displacement of the target, and then calculate the apparent pixel velocity (unit: pixels / frame). First, under the dynamic mesh system and fixed pixel coordinate system, according to the pixel coordinates of the target object at the beginning and end of crossing the mesh and the number of frames of motion, calculate the apparent pixel velocity of the target object between the beginning and end frames of crossing the mesh, and then subtract the motion of the UAV (compensated by global optical flow velocity) to obtain the true pixel velocity of the target object.

[0126] Horizontal apparent pixel speed The calculation formula is:

[0127] ;

[0128] in, , These represent the horizontal pixel coordinates of the center point of the target object across the grid's starting and ending frames, respectively. , These represent the video frame sequence numbers that span the start and end frames of the grid, respectively, with the difference representing the time interval.

[0129] Vertical apparent pixel speed The calculation formula is:

[0130] ;

[0131] in, , These represent the vertical pixel coordinates of the target object at the center point of the starting and ending frames across the grid, respectively.

[0132] The pixel motion velocity of the UAV is estimated by using the global optical flow field, and the target velocity is compensated by the following calculation formula:

[0133] ;

[0134] ;

[0135] in, , These represent the actual pixel velocities of the target object in the horizontal and vertical directions after motion compensation (i.e., the independent velocities after eliminating drone motion interference). , These represent the global average pixel motion speeds of the drone itself in the horizontal and vertical directions, calculated by the optical flow module, respectively.

[0136] Suppose that the pixel velocity of the target object is sampled and calculated n1 times, resulting in n1 sets of true pixel velocities of the target object in the horizontal and vertical directions, where the nth pixel velocity is... i The true pixel velocities of the target object in the horizontal and vertical directions are denoted as follows: and , ;but

[0137] ;

[0138] ;

[0139] in, and These represent the average true pixel velocity of the target object in the horizontal and vertical directions, respectively.

[0140] The total amplitude of the true pixel velocity of the target object is calculated using the Euclidean norm formula as follows:

[0141] ;

[0142] in, This represents the total amplitude of the actual pixel velocity of the target object relative to the ground. This allows for a clear calculation of the real-time pixel velocity of the target object's actual motion. By comparing this velocity with the drone's own velocity, the independent motion of the target object relative to the drone can be clearly separated.

[0143] This allows for the clear calculation of the real-time pixel velocity of the target object's actual movement. By comparing this velocity with the optical flow velocity, it becomes clear how the target object moves relative to the drone.

[0144] (4) Target state filtering based on stationary probability. Due to the movement of the UAV, the target object will cause visual motion regardless of whether it is moving or stationary. To avoid incorrect speed calculations for illegally parked vehicles or stationary obstacles, this invention performs state classification based on stationary probability before physical speed mapping. The number of stationary frames and the total number of frames for each target object are recorded. If the true pixel velocity of a target in a certain frame is... Less than a tiny threshold (e.g., 0.5), in this invention, Numerically, it is related to the total amplitude of the actual pixel velocity. If the frames are identical, they are recorded as instantaneous still frames. The probability of stillness is calculated; when the probability exceeds a threshold (e.g., 0.8), the target object is considered stationary, and further velocity calculations are not performed. The calculation formula is as follows:

[0145] ;

[0146] in, It is the number of still frames. That is the total number of frames.

[0147] Trajectories are drawn in a fused space combining the superimposed pixel coordinate system and the dynamic mesh coordinate system, reflecting both the target's absolute position and its motion relative to the drone. An adjustable minimum number of tracking frames ensures the reliability and stability of stationary detection, while simultaneously observing the object's relative motion to the dynamic mesh in each frame and its trajectory within the minimum number of tracking frames. An adjustable minimum number of tracking frames is set to ensure the reliability and stability of stationary detection. The relative motion of the object with the dynamic mesh in each frame, and its trajectory within the minimum number of tracking frames, are observed.

[0148] Drawing real-time dynamic trajectory graphs, such as Figure 2 As shown, it can track objects and provide their ID, category, and speed. The white area represents moving objects with speed, including horizontal and vertical speeds; the gray area represents the speed of objects being calculated; and the red area represents objects that have been repeatedly identified as stationary within the minimum number of tracking frames. The system determines whether an object is stationary based on the probability of it being stationary.

[0149] The dynamic display shows the target's displacement path within the minimum number of tracking frames. The formula for calculating the trajectory variance is:

[0150] ;

[0151] Where μ represents the mean (or average displacement vector) of the target displacement within the minimum number of tracking frames. This strengthens the determination of stationary status. The threshold value is dynamically adjusted according to the scenario requirements (such as illegal parking detection in traffic monitoring and obstacle recognition in security monitoring). For illegal parking detection, a lower speed threshold is needed; otherwise, slowly moving vehicles will be mistakenly identified as stationary. Security monitoring, on the other hand, requires a higher probability threshold to avoid mistaking temporarily parked vehicles for obstacles.

[0152] To facilitate understanding, an example is provided here. When detecting illegal parking, the target vehicle is continuously tracked for 30 frames, and the pixel velocity of the vehicle in each frame is calculated using motion compensation. In this example, 25 out of the 30 frames meet the condition. According to (3a), the probability that the object is stationary is 0.8333. The preset probability threshold is 0.8 (i.e., the vehicle must be stationary for at least 80% of the frames within the detection period). Since the probability of the object being stationary is 0.8333, which is greater than 0.8, the judgment condition is met, and the system confirms that the vehicle is stationary.

[0153] (5) Establish a mapping model between "real pixel velocity" and "real physical velocity" to achieve accurate estimation of the real physical velocity of moving objects in the video. This step aims to estimate the real pixel velocity of the target object calculated in step (3), i.e., the pixel-level velocity after eliminating the interference of UAV self-motion. The optical flow is converted into real physical velocity (unit: meters per second). Although optical flow technology captures pixel-level motion information, the scale uncertainty problem must be solved when converting it into physical velocity. This requires taking into account key factors such as camera intrinsic parameters (e.g., focal length) and UAV extrinsic parameters (e.g., altitude, gimbal pitch angle).

[0154] Plot a matrix heatmap of the relationship between drone speed, drone altitude, and average optical flow, such as... Figure 3 As shown, it is clear that the speed and altitude of the drone significantly affect the average optical flow. Therefore, the data that needs to be considered includes optical flow velocity, camera intrinsic parameters, and drone extrinsic parameters, including the observed optical flow velocity, camera focal length, gimbal pitch angle, drone speed, and drone altitude. The camera focal length determines how the image is projected onto the pixels, while the drone's position (e.g., altitude) and attitude (e.g., gimbal pitch angle) affect the calculation of the viewpoint and ground distance. When the drone moves, its trajectory needs to be estimated in real time to separate the relative motion of the target object.

[0155] (5-1) Clean the UAV flight data, divide it into features and labels, split the training set and test set, ensure that 80% is used for training and 20% is used for final evaluation, and ensure that the results are reproducible by fixing the random seed to 42.

[0156] (5-2) Construct and compare various models, including the physical model, linear regression, random forest, gradient boosting tree, multilayer perceptron (variables), and XGboost. Based on the results of this document, the physical model and multilayer perceptron regression modeling will be the main focus.

[0157] ① Physical Model: The aim is to establish a mapping relationship between the image plane (the image taken by the drone) and the real world ground. Using the principle of similar triangles, all parts related to the camera's internal parameters (focal length, pixel size, etc.) and some environmental correction factors are merged into a single, configurable conversion coefficient.

[0158] The comprehensive conversion coefficient is calculated to improve the efficiency of subsequent forecasts. The calculation formula is as follows:

[0159] ;

[0160] in, In the physical principles Partial engineering simulation (comprehensive conversion coefficients); This represents the fundamental factor determined based on the camera sensor size and the base focal length. Indicates the drone's flight altitude relative to the ground; This represents the calibration coefficient used to fine-tune environmental errors.

[0161] The formula for converting pixel velocity into mapped real velocity, and finally calculating the real velocity, is as follows:

[0162] ;

[0163] in, This represents the calculated actual physical velocity of the target object (unit: m / s). In this invention, the actual pixel velocity of the target object is represented by pixels per frame. Numerically, it is related to the total amplitude of the actual pixel velocity. same; This indicates the video's frame rate (FPS).

[0164] ② Taking a three-layer multilayer perceptron regression model as an example, d→128→64→1.

[0165] Standardize and output via forward propagation (three-layer ReLU network). (That is, the actual physical velocity predicted by the model), the process is as follows:

[0166] ;

[0167] ;

[0168] ;

[0169] ;

[0170] in, This represents the standardized input feature vector (including pixel velocity, height, etc.). , , These represent the weight matrices of the first, second, and third layers of the network, respectively. , , These represent the corresponding bias vectors; Indicates the first The activation output value of the layer; , These represent the linear weighted sums of the first and second layers of the network, respectively. This represents the predicted actual physical velocity in the final output.

[0171] The loss function, using mean squared error, is calculated using the following formula:

[0172] ;

[0173] in, Indicates the number of samples in the training batch; Indicates the first The predicted true physical velocity for each sample; Indicates the first The true speed label of each sample.

[0174] Backpropagation is used to write down the gradient for each single sample, and the average is taken during batch training.

[0175] ;

[0176] in, Element-wise multiplication; It is the gradient of ReLU (1 if the element is greater than 0, and 0 otherwise). , , These represent the error terms of the first, second, and third layers of the network, respectively. y This represents the true velocity value (scalar), corresponding to the label of a single sample.

[0177] Parameter updates and Adam optimizations:

[0178] ;

[0179] Where W represents the parameters to be trained in the network (including all weight matrices and bias vectors). Indicates the learning rate; and These represent the first-order moment estimate and the second-order moment estimate after gradient bias correction, respectively; This is to prevent tiny constants with a denominator of zero.

[0180] (5-3) Train the model and evaluate the performance of different models by combining the mean absolute error (MAE) and the coefficient of determination (R²). This allows for a systematic comparison of the performance of different algorithms on the "UAV horizontal speed" regression task, and the selection of the optimal model.

[0181] The formula for calculating the mean absolute error is:

[0182] ;

[0183] in, Indicates the total number of test samples; Indicates the first The true value of each sample; Indicates the first The predicted value for each sample.

[0184] The formula for calculating the coefficient of determination is:

[0185] ;

[0186] in, This represents the sum of squared residuals. This represents the total sum of squares. This represents the mean of the test set labels.

[0187] (5-4) Evaluate the results of different models. In cases where high-quality, well-labeled training data is lacking or computer resources are limited, use a physical model for calculation. When data is abundant and higher accuracy is desired, using a machine learning model is superior. Among them, the multilayer perceptron regression model has the smallest mean absolute error. Therefore, this multilayer perceptron neural network model is selected as the optimal model. The optimal model is the mapping model. The mapping model is used to predict the true velocity of objects. Figure 4 The results show that the predicted and actual values ​​are very close, indicating that the prediction results are highly accurate.

[0188] By combining physical models with neural networks, the accuracy of velocity estimation in complex scenarios is significantly improved. This systematic approach... Figure 5 This technology effectively transforms optical flow information from images into a reliable estimate of the target object's true physical velocity, providing crucial support for UAV monitoring, tracking, and measurement applications, and enabling more accurate vehicle speed estimation. It liberates UAVs from the limitations of traditional fixed observation platforms, allowing them to function as flexible and efficient mobile speed measurement tools, greatly expanding their application scenarios. Furthermore, it accurately eliminates the interference of the UAV's own motion on the target's apparent motion, thereby measuring the target's true and independent velocity.

[0189] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for dynamic target velocity measurement of unmanned aerial vehicles based on optical flow self-motion compensation, characterized in that, Includes the following steps: (1) Read the UAV video data frame by frame, use the RAFT model to calculate the dense optical flow field of the whole image of the continuous frames, and then calculate the average optical flow of all effective pixels to obtain the global optical flow vector representing the motion of the UAV itself. Based on the global optical flow vector of the UAV's own motion, establish a dynamic grid system and a fixed pixel coordinate system. (2) Combine YOLOv8 with Kalman filter to perform real-time detection and tracking of target objects, and obtain the bounding box, ID, category and pixel coordinates of each object; (3) Under the dynamic grid system and fixed pixel coordinate system, the apparent pixel velocity of the target object is calculated based on the pixel coordinates of the target object at the start and end of the grid line crossing and the number of frames of motion. The actual pixel velocity of the target object is obtained by removing the UAV’s own motion velocity from the apparent pixel velocity of the target object. (4) When the actual pixel velocity of the target object in a frame is less than a small threshold, it is recorded as an instantaneous still frame. If the probability of an instantaneous still frame exceeds the threshold, the target object is determined to be in a stationary state; otherwise, it is in a moving state. (5) Establish a mapping model that maps the optical flow velocity of the image to the real physical velocity, and convert the real pixel velocity of the moving target object into the real physical velocity. Establishing a dynamic grid system: By using the dense optical flow field of the entire image calculated by RAFT, the mean optical flow of all effective pixels is calculated to obtain the global optical flow vector representing the motion of the UAV itself. Maintain a global grid offset. Each frame is updated cumulatively based on global optical flow: ； in, This represents the global grid offset at the current time t, used to record the cumulative distance the entire grid has moved on the image plane. This represents the global grid offset at the previous time t-1. The global optical flow vector (u,v) at the current time t represents the overall image translation caused by the UAV's own motion; Modular operations are used to implement conveyor belt-style grid line numbering, and the grid line drawing positions are determined by... OK, S represents the grid size. This indicates the reference position of the initial grid lines.

2. The UAV dynamic target velocity measurement method based on optical flow self-motion compensation according to claim 1, characterized in that, The RAFT model calculates the dense optical flow field across the entire image using the following steps: ① Feature extraction: The model first extracts features from the two input frames, that is, it uses a feature encoder to generate feature maps: ； in, This represents a feature extraction network. I Indicates the input image; ② Correlation Calculation: RAFT calculates the correlation between all pixel pairs, forming a 4D correlation volume for each pixel location. and Calculate the dot product of eigenvectors : ； in, Indicates the feature channel dimension. Represents the first frame image After feature extraction network In the obtained feature map, located in spatial position First The characteristic values ​​of each channel Represents the first frame image After feature extraction network In the obtained feature map, located in spatial position First The characteristic values ​​of each channel; ③ Initialize optical flow: Initial optical flow field Set to zero: ； ④ Iterative Update: Iterative updates are performed using GRU units. In each iteration, the model estimates the optical flow based on the current optical flow. Relevance features are retrieved from the correlation volume, and the optical flow is updated by combining them with contextual features: ； in, Indicates the first The updated optical flow field after the next iteration. It is the amount of optical flow update predicted by the GPU unit; ； in, This indicates the correlation features found from the correlation volume based on the current optical flow estimate; ⑤ Upsampling: The final optical flow field is restored to the original image resolution through upsampling. RAFT uses convex upsampling to improve the smoothness of the optical flow field. ； in, This represents the final optical flow field restored to its original resolution. This indicates convex upsampling, where k is the number of iterations.

3. The UAV dynamic target velocity measurement method based on optical flow self-motion compensation according to claim 1, characterized in that, The calculation process for the true pixel velocity of the target object is as follows: Horizontal apparent pixel speed The calculation formula is: ； in, These represent the horizontal pixel coordinates of the center point of the target object across the grid's starting and ending frames, respectively. These represent the video frame sequence numbers that span the start and end frames of the grid, respectively, with the difference representing the time interval. Vertical apparent pixel speed The calculation formula is: ； in, These represent the vertical pixel coordinates of the target object at the center point of the starting and ending frames across the grid, respectively. The pixel motion velocity of the UAV is estimated by using the global optical flow field, and the target velocity is compensated by the following calculation formula: ； ； in, These represent the actual pixel velocities of the target object in the horizontal and vertical directions, respectively, after motion compensation. These represent the global average pixel motion speeds of the drone itself in the horizontal and vertical directions, respectively, calculated by the optical flow module. Suppose that the pixel velocity of the target object is sampled and calculated n1 times, resulting in n1 sets of true pixel velocities of the target object in the horizontal and vertical directions, where the nth pixel velocity is... i The true pixel velocities of the target object in the horizontal and vertical directions are denoted as follows: and ;but ； ； in, and These represent the average true pixel velocity of the target object in the horizontal and vertical directions, respectively. The total amplitude of the true pixel velocity of the target object is calculated using the Euclidean norm formula as follows: ； in, This represents the total amplitude of the actual pixel velocity of the target object relative to the ground.

4. The UAV dynamic target velocity measurement method based on optical flow self-motion compensation according to claim 1, characterized in that, Formula for determining if a target object is at rest: ； in, It is the number of still frames. It is the total number of frames; by setting a minimum number of tracking frames, the total number of frames is limited to ensure the reliability and stability of the static judgment.

5. The UAV dynamic target velocity measurement method based on optical flow self-motion compensation according to claim 1, characterized in that, Use a physics model or machine learning model to convert the true pixel velocity of the target object into the true physical velocity.

6. The UAV dynamic target velocity measurement method based on optical flow self-motion compensation according to claim 5, characterized in that, When high-quality, well-labeled training data is lacking or computing resources are limited, physical models are used for computation; when there is sufficient data and higher accuracy is desired, machine learning models are used for computation.

7. The UAV dynamic target velocity measurement method based on optical flow self-motion compensation according to claim 5, characterized in that, The physical model: ①The physical model aims to establish a mapping relationship between the image plane and the real world ground. Using the principle of similar triangles, it combines all the parts related to the camera's internal parameters, as well as some environmental correction factors, into a single, configurable conversion coefficient. The comprehensive conversion coefficient is calculated to improve the efficiency of subsequent forecasts. The calculation formula is as follows: ； in, In the physical principles Partial engineering simulation; This represents the fundamental factor determined based on the camera sensor size and the base focal length. Indicates the drone's flight altitude relative to the ground; This represents the calibration coefficient used for fine-tuning environmental errors; The formula for converting pixel velocity into mapped real velocity, and finally calculating the real velocity, is as follows: ； in, This represents the calculated actual physical velocity of the target object; This represents the actual pixel velocity of the target object; This indicates the video's frame rate.

8. The UAV dynamic target velocity measurement method based on optical flow self-motion compensation according to claim 5, characterized in that, Using machine learning models to convert the true pixel velocity of a target object into its true physical velocity includes: (5-1) Clean the UAV flight data, divide it into features and labels, and split it into training and test sets; (5-2) Construct various models that map the relationship between optical flow velocity in the image and real physical velocity, including physical models, linear regression, random forest, gradient boosting tree, multilayer perceptron and XGboost; (5-3) Train multiple models separately, evaluate the performance of different models by combining mean absolute error and coefficient of determination, systematically compare the performance of different models on the "UAV horizontal speed" regression task, and then select the optimal model.

9. The UAV dynamic target velocity measurement method based on optical flow self-motion compensation according to claim 8, characterized in that, The formula for calculating the mean absolute error is: ； in, Indicates the total number of test samples; Indicates the first The true value of each sample; Indicates the first Predicted values ​​for each sample; The formula for calculating the coefficient of determination is: ； in, Represents the sum of squares of regression; This represents the total sum of squares of deviations; This represents the mean of the test set labels.

Images