Vehicle speed measurement method during unmanned aerial vehicle highway inspection based on unmanned aerial vehicle nest

Abstract

The invention relates to the technical field of vehicle speed measurement, and particularly discloses a vehicle speed measurement method during unmanned aerial vehicle highway inspection based on an unmanned aerial vehicle nest, and the method comprises the steps: enabling the unmanned aerial vehicle nest to receive a task instruction, carrying out the highway inspection based on an inspection path after the unmanned aerial vehicle takes off, carrying out the multi-mode road surface intelligent segmentation, and outputting a binary road surface mask, time-space cooperative vehicle tracking is carried out, a vehicle trajectory data set with a unique ID is obtained, unmanned aerial vehicle RTK positioning data and ground marker coordinates are obtained, a pixel-physical space mapping table is output, millimeter wave radar altimeter data is collected for vehicle speed measurement, and a vehicle speed vector with a timestamp is output. The problems of vehicle missing detection and speed measurement blind areas caused by a traditional speed measurement method are solved, and the problem that efficient and accurate data support cannot be provided for highway management is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle speed measurement, and in particular to a vehicle speed measurement method during drone highway inspection based on a drone nest. Background Art

[0002] With economic development, the number of vehicles on highways continues to increase, and traffic flows are becoming increasingly dense. This makes it difficult to meet the demand for real-time, accurate vehicle speed information, making it difficult to promptly detect and address speeding violations. Modern highway management places increasingly high demands on the accuracy and real-time nature of traffic data. In recent years, drone technology has rapidly developed, significantly improving flight stability, endurance, and the performance of onboard equipment. Drones can operate in diverse environments and weather conditions, flexibly reaching various locations on highways for inspection and speed measurement, providing a new technical means for speed measurement on highways. Currently, intelligent traffic management is a key development direction in the transportation sector. Drone highway inspection and vehicle speed measurement methods are a key component of intelligent traffic management systems. They can seamlessly integrate and share data with other traffic management systems, enabling comprehensive, intelligent management of highway traffic, improving the efficiency and quality of traffic management, and providing a safer and more convenient travel environment for the public.

[0003] At present, there are still some deficiencies in the research on vehicle speed measurement, which are specifically reflected in:

[0004] Fixed cameras take photos and measure speed at fixed points: The location is fixed, and there are monitoring blind spots. Drivers can easily evade speed measurement by slowing down near the camera and speeding in other areas. It is impossible to fully and accurately monitor the speed of vehicles on the highway.

[0005] Interval speed measurement: It can only obtain the average vehicle speed within a certain interval, and cannot timely reflect the instantaneous speed of the vehicle at each moment. It is difficult to effectively monitor vehicles that have instantaneous speeding behavior within the interval, and many traffic accidents are often caused by instantaneous speeding.

[0006] Due to its limitations, traditional speed measurement methods cannot provide efficient and accurate data support for highway management. Summary of the Invention

[0007] In response to the shortcomings of the existing technology, the present invention provides a vehicle speed measurement method during drone highway inspection based on a drone nest, which can effectively solve the problems involved in the above-mentioned background technology.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: a vehicle speed measurement method during drone highway inspection based on a drone nest, comprising the following steps: the drone nest receives a task instruction, and the drone takes off to conduct a highway inspection based on the inspection path; the drone original video stream is collected, a high-precision vector map of the highway is obtained, multimodal road surface intelligent segmentation is performed, and a binary road surface mask is output; the vehicle detection frame and the drone IMU posture data in the drone original video stream are collected, and combined with the binary road surface mask, spatiotemporal collaborative vehicle tracking is performed to obtain a vehicle trajectory dataset with a unique ID; the drone RTK positioning data and ground marker coordinates are obtained, and a pixel-physical space mapping table is output in combination with the vehicle detection frame in the drone original video stream; based on the vehicle trajectory dataset with a unique ID and the pixel-physical space mapping table, the millimeter wave radar altimeter data is collected to measure the vehicle speed, and a vehicle speed vector with a timestamp is output.

[0009] As a further method, the drone nest receives the mission instruction, and the drone takes off and conducts highway inspection based on the inspection path. The specific analysis process is as follows: the ground control station sends the mission instruction to the drone nest through the satellite communication network, and the drone nest receives the mission instruction; the drone nest checks the drone's battery level, opens the nest door, and starts the navigation system and communication system; when the drone nest checks that the drone's battery level reaches the drone's battery level threshold stored in the database, and successfully opens the nest door, and starts the navigation system and communication system, a take-off instruction is issued to the drone, and the drone takes off from the nest under the action of its own flight control system; the drone relies on its own navigation system and the navigation information provided by the ground control station to conduct highway inspections according to the inspection path preset in the mission instruction.

[0010] As a further method, the original video stream of the drone is collected, and a high-precision vector map of the highway is obtained. Multimodal road surface intelligent segmentation is performed and a binary road surface mask is output. The specific analysis process is as follows: the original video stream of the drone is collected, including dual channels of visible light images and infrared images; a high-precision vector map of the highway is obtained, including GPS coordinates of lane lines and guardrails; the infrared image of the original video stream of the drone is input into the trained convolutional neural network model, and the probability of each pixel in the infrared image belonging to a different material is output; pixels belonging to non-asphalt areas are filtered according to the probability output by the convolutional neural network model: when the convolutional neural network model predicts that the probability of a pixel belonging to asphalt is greater than the asphalt probability threshold stored in the database, the pixel is marked as an asphalt area and retained as a road surface pixel; when the convolutional neural network model predicts that the probability of a pixel belonging to asphalt is not greater than the asphalt probability threshold stored in the database, the pixel is marked as a non-asphalt area, and the pixels belonging to the non-asphalt area are filtered and not retained;

[0011] Using the positioning data and attitude data of the UAV, combined with the coordinate information of the high-precision vector map, the lane line coordinates of the vector map are projected into the image space through the perspective transformation matrix: the positioning data of the UAV is specifically the GPS coordinates, and the attitude data of the UAV is specifically the pitch angle, yaw angle, and roll angle; based on the GPS coordinates of the UAV and the coordinate system of the vector map, the position of the vector map in the UAV coordinate system is calculated; the focal length, principal point coordinates, rotation matrix, and translation vector of the UAV are collected to construct the perspective transformation matrix, and the projection position of the vector map in the image space is calculated based on the attitude data of the UAV; the lane line coordinates of the vector map are mapped into the image space through the perspective transformation matrix to obtain the pixel coordinates of the lane line on the image; a reference coordinate system is established with the center line of the projected lane line as the x-axis of the reference coordinate system and the normal direction of the lane line as the y-axis;

[0012] Dynamically detect accumulated water and reflective areas and repair image anomalies; binarize the filtered asphalt area and the repaired image, set the pixel value of the asphalt area to 1, and the pixel value of the non-asphalt area to 0, to obtain a binary mask image, and output a binary road surface mask; if new obstacles or construction areas are found in the binary road surface mask, and these areas are inconsistent with the information in the vector map, a road topology change warning is triggered, and the warning information includes the coordinates and area of ​​the changed area.

[0013] As a further method, water accumulation and reflective areas are dynamically detected and image anomalies are repaired. The specific analysis process is as follows: edge detection is performed on the visible light image of the original video stream of the drone, edge information in the image is extracted, and edge detection and region segmentation algorithms in computer vision are used to detect water accumulation and reflective areas; the detected water accumulation and reflective areas are repaired using a trained generative adversarial network; the detected water accumulation and reflective areas are input into the trained generative adversarial network, and the generator outputs the repaired normal road surface image.

[0014] As a further approach, we collected vehicle detection frames from the drone's original video stream and the drone's IMU pose data. Combined with the binary road mask, we performed spatiotemporal collaborative vehicle tracking, generating a vehicle trajectory dataset with unique IDs. The specific analysis process involved running the trained YOLO-X model within the constraints of the binary road mask, detecting vehicles in the asphalt area within the binary road mask. Only when a detection frame was completely within the asphalt area of ​​the binary road mask was it considered a valid vehicle detection result. Furthermore, we combined the IMU data to compensate for target offsets caused by drone motion.

[0015] Short-term trajectory: Based on the Kalman prediction of three adjacent frames, the Kalman filter is used to predict and correct the vehicle's short-term trajectory: the state vector of the Kalman filter is initialized, including the vehicle's position and velocity; the vehicle's position in the next frame is predicted based on the vehicle's motion model; the prediction result is corrected using the detection box information of the three adjacent frames, and the state vector of the Kalman filter is updated based on the position and velocity information of the detection box, thereby obtaining a more accurate short-term trajectory prediction result;

[0016] Mid-time trajectory: trajectory smoothing optimization within a sliding window. A sliding window algorithm is used to smooth and optimize the vehicle's mid-time trajectory. The sliding window is as large as 10 frames. The vehicle's trajectory is fitted within the sliding window and smoothed using the Bezier curve fitting method. Based on the fitting results, the vehicle's trajectory is corrected and the fitted trajectory is used as the vehicle's true trajectory. The jitter and noise in the original trajectory are smoothed to obtain a more stable mid-time trajectory.

[0017] Long-term trajectory, cross-camera ID matching, in the case of multi-machine collaboration, the long-term trajectory of the vehicle ID matching: collect vehicle detection frame information captured by multiple drones, including the vehicle's position, speed, and appearance features; obtain the vehicle's color, texture, speed, and acceleration, and calculate the similarity between vehicles captured by different drones; based on the similarity, the vehicle ID is matched. If the similarity of vehicles captured by two drones exceeds the similarity threshold stored in the database, they are considered to be the same vehicle, achieving cross-camera ID matching and obtaining the long-term trajectory of the vehicle;

[0018] The short-term, medium-term, and long-term trajectory information of vehicles are integrated into a dataset, and a unique ID is assigned to each vehicle. Each row in the dataset represents a vehicle trajectory point, including the vehicle's unique ID, frame number, pixel coordinates (x, y), movement direction, and timestamp, resulting in a vehicle trajectory dataset with a unique ID. Abnormal trajectories are marked: if the vehicle's movement direction is opposite to the normal driving direction of the lane, it is marked as wrong-way; if the vehicle crosses multiple lane lines in a short period of time, it is marked as a sudden lane change. Abnormal trajectory marking information includes the vehicle's unique ID, abnormality type, occurrence time, and occurrence location.

[0019] As a further method, IMU data is combined to compensate for the target offset caused by the shaking of the drone. The specific analysis process is as follows: the IMU data of the drone is used to compensate for the target offset caused by the shaking of the drone. The IMU data of the drone includes the data of the drone accelerometer, drone gyroscope, and drone magnetometer; the real-time attitude changes of the drone are calculated based on the IMU data, including changes in pitch angle, yaw angle, and roll angle; based on the attitude changes, the offset pixel value of the target on the image plane is calculated; and the vehicle detection frame is compensated for the offset accordingly so that the detection frame can accurately track the target vehicle.

[0020] As a further method, the RTK positioning data of the UAV and the coordinates of the ground markers are obtained, and the pixel-physical space mapping table is output in combination with the vehicle detection frame in the original video stream of the UAV. The specific analysis process is as follows: the benchmark scale is established by using the reflective markers placed; the perspective transformation matrix is ​​dynamically adjusted according to the RTK positioning data and attitude data of the UAV. The UAV attitude data includes the UAV height, UAV pitch angle, and UAV yaw angle; the real-time position information of the UAV is obtained according to the RTK positioning data of the UAV; the rotation matrix and translation vector of the UAV are calculated according to the attitude data of the UAV; the perspective transformation matrix is ​​constructed by combining the focal length and principal point coordinates of the UAV. When the drone's altitude, pitch angle, and yaw angle change, the rotation matrix and translation vector are updated in real time, and the perspective transformation matrix is ​​dynamically adjusted. The lane line curvature data in the high-precision vector map of the highway is used to reversely infer the lens distortion parameters: the true curvature of the lane line is calculated based on the lane line curvature information in the vector map; the edge information of the lane line is extracted from the drone's visible light image, and the curvature of the lane line in the visible light image is calculated; the lens distortion parameters are reversed by comparing the true curvature of the lane line with the curvature in the visible light image; the pixel-physical space mapping table is updated in real time based on the dynamically adjusted perspective transformation matrix and the reversed lens distortion parameters.

[0021] As a further method, a benchmark scale is established using placed reflective markers. The specific analysis process is as follows: obtaining the spacing of the placed reflective markers stored in the database; using the target detection algorithm to identify the reflective markers in the visible light image taken by the drone, and calculating the pixel distance between adjacent reflective markers in the visible light image; and calculating the pixel scale factor of the visible light image based on the spacing of the placed reflective markers stored in the database and the pixel distance between adjacent reflective markers in the visible light image.

[0022] As a further approach, based on a vehicle trajectory dataset with a unique ID and a pixel-to-physical space mapping table, millimeter-wave radar altimeter data is collected to measure vehicle speed and output a timestamp vehicle speed vector. The specific analysis process is as follows: The five consecutive frames with the largest displacement of the trajectory points are selected from the vehicle trajectory dataset with a unique ID; the pixel displacement of the trajectory points is converted to actual physical displacement according to the pixel-to-physical mapping table; the time difference between these five frames is calculated based on the timestamp to obtain the vehicle speed, which is recorded as the basic speed measurement result;

[0023] When a speeding vehicle is detected, nearby drones are dispatched to synchronously collect trajectory data for the vehicle: The location of the neighboring drones is determined based on the drone's RTK positioning data; synchronization collection instructions are sent to the neighboring drones, causing them to simultaneously collect trajectory data for the speeding vehicle; the trajectory data collected by the neighboring drones is collected; and the trajectory data collected by multiple drones is processed using triangulation to eliminate perspective errors: a triangulation model is constructed based on the position and attitude data of the neighboring drones; and the triangulation model is used to fuse the trajectory data collected by different drones to calculate the vehicle's three-dimensional position and speed. The radar altitude data is corrected for scale changes to obtain a corrected speed measurement result.

[0024] The optical flow method is used to analyze the number of rotations of the vehicle tires to assist in verifying the vehicle speed: the optical flow method is used to process the image sequence of the vehicle tires to calculate the rotation speed of the tires; the vehicle speed is calculated based on the tire rotation speed and the vehicle's wheel diameter; the vehicle speed calculated by the optical flow method is compared with the corrected speed measurement result. If the difference between the vehicle speed calculated by the optical flow method and the corrected speed measurement result is greater than the vehicle speed difference threshold stored in the database, the corrected speed measurement result is updated to the vehicle speed calculated by the optical flow method to obtain an updated speed measurement result; based on the updated speed measurement result, the vehicle's longitudinal speed component and latitudinal speed component are obtained; the vehicle's speed information is integrated into a data structure, which includes the vehicle's unique ID, longitudinal speed component, latitudinal speed component, and timestamp, and outputs a vehicle speed vector with a timestamp.

[0025] As a further method, radar altitude data is used to correct scale changes. The specific analysis process is as follows: millimeter-wave radar altimeter data is used to correct the scale changes caused by the drone's ascent and descent: the drone's flight altitude is obtained in real time based on the millimeter-wave radar altimeter data; the scale factor in the pixel-to-physical mapping table is adjusted according to the drone's altitude change; the corrected scale factor is used to recalculate the vehicle's speed and record it as the corrected speed measurement result.

[0026] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:

[0027] The present application provides a vehicle speed measurement method for unmanned aerial vehicle highway inspection based on a drone nest, which realizes accurate mapping from pixel-level image analysis to physical space speed calculation through the cooperation of original video stream of the unmanned aerial vehicle, high-precision vector map, IMU attitude data, RTK positioning data and millimeter wave radar altimeter data. The vehicle is uniquely identified and a trajectory data set is generated, which can continuously track the vehicle's motion state in time and space dimensions, solving the vehicle missing detection and speed measurement blind area problems caused by the limited view angle of traditional fixed cameras. The millimeter wave radar altimeter data can assist in correcting the influence of the change of the flight height of the unmanned aerial vehicle on the speed measurement, and the IMU attitude data compensates for the errors caused by the pitching and yawing of the unmanned aerial vehicle. The unmanned aerial vehicle can take off from the nest autonomously, flexibly adjust the inspection range based on the preset path or real-time instructions, adapt to different road sections, and have stronger technical iteration flexibility. BRIEF DESCRIPTION OF DRAWINGS

[0028] The present application will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present application. For ordinary skilled persons in the art, other drawings can be obtained without creative labor on the basis of the following drawings.

[0029] Figure 1 The present application provides a vehicle speed measurement method for unmanned aerial vehicle highway inspection based on a drone nest. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0031] Referring to Figure 1 The present application provides a vehicle speed measurement method for unmanned aerial vehicle highway inspection based on a drone nest, which includes: the drone nest receives task instructions, and the unmanned aerial vehicle takes off and performs highway inspection based on the inspection path.

[0032] The specific analysis process is as follows: the ground control station sends task instructions to the drone nest through a satellite communication network, and the drone nest receives the task instructions; the drone nest checks the power of the unmanned aerial vehicle, opens the nest cabin door, and starts the navigation system and communication system; when the drone nest checks that the power of the unmanned aerial vehicle reaches the power threshold stored in the database, and successfully opens the nest cabin door and starts the navigation system and communication system, the take-off instruction is sent to the unmanned aerial vehicle, and the unmanned aerial vehicle takes off from the nest under the action of its own flight control system; the unmanned aerial vehicle performs highway inspection according to the preset inspection path in the task instruction by relying on the navigation system carried by itself and the navigation information provided by the ground control station.

[0033] The drone nest receives mission instructions and controls the drone's takeoff and inspection process, achieving a fully automated and intelligent operation mode through systematic process design. Using satellite communication networks, the ground control station overcomes geographical limitations and remotely transmits mission instructions to the drone nest. Upon receiving the instructions, the nest automatically performs hardware status checks, including checking the drone's battery level, opening the hatch, and activating the navigation and communication systems. The drone's takeoff command is triggered only when the drone's battery level reaches a preset threshold in the database and all systems complete self-tests, thus avoiding flight risks caused by insufficient power or equipment failure.

[0034] After taking off, the drone relies on the dual navigation information provided by its own navigation system and the ground control station to accurately follow the inspection route preset in the mission instructions. Whether it is a long-distance, complex road section with many bends, or a remote area without network coverage, it can achieve a full-range inspection without blind spots.

[0035] Transforming manual intervention into an automated verification and execution process significantly improves task response speed and inspection efficiency, while reducing manpower and operation and maintenance costs. The full recording and systematic management of task data also provides data support for subsequent traffic management decision-making optimization, laying a solid foundation for the digital and intelligent development of smart transportation.

[0036] Collect the original video stream from the drone, obtain a high-precision vector map of the highway, perform multimodal road intelligent segmentation, and output a binary road mask.

[0037] The specific analysis process is as follows: collecting the original video stream from the drone, including dual channels of visible light images and infrared images; obtaining a high-precision vector map of the highway, including the GPS coordinates of lane lines and guardrails; inputting the infrared image of the original video stream from the drone into the trained convolutional neural network model, and outputting the probability that each pixel in the infrared image belongs to a different material; filtering out pixels belonging to non-asphalt areas based on the probability output by the convolutional neural network model: when the convolutional neural network model predicts that the probability of a certain pixel belonging to asphalt is greater than the asphalt probability threshold stored in the database, the pixel is marked as an asphalt area and retained as a road surface pixel; when the convolutional neural network model predicts that the probability of a certain pixel belonging to asphalt is not greater than the asphalt probability threshold stored in the database, the pixel is marked as a non-asphalt area, and the pixels belonging to the non-asphalt area are filtered out and not retained;

[0038] Using the positioning data and attitude data of the UAV, combined with the coordinate information of the high-precision vector map, the lane line coordinates of the vector map are projected into the image space through the perspective transformation matrix: the positioning data of the UAV is specifically the GPS coordinates, and the attitude data of the UAV is specifically the pitch angle, yaw angle, and roll angle; based on the GPS coordinates of the UAV and the coordinate system of the vector map, the position of the vector map in the UAV coordinate system is calculated; the focal length, principal point coordinates, rotation matrix, and translation vector of the UAV are collected to construct the perspective transformation matrix, and the projection position of the vector map in the image space is calculated based on the attitude data of the UAV; the lane line coordinates of the vector map are mapped into the image space through the perspective transformation matrix to obtain the pixel coordinates of the lane line on the image; a reference coordinate system is established with the center line of the projected lane line as the x-axis of the reference coordinate system and the normal direction of the lane line as the y-axis;

[0039] Dynamically detect accumulated water and reflective areas and repair image anomalies; perform edge detection on the visible light image of the original drone video stream, extract edge information in the image, and use edge detection and region segmentation algorithms in computer vision to detect accumulated water and reflective areas; use a trained generative adversarial network to repair the detected accumulated water and reflective areas; input the detected accumulated water and reflective areas into the trained generative adversarial network, and the generator outputs the repaired normal road image.

[0040] The filtered asphalt area and the repaired image are binarized, with the pixel values ​​of the asphalt area set to 1 and the pixel values ​​of the non-asphalt area set to 0 to obtain a binary mask image, and a binary road surface mask is output. If new obstacles or construction areas are found in the binary road surface mask and these areas are inconsistent with the information in the vector map, a road topology change warning is triggered, and the warning information includes the coordinates and area of ​​the changed area.

[0041] The integrated use of multi-source data and advanced algorithms provides precise, efficient, and intelligent support for highway inspections. By collecting raw drone video streams of both visible light and infrared images, combined with high-precision vector map data, and using convolutional neural networks to predict pixel-level material probabilities in infrared images, the system accurately filters non-asphalt areas based on asphalt probability thresholds, effectively eliminating interference from the surrounding road environment and focusing on the main road surface where vehicles are traveling.

[0042] Leveraging drone positioning and attitude data and vector map coordinates, a perspective transformation matrix is ​​constructed to accurately project lane lines from vector space to image space, establishing a reliable image reference coordinate system and laying the spatial foundation for subsequent vehicle position and motion analysis. Edge detection and region segmentation algorithms are used to locate problematic areas in visible light images, such as water accumulation and reflections, which can interfere with detection. A generative adversarial network is then used for intelligent inpainting to eliminate image noise and interference.

[0043] The processed asphalt area and the patch image are binarized to generate a clear road surface mask. This not only intuitively presents the road condition, but also allows for real-time comparison with the vector map to promptly detect new obstacles, construction areas, and other topological changes, triggering precise warnings and providing key information such as the coordinates and area of ​​the changed area.

[0044] It improves the accuracy and completeness of road information collection, enhances the ability to perceive complex road conditions and emergencies, provides a high-quality data basis for subsequent vehicle tracking, speed measurement and traffic management decisions, and greatly improves the efficiency and safety of highway inspections.

[0045] The vehicle detection frames and IMU attitude data of the drone are collected from the original video stream of the drone. Combined with the binary road mask, spatiotemporal collaborative vehicle tracking is performed to obtain a vehicle trajectory dataset with a unique ID.

[0046] The specific analysis process is as follows: run the trained YOLO-X model within the constraints of the binary road mask, perform vehicle detection on the asphalt area in the binary road mask, and only when a detection frame is completely within the asphalt area of ​​the binary road mask, is the detection frame determined to be a valid vehicle detection result; combine IMU data to compensate for the target offset caused by the shaking of the drone; use the drone's IMU data to compensate for the target offset caused by the shaking of the drone. The drone's IMU data includes data from the drone's accelerometer, drone's gyroscope, and drone's magnetometer; calculate the drone's real-time attitude changes based on the IMU data, including pitch, yaw, and roll angle changes; calculate the target's offset pixel value on the image plane based on the attitude changes; perform corresponding offset compensation on the vehicle detection frame so that the detection frame can accurately track the target vehicle.

[0047] Short-term trajectory: Based on the Kalman prediction of three adjacent frames, the Kalman filter is used to predict and correct the vehicle's short-term trajectory: the state vector of the Kalman filter is initialized, including the vehicle's position and velocity; the vehicle's position in the next frame is predicted based on the vehicle's motion model; the prediction result is corrected using the detection box information of the three adjacent frames, and the state vector of the Kalman filter is updated based on the position and velocity information of the detection box, thereby obtaining a more accurate short-term trajectory prediction result;

[0048] Mid-time trajectory: trajectory smoothing optimization within a sliding window. A sliding window algorithm is used to smooth and optimize the vehicle's mid-time trajectory. The sliding window is as large as 10 frames. The vehicle's trajectory is fitted within the sliding window and smoothed using the Bezier curve fitting method. Based on the fitting results, the vehicle's trajectory is corrected and the fitted trajectory is used as the vehicle's true trajectory. The jitter and noise in the original trajectory are smoothed to obtain a more stable mid-time trajectory.

[0049] Long-term trajectory, cross-camera ID matching, in the case of multi-machine collaboration, the long-term trajectory of the vehicle ID matching: collect vehicle detection frame information captured by multiple drones, including the vehicle's position, speed, and appearance features; obtain the vehicle's color, texture, speed, and acceleration, and calculate the similarity between vehicles captured by different drones; based on the similarity, the vehicle ID is matched. If the similarity of vehicles captured by two drones exceeds the similarity threshold stored in the database, they are considered to be the same vehicle, achieving cross-camera ID matching and obtaining the long-term trajectory of the vehicle;

[0050] The short-term, medium-term, and long-term trajectory information of vehicles are integrated into a dataset, and a unique ID is assigned to each vehicle. Each row in the dataset represents a vehicle trajectory point, including the vehicle's unique ID, frame number, pixel coordinates (x, y), movement direction, and timestamp, resulting in a vehicle trajectory dataset with a unique ID. Abnormal trajectories are marked: if the vehicle's movement direction is opposite to the normal driving direction of the lane, it is marked as wrong-way; if the vehicle crosses multiple lane lines in a short period of time, it is marked as a sudden lane change. Abnormal trajectory marking information includes the vehicle's unique ID, abnormality type, occurrence time, and occurrence location.

[0051] Precise vehicle tracking is achieved through the integration of multi-dimensional technologies. The detection range is constrained by a binary road mask, which can significantly reduce interference from non-road areas, improve the accuracy and efficiency of vehicle detection, and ensure that the detection frame is only for vehicles within the effective road area. The IMU attitude data is used to compensate for the target offset caused by the shaking of the drone, and the detection error caused by the change of the drone's flight attitude can be dynamically corrected, so that the vehicle detection frame can still stably lock the target in a complex flight environment. The Kalman filter is used for short-term trajectory prediction and correction, and the position changes between adjacent frames can be dynamically optimized based on the vehicle motion model, effectively reducing the detection noise between short-term frames and improving the continuity of the trajectory in the initial stage.

[0052] The use of sliding windows combined with Bezier curve fitting for ongoing trajectory smoothing can systematically filter trajectory jitter and noise within a 10-frame range, making the vehicle's motion trajectory over a period of time more consistent with actual physical laws and enhancing the reliability of trajectory data; cross-camera ID matching technology calculates similarity by fusing vehicle appearance features (color, texture) and motion features (speed, acceleration), and can accurately associate the long-term trajectory of the same vehicle when multiple drones work together, breaking the field of view limitations of single-machine inspections and realizing full-trip vehicle tracking.

[0053] Assigning a unique ID to the vehicle and integrating multi-period trajectory information to form a structured trajectory dataset facilitates subsequent time-series analysis of vehicle movement behavior. Abnormal trajectory marking (reversing, sudden lane changes) can identify dangerous driving behaviors in real time, providing data support for traffic supervision and safety warnings.

[0054] Obtain the drone's RTK positioning data and ground marker coordinates, and output a pixel-to-physical space mapping table based on the vehicle detection frame in the drone's original video stream.

[0055] The specific analysis process is as follows: use the placed reflective markers to establish a benchmark scale; obtain the spacing of the placed reflective markers stored in the database; use the target detection algorithm to identify the reflective markers in the visible light image taken by the drone, and calculate the pixel distance between adjacent reflective markers in the visible light image; and calculate the pixel scale factor of the visible light image based on the spacing of the placed reflective markers stored in the database and the pixel distance between adjacent reflective markers in the visible light image.

[0056] Dynamically adjust the perspective transformation matrix based on the UAV's RTK positioning data and attitude data. The UAV attitude data includes the UAV's altitude, pitch angle, and yaw angle: obtain the UAV's real-time position information based on the UAV's RTK positioning data; calculate the UAV's rotation matrix and translation vector based on the UAV's attitude data; construct a perspective transformation matrix based on the UAV's focal length and principal point coordinates. When the UAV's altitude, pitch angle, and yaw angle change, update the rotation matrix and translation vector in real time and dynamically adjust the perspective transformation matrix; use the lane line curvature data in the high-precision vector map of the highway to reversely infer the lens distortion parameters: calculate the true curvature of the lane line based on the lane line curvature information in the vector map; extract the lane line edge information in the UAV's visible light image, and calculate the curvature of the lane line in the visible light image; reversely infer the lens distortion parameters by comparing the true curvature of the lane line with the curvature in the visible light image; update the pixel-physical space mapping table in real time based on the dynamically adjusted perspective transformation matrix and the reversed lens distortion parameters.

[0057] By placing reflective markers to establish a benchmark scale and combining the database storage spacing with the image pixel distance to calculate the pixel scale factor, the benchmark conversion accuracy between physical space and image pixels can be ensured. The perspective transformation matrix is ​​dynamically adjusted based on the drone's RTK positioning data and attitude data (altitude, pitch angle, yaw angle, etc.), which can adapt to changes in the drone's position and attitude during flight and ensure the accuracy of spatial mapping under different perspectives.

[0058] By inferring lens distortion parameters using lane curvature data from high-precision highway vector maps, the impact of lens distortion on spatial mapping can be corrected in real time. Multi-source data fusion and a dynamic calibration mechanism achieve high-precision and real-time updates of the pixel-to-physical space mapping table, providing a reliable coordinate transformation foundation for subsequent conversion of vehicle speeds into physical space, ensuring the accuracy and robustness of speed measurement results.

[0059] Based on a vehicle trajectory dataset with a unique ID and a pixel-physical space mapping table, the system collects millimeter-wave radar altimeter data to measure vehicle speed and outputs a vehicle speed vector with a timestamp.

[0060] The specific analysis process is as follows: From the vehicle trajectory dataset with a unique ID, select the five consecutive frames with the largest displacement of the trajectory points; according to the pixel-to-physical mapping table, convert the pixel displacement of the trajectory points into the actual physical displacement; calculate the time difference between these five frames based on the timestamps, and thus obtain the vehicle speed, which is recorded as the basic speed measurement result;

[0061] When a speeding vehicle is detected, nearby drones are dispatched to synchronously collect trajectory data for the vehicle. The system determines the location of the neighboring drones based on their RTK positioning data. Synchronous collection instructions are sent to the neighboring drones, causing them to simultaneously collect trajectory data for the speeding vehicle. The trajectory data collected by the neighboring drones is then collected. Triangulation is used to process the trajectory data collected by multiple drones to eliminate perspective errors. A triangulation model is constructed based on the position and attitude data of the neighboring drones. Using the triangulation model, the trajectory data collected by different drones is fused to calculate the vehicle's three-dimensional position and speed.

[0062] Radar altitude data is corrected for scale changes to obtain a corrected speed measurement result: millimeter-wave radar altimeter data is used to correct for scale changes caused by the drone's ascent and descent. The drone's flight altitude is obtained in real time based on the millimeter-wave radar altimeter data. The scale factor in the pixel-to-physical mapping table is adjusted based on the drone's altitude change. The vehicle's speed is recalculated using the corrected scale factor and recorded as the corrected speed measurement result.

[0063] The optical flow method is used to analyze the number of rotations of the vehicle tires to assist in verifying the vehicle speed: the optical flow method is used to process the image sequence of the vehicle tires to calculate the rotation speed of the tires; the vehicle speed is calculated based on the tire rotation speed and the vehicle's wheel diameter; the vehicle speed calculated by the optical flow method is compared with the corrected speed measurement result. If the difference between the vehicle speed calculated by the optical flow method and the corrected speed measurement result is greater than the vehicle speed difference threshold stored in the database, the corrected speed measurement result is updated to the vehicle speed calculated by the optical flow method to obtain an updated speed measurement result; based on the updated speed measurement result, the vehicle's longitudinal speed component and latitudinal speed component are obtained; the vehicle's speed information is integrated into a data structure, which includes the vehicle's unique ID, longitudinal speed component, latitudinal speed component, and timestamp, and outputs a vehicle speed vector with a timestamp.

[0064] Through the deep fusion of multi-source data and a dynamic calibration mechanism, high-precision and highly reliable speed measurement is achieved. In the basic speed measurement process, the pixel displacement of the vehicle trajectory point is accurately converted to actual physical displacement based on a pixel-to-physical mapping table. The time difference is calculated by combining five consecutive frames of data, effectively smoothing the measurement noise and ensuring the stability of the basic speed measurement results. The millimeter-wave radar altimeter senses the drone's flight altitude changes in real time and dynamically adjusts the pixel-to-physical mapping scale factor to eliminate measurement scale deviations caused by the drone's ascent and descent.

[0065] For speeding vehicles, this solution innovatively dispatches nearby drones to conduct synchronous trajectory collection, fuses data from multiple drones through triangulation, breaks through the perspective distortion limitations of a single perspective, achieves accurate speed measurement in three-dimensional space, and significantly improves measurement accuracy in complex scenarios.

[0066] Optical flow analysis is used to analyze vehicle tire rotation speeds, which are then cross-validated with trajectory speed measurements. When the difference between the two exceeds a threshold, the speed measurement result is automatically updated, further enhancing the system's robustness under extreme conditions such as occlusion and perspective changes. The final output, a timestamped velocity vector, not only contains structured information such as the vehicle's unique ID and latitude and longitude velocity components, but can also be linked to vehicle trajectory data to support advanced applications such as traffic flow modeling and abnormal behavior analysis. Through its multi-sensor redundant design and dynamic adaptive mechanism, the system significantly reduces its reliance on specific environments or equipment, effectively improving its reliability and practicality in all-weather, unmanned traffic monitoring scenarios.

[0067] The above content is merely an example and explanation of the structure of the present invention. Those skilled in the art may make various modifications or additions to the described specific embodiments or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the scope of protection of the present invention.

Claims

1. A vehicle speed measurement method during drone highway inspection based on a drone nest, characterized in that: The following steps are involved: The drone nest receives the mission instructions, and after taking off, the drone conducts highway inspections based on the inspection path; Collect the original video stream from the drone, obtain a high-precision vector map of the highway, perform multimodal road intelligent segmentation, and output a binary road mask; The vehicle detection frames and IMU attitude data from the original drone video stream are collected and combined with the binary road mask to perform spatiotemporal collaborative vehicle tracking to obtain a vehicle trajectory dataset with a unique ID. Obtain the drone's RTK positioning data and ground marker coordinates, and output a pixel-to-physical space mapping table based on the vehicle detection frame in the drone's original video stream; Based on a vehicle trajectory dataset with a unique ID and a pixel-physical space mapping table, the system collects millimeter-wave radar altimeter data to measure vehicle speed and outputs a vehicle speed vector with a timestamp.

2. The vehicle speed measurement method during highway inspection based on a drone nest according to claim 1 is characterized by: The drone nest receives the mission instruction, and after taking off, the drone conducts highway inspection based on the inspection path. The specific analysis process is as follows: The ground control station sends the mission instructions to the UAV nest through the satellite communication network, and the UAV nest receives the mission instructions; The drone nest checks the drone's battery level, opens the nest door, and starts the navigation and communication systems; When the drone nest checks that the drone's battery level reaches the drone's battery level threshold stored in the database, and the nest door is successfully opened, and the navigation system and communication system are started, a take-off command is issued to the drone, and the drone takes off from the nest under the action of its own flight control system; The drone relies on its own navigation system and navigation information provided by the ground control station to conduct highway inspections according to the inspection routes preset in the mission instructions.

3. The vehicle speed measurement method during highway inspection using a drone based on a drone nest according to claim 1 is characterized by: The method collects the original video stream of the drone, obtains the high-precision vector map of the highway, performs multimodal road surface intelligent segmentation, and outputs a binary road surface mask. The specific analysis process is as follows: Collect the original video stream of the drone, including dual channels of visible light image and infrared image; Obtain high-precision vector maps of highways, including lane lines and guardrail GPS coordinates; Input the infrared image of the drone's original video stream into the trained convolutional neural network model and output the probability that each pixel in the infrared image belongs to a different material; According to the probability output by the convolutional neural network model, pixels belonging to non-asphalt areas are filtered: When the convolutional neural network model predicts that the probability of a pixel belonging to asphalt is greater than the asphalt probability threshold stored in the database, the pixel is marked as an asphalt area and retained as a road surface pixel; When the convolutional neural network model predicts that the probability of a pixel belonging to asphalt is not greater than the asphalt probability threshold stored in the database, the pixel is marked as a non-asphalt area, and the pixels belonging to the non-asphalt area are filtered out and not retained; Using the positioning data and attitude data of the drone, combined with the coordinate information of the high-precision vector map, the lane line coordinates of the vector map are projected into the image space through the perspective transformation matrix: The positioning data of the drone is specifically the GPS coordinates, and the attitude data of the drone is specifically the pitch angle, yaw angle, and roll angle; According to the GPS coordinates of the drone and the coordinate system of the vector map, the position of the vector map in the drone coordinate system is calculated; The focal length, principal point coordinates, rotation matrix, and translation vector of the drone are collected to construct a perspective transformation matrix. Based on the drone's attitude data, the projection position of the vector map in the image space is calculated. Map the lane line coordinates of the vector map to the image space through the perspective transformation matrix to obtain the pixel coordinates of the lane line on the image; Establish a reference coordinate system with the center line of the projected lane line as the x-axis and the normal direction of the lane line as the y-axis. Dynamically detect water accumulation and reflective areas and repair image anomalies; The filtered asphalt area and the repaired image are binarized, the pixel value of the asphalt area is set to 1, and the pixel value of the non-asphalt area is set to 0, to obtain a binary mask image, and the binary road surface mask is output; If new obstacles or construction areas are found in the binary road mask and these areas are inconsistent with the information in the vector map, a road topology change warning will be triggered. The warning information includes the coordinates and area of ​​the changed area.

4. The vehicle speed measurement method for drone highway inspection based on drone nest according to claim 3 is characterized by: The dynamic detection of water accumulation and reflective areas and the repair of image anomalies are specifically analyzed as follows: Perform edge detection on the visible light image of the drone's original video stream, extract edge information from the image, and use edge detection and region segmentation algorithms in computer vision to detect accumulated water and reflective areas; Use the trained generative adversarial network to repair the detected water accumulation and reflective areas; The detected water accumulation and reflective areas are input into the trained generative adversarial network, and the generator outputs the repaired normal road surface image.

5. The vehicle speed measurement method during highway inspection using a drone based on a drone nest according to claim 1 is characterized by: The vehicle detection frame and IMU posture data of the drone are collected in the original video stream of the drone, and combined with the binary road mask to perform spatiotemporal collaborative vehicle tracking to obtain a vehicle trajectory dataset with a unique ID. The specific analysis process is as follows: Run the trained YOLO-X model within the constraints of the binary road mask to detect vehicles in the asphalt area of ​​the binary road mask. Only when a detection box is completely within the asphalt area of ​​the binary road mask is the detection box considered a valid vehicle detection result. Combined with IMU data to compensate for target offset caused by drone shaking; Short-term trajectory, based on the Kalman prediction of the adjacent three frames, uses the Kalman filter to predict and correct the short-term trajectory of the vehicle: Initialize the state vector of the Kalman filter, including the position and speed of the vehicle; Predict the vehicle's position in the next frame based on the vehicle's motion model; The prediction results are corrected using the detection frame information of the three adjacent frames. The state vector of the Kalman filter is updated based on the position and velocity information of the detection frame, thereby obtaining a more accurate short-term trajectory prediction result. Mid-time trajectory, trajectory smoothing optimization within the sliding window, using the sliding window algorithm to smooth the vehicle's mid-time trajectory: The maximum sliding window is 10 frames. The vehicle trajectory is fitted within the sliding window and the Bezier curve fitting method is used to smooth the vehicle trajectory. Based on the fitting results, the vehicle's trajectory is corrected and the fitted trajectory is used as the vehicle's true trajectory. The jitter and noise in the original trajectory are smoothed to obtain a more stable mid-time trajectory. Long-term trajectory, cross-camera ID matching, in the case of multi-machine collaboration, the long-term trajectory of the vehicle ID matching: Collect vehicle detection frame information captured by multiple drones, including the vehicle's location, speed, and appearance characteristics; Obtain the color, texture, speed, and acceleration of the vehicle, and calculate the similarity between vehicles photographed by different drones; Based on the similarity, the vehicle ID is matched. If the similarity of the vehicles photographed by two drones exceeds the similarity threshold stored in the database, they are considered to be the same vehicle, achieving cross-camera ID matching and obtaining the long-term trajectory of the vehicle. The short-term, medium-term, and long-term trajectory information of the vehicle are integrated into a dataset. A unique ID is assigned to each vehicle. Each row in the dataset represents a vehicle trajectory point, including the vehicle's unique ID, frame number, pixel coordinates (x, y), movement direction, and timestamp. This yields a vehicle trajectory dataset with a unique ID. Mark abnormal trajectories: If the vehicle's movement direction is opposite to the normal driving direction of the lane it is in, it is marked as wrong-way; if the vehicle crosses multiple lane lines in a short period of time, it is marked as a sudden lane change. The abnormal trajectory marking information includes the vehicle's unique ID, abnormality type, occurrence time, and occurrence location.

6. The vehicle speed measurement method during highway inspection using a drone based on a drone nest according to claim 5 is characterized by: The target offset caused by the shaking of the drone is compensated by combining IMU data. The specific analysis process is as follows: Use the drone's IMU data to compensate for target offset caused by drone shaking. The drone's IMU data includes data from the drone's accelerometer, drone's gyroscope, and drone's magnetometer. Calculate the real-time attitude changes of the drone based on IMU data, including pitch angle, yaw angle, and roll angle changes; According to the posture change, the offset pixel value of the target on the image plane is calculated; The vehicle detection frame is compensated for its offset accordingly so that the detection frame can accurately track the target vehicle.

7. The vehicle speed measurement method for drone highway inspection based on drone nest according to claim 1 is characterized by: The acquisition of drone RTK positioning data and ground marker coordinates, combined with the vehicle detection frame in the drone's original video stream to output a pixel-physical space mapping table, is performed in the following specific analysis process: Establish benchmarks using placed reflective markers; Dynamically adjust the perspective transformation matrix based on the drone's RTK positioning data and attitude data. The drone's attitude data includes the drone's altitude, drone's pitch angle, and drone's yaw angle: Obtain the real-time location information of the drone based on its RTK positioning data; Calculate the rotation matrix and translation vector of the drone based on the drone's attitude data; Combined with the focal length and principal point coordinates of the drone, a perspective transformation matrix is ​​constructed. When the drone's altitude, pitch angle, and yaw angle change, the rotation matrix and translation vector are updated in real time to dynamically adjust the perspective transformation matrix. Using the lane curvature data from the high-precision vector map of the highway, we can infer the lens distortion parameters: Calculate the actual curvature of the lane line based on the lane line curvature information in the vector map; In the visible light image of the UAV, the edge information of the lane line is extracted and the curvature of the lane line in the visible light image is calculated; By comparing the actual curvature of the lane line with the curvature in the visible light image, the lens distortion parameters are inferred. The pixel-physical space mapping table is updated in real time based on the dynamically adjusted perspective transformation matrix and the inverse lens distortion parameters.

8. The vehicle speed measurement method for drone highway inspection based on drone nest according to claim 7 is characterized by: The reference scale is established by using the reflective markers placed, and the specific analysis process is as follows: Obtaining the spacing of the placed reflective markers stored in the database; Use the target detection algorithm to identify reflective markers in the visible light images taken by the drone and calculate the pixel distance between adjacent reflective markers in the visible light images: The pixel scale factor of the visible light image is calculated based on the spacing between the placed reflective markers stored in the database and the pixel distance between adjacent reflective markers in the visible light image.

9. The vehicle speed measurement method during highway inspection using a drone based on a drone nest according to claim 1 is characterized by: The vehicle trajectory dataset with a unique ID and the pixel-physical space mapping table are used to collect millimeter-wave radar altimeter data for vehicle speed measurement, and output a vehicle speed vector with a timestamp. The specific analysis process is as follows: Select the 5 consecutive frames with the largest displacement among the trajectory points from the vehicle trajectory dataset with a unique ID; According to the pixel-physical mapping table, the pixel displacement of the trajectory point is converted into the actual physical displacement. The time difference between the five frames is calculated based on the timestamp to obtain the vehicle speed, which is recorded as the basic speed measurement result. When a vehicle is detected speeding, nearby drones are dispatched to synchronously collect the vehicle's trajectory data: Determine the location of neighboring drones based on the drone’s RTK positioning data; Send synchronous collection instructions to neighboring drones, so that they can collect trajectory data of speeding vehicles at the same time; Collect trajectory data collected by neighboring drones; Use triangulation to process trajectory data collected by multiple aircraft to eliminate perspective errors: Build a triangulation model based on the position and attitude data of neighboring drones; Using a triangulation model, the trajectory data collected by different drones is fused to calculate the three-dimensional position and velocity of the vehicle; The radar altitude data is corrected for scale changes to obtain a corrected velocity measurement result; Use the optical flow method to analyze the number of rotations of the vehicle tires to assist in verifying the vehicle's speed: The image sequence of the vehicle tire is processed using the optical flow method to calculate the rotation speed of the tire; Calculate the vehicle's speed based on the tire's rotation speed and the vehicle's wheel diameter; The vehicle speed calculated by the optical flow method is compared with the corrected speed measurement result. If the difference between the vehicle speed calculated by the optical flow method and the corrected speed measurement result is greater than the speed difference threshold stored in the database, the corrected speed measurement result is updated to the vehicle speed calculated by the optical flow method to obtain an updated speed measurement result. Obtaining the vehicle's longitudinal velocity component and latitudinal velocity component based on the updated velocity measurement results; The vehicle's speed information is integrated into a data structure, including the vehicle's unique ID, longitude speed component, latitude speed component, and timestamp, and the vehicle speed vector with a timestamp is output.

10. The vehicle speed measurement method during highway inspection using a drone based on a drone nest according to claim 9, characterized in that: The radar height data correction scale changes, the specific analysis process is as follows: Use millimeter-wave radar altimeter data to correct for scale changes caused by the drone's ascent and descent: According to the millimeter wave radar altimeter data, the flight altitude of the drone is obtained in real time; Adjust the scale factor in the pixel-to-physical mapping table according to the altitude change of the drone; The corrected scale factor is used to recalculate the vehicle's speed, which is recorded as the corrected speed measurement result.

Images