Traffic event handling method and system based on unmanned aerial vehicle visual servo

Abstract

The invention provides a traffic event handling method and system based on unmanned aerial vehicle visual servo, and relates to the technical field of intelligent traffic. The method comprises the following steps: acquiring image data and video data of a target area; based on the target area image data and the video data, calculating the pixel position of the target vehicle, and calculating the relative position of the unmanned aerial vehicle and the target vehicle; the relative position comprises a horizontal distance and a deviation angle; calculating an actual position of the target vehicle based on the relative position and the pixel position; and taking the actual position of the target vehicle as a final target point, planning an approaching route of the unmanned aerial vehicle, and controlling the unmanned aerial vehicle to hover at a preset height above the final target point to execute task operation corresponding to the event type. The method and the device are used in a traffic violation disposal process based on the unmanned aerial vehicle, and solve a technical problem that the unmanned aerial vehicle cannot realize precise hovering of a specific target in traffic management in the prior art.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation technology, and in particular to a method and system for handling traffic incidents based on UAV visual servoing. Background Technology

[0002] Traffic anomalies such as illegal parking, traffic accidents, vehicles driving against traffic, and debris spills causing blockages occur frequently. If these incidents are not responded to and handled quickly, they can easily lead to serious secondary accidents and traffic congestion. Existing traffic incident management methods mainly include fixed roadside surveillance camera systems, manual patrol vehicles, and drone-based auxiliary inspections, but each method has its shortcomings. Fixed surveillance systems are limited by installation location, field of view, and image resolution, making it impossible to achieve blind-spot-free coverage of the entire road segment. Furthermore, incident identification largely relies on manual monitoring, leading to missed detections, misjudgments, and response delays. Manual patrols are constrained by patrol frequency, road conditions, and geographical location, making it difficult to guarantee arrival at the scene within the critical time window after an incident occurs. While existing drone inspections offer the advantage of high mobility, they mostly rely on manual remote control or preset flight paths. Target positioning mainly depends on GNSS, whose meter-level accuracy cannot meet the requirements for precise identification, accurate positioning, and stable hovering of specific traffic incident targets. Therefore, a method is urgently needed to solve the technical problem of drones' inability to achieve precise hovering of specific targets in traffic management. Summary of the Invention

[0003] This application provides a traffic incident handling method and system based on UAV visual servoing, which solves the technical problem that UAVs cannot achieve precise hovering over specific targets in traffic management in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution: Firstly, a traffic incident handling method based on UAV visual servoing is provided, comprising: acquiring image data and video data of the target area; calculating the pixel position of the target vehicle based on the image data and video data of the target area, and calculating the relative position between the UAV and the target vehicle; the relative position includes horizontal distance and deviation angle; calculating the actual position of the target vehicle based on the relative position and pixel position; planning the UAV approach route with the actual position of the target vehicle as the final target point, and controlling the UAV to hover at a preset height above the final target point to perform task operations corresponding to the event type.

[0005] In conjunction with the first aspect mentioned above, in one possible implementation, before acquiring image data and video data of the target area, the UAV-based traffic violation handling method further includes: acquiring traffic anomaly warning information; based on the traffic anomaly warning information, planning the optimal route from the UAV to the initial target airspace through a strategy combining optimal path search and local obstacle avoidance; and scheduling the UAV to the initial target airspace based on the optimal route from the UAV to the initial target airspace, and collecting image data and video data of the target area in the initial target airspace.

[0006] In conjunction with the first aspect mentioned above, in one possible implementation, the pixel position of the target vehicle is calculated based on the target area image data and video data, and the relative position of the UAV and the target vehicle is calculated. This includes: detecting the target vehicle in the image data and video data to identify the location region of the target vehicle in the image; extracting the feature point coordinates of the target vehicle based on the location region and calculating the pixel position of the target vehicle; obtaining the real-time flight altitude, attitude angle, and camera intrinsic and extrinsic parameters of the UAV; and calculating the horizontal distance and deviation angle between the UAV and the target vehicle through spatial geometric back projection based on the pixel position, camera intrinsic and extrinsic parameters, real-time flight altitude of the UAV, and attitude angle.

[0007] In conjunction with the first aspect mentioned above, in one possible implementation, target vehicle detection is performed on image data and video data to identify the location region of the target vehicle in the image. This includes: extracting a mask point set of the highway based on the image data and video data, and using the minimum bounding rectangle rotation algorithm to obtain the minimum bounding rectangle, the long side of the minimum bounding rectangle rotation, and the short side of the minimum bounding rectangle rotation; determining the main direction of the highway based on the direction of the long side of the minimum bounding rectangle rotation, and adjusting the camera imaging direction to be consistent with the direction of the long side of the minimum bounding rectangle rotation; performing vehicle search on the image in the minimum bounding rectangle rotation short side field of view mode; if no target vehicle is detected, rotating the camera field of view by 90 degrees and switching to the minimum bounding rectangle rotation long side field of view mode to continue the search, obtaining candidate vehicle regions, and determining the location region of the target vehicle in the image.

[0008] In conjunction with the first aspect mentioned above, in one possible implementation, the actual location of the target vehicle is taken as the final target point. A drone approach route is planned, and the drone is controlled to hover at a preset height above the final target point to perform tasks corresponding to the event type. This includes: constructing a feasible flight space for the drone based on the actual location of the target vehicle and the drone's current flight position; using the actual location of the target vehicle as the final target point, within the feasible flight space, a path search algorithm is used to determine the drone's approach route from its current flight position to the final target point; controlling the drone to fly to a preset height above the final target point and hover according to the approach route from its current flight position to the final target point; and in the hovering state, performing corresponding tasks according to the event type, including evidence collection for violations, warning broadcasts, or information feedback.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the actual position of the target vehicle is taken as the final target point. Within the feasible flight space of the UAV, a path search algorithm is used to determine the approximation route of the UAV from its current flight position to the final target point. This includes: constructing a gridded model of the feasible flight space based on the current position of the UAV, flight altitude limitations, and obstacle distribution; calculating a feasible path in the gridded model using the current flight position as the starting point and the final target point as the ending point through the path search algorithm; and smoothing the trajectory based on the feasible path to obtain an approximation route suitable for UAV flight control.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, controlling the UAV to hover at a preset altitude above the final target point based on the approximation route includes: generating a desired pose sequence of the UAV according to the path points of the approximation route; controlling the speed, acceleration, and attitude angle of the UAV based on the desired pose sequence to drive the UAV to fly along the approximation route; adjusting the altitude based on the real-time altitude information of the UAV during the approximation process to gradually bring the UAV closer to the preset altitude; and controlling the UAV to enter a hovering mode and stabilize at the preset altitude above the final target point when both the position error and altitude error are less than preset thresholds.

[0011] Secondly, a traffic incident handling system based on UAV visual servoing is provided, comprising: an image acquisition module, a target detection module, a position calculation module, a path planning module, and a control module; the image acquisition module is used to acquire image data and video data of the target area; the target detection module is used to identify the position area of ​​the target vehicle in the image based on the image data and video data; the position calculation module is used to determine the pixel position of the target vehicle based on the position area of ​​the target vehicle in the image, and then determine the actual position of the target vehicle; the path planning module is used to plan the approach route of the UAV with the actual position of the target vehicle as the final target point; the control module is used to control the UAV to fly along the approach route, hover at a preset height above the final target point, and perform task operations corresponding to the event type.

[0012] In conjunction with the second aspect above, in one possible implementation, the position calculation module includes: a relative position calculation module and an actual position calculation module; the relative position calculation module is used to determine the pixel position of the target vehicle based on the target vehicle's position area, and calculate the relative position between the UAV and the target vehicle, the relative position including the horizontal distance and the deviation angle; the actual position calculation module is used to calculate the actual position of the target vehicle based on the pixel position and the relative position.

[0013] In conjunction with the second aspect mentioned above, in one possible implementation, the control module includes: a flight control module and a mission execution module; the flight control module is used to control the UAV to fly along the approach route and hover at a preset altitude above the final target point; the mission execution module is used to perform tasks such as traffic violation evidence collection, warning broadcasting, or information feedback according to the event type.

[0014] This application provides a traffic incident handling method and system based on UAV visual servoing. By acquiring image and video data of the target area and performing pixel-level position calculations on target vehicles, the UAV can obtain the precise position of the target vehicles in a two-dimensional imaging plane in real time. Furthermore, based on the pixel position difference, viewing angle relationship, and imaging parameters between the UAV and the target vehicle, the relative position, including horizontal distance and deviation angle, is calculated to achieve high-precision estimation of vehicle orientation and distance, effectively avoiding the inaccurate positioning problems caused by obstruction or weak signals in traditional GPS or vehicle tag-based methods. Subsequently, the actual geographical location of the target vehicle is calculated by combining pixel position and relative position, enabling the UAV to accurately determine the spatial coordinates of vehicles violating traffic rules in dynamic traffic environments. Using this actual location as the final target point, an approximation route is planned, allowing the UAV to automatically generate the optimal trajectory based on the real-time environment, avoiding buildings, road structures, and traffic flow. Ultimately, by controlling the drone to hover at a preset height above the target vehicle to perform corresponding tasks, the system ensures accurate task execution and stable movements, improves the efficiency of handling violations, and realizes an integrated automated process of perception, positioning, planning, and handling. This solves the technical problem that existing technologies cannot enable drones to accurately hover over specific targets in traffic management.

[0015] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0016] Figure 1 A system architecture diagram of a traffic incident handling system based on UAV visual servoing provided in this application embodiment; Figure 2 A flowchart illustrating a traffic incident handling method based on UAV visual servoing, provided for an embodiment of this application; Figure 3 A flowchart illustrating another traffic incident handling method based on UAV visual servoing provided in this application embodiment; Figure 4A flowchart illustrating another traffic incident handling method based on UAV visual servoing provided in this application embodiment; Figure 5 This is a schematic diagram showing the correspondence between the long and short sides, provided in an embodiment of this application. Detailed Implementation

[0017] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0018] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0019] The traffic violation processing method based on drones provided in this application can be applied to, for example... Figure 1 The drone-based traffic violation processing system shown in the image, such as Figure 1 As shown, the system includes: an image acquisition module 101, a target detection module 102, a position calculation module 103, a path planning module 104, and a control module 105; The system includes: an image acquisition module 101 for acquiring image and video data of the target area; a target detection module 102 for identifying the location of the target vehicle in the image based on the image and video data; a location calculation module 103 for determining the pixel position of the target vehicle based on its location in the image, and then determining the actual position of the target vehicle; a path planning module 104 for planning the approach route of the UAV using the actual position of the target vehicle as the final target point; and a control module 105 for controlling the UAV to fly along the approach route, hover at a preset height above the final target point, and perform task operations corresponding to the event type.

[0020] To address the technical problem that existing technologies cannot achieve precise hovering of drones towards specific targets in traffic management, this application provides a traffic incident handling method based on drone visual servoing. The method includes: acquiring image data and video data of a target area; calculating the pixel position of a target vehicle based on the image data and video data, and calculating the relative position between the drone and the target vehicle; the relative position includes horizontal distance and deviation angle; calculating the actual position of the target vehicle based on the relative position and pixel position; planning a drone approach route using the actual position of the target vehicle as the final target point, and controlling the drone to hover at a preset height above the final target point to perform a task operation corresponding to the event type.

[0021] Figure 2 A flowchart illustrating the traffic violation processing method based on unmanned aerial vehicles (UAVs) provided in this application embodiment is shown below. Figure 2 As shown, the method includes: S201. Acquire image data and video data of the target area.

[0022] Among them, target area image data and video data refer to static frame images and continuous frame video streams collected by the airborne visual acquisition equipment carried by the UAV in the initial airspace of the target. The image data includes single high-resolution images, and the video data includes time-series images composed of several frames, which are used for target detection and tracking.

[0023] In one possible implementation, the system acquires traffic anomaly warning information; based on the traffic anomaly warning information, it plans the optimal route from the UAV to the initial target airspace through a strategy combining optimal path search and local obstacle avoidance; based on the optimal route from the UAV to the initial target airspace, it dispatches the UAV to the initial target airspace and collects image and video data of the target area in the initial target airspace.

[0024] It should be noted that, in the embodiments of this application, the traffic operation status is perceived and analyzed in real time by fixed monitoring equipment, roadside sensing units or edge intelligent computing nodes deployed along the road. Algorithms such as image recognition, target detection and trajectory analysis are used to identify abnormal traffic events, including traffic accidents, illegal parking, driving in the wrong direction, vehicle rear-end collisions and road congestion. When an abnormal traffic event is detected, the system generates alarm information, which includes at least the event type, initial location coordinates (latitude and longitude information or road sign information), detection timestamp and event confidence index.

[0025] It should also be noted that the acquisition of image and video data should take into account the impact of lighting conditions, motion blur and camera parameters on subsequent positioning accuracy. Therefore, the preprocessing stage can include frame selection strategy, adaptive gain and exposure settings, and multi-frame fusion when necessary to ensure the reliability of subsequent pixel-level position extraction.

[0026] This step provides an accurate raw data foundation for subsequent identification, relative position calculation, and actual position back projection by acquiring high-quality, pre-processed image and video data, thereby directly affecting the positioning accuracy and evidence collection results.

[0027] S202. Based on the target area image data and video data, calculate the pixel position of the target vehicle and the relative position of the UAV and the target vehicle.

[0028] Pixel position refers to the pixel coordinates of the center point or representative feature point of the target vehicle in the image coordinate system (two-dimensional pixel grid); relative position refers to the geometric representation of the UAV relative to the target vehicle in the body coordinate system or horizon coordinate system, mainly including horizontal distance and deviation angle.

[0029] In one possible implementation, the system uses a target detection model to locate vehicles in preprocessed video keyframes and extract target bounding boxes or masks; it extracts key points or center points from candidate regions to obtain pixel coordinates; and then, by combining the UAV's real-time attitude, flight altitude, and known camera intrinsic and extrinsic parameters, it calculates the horizontal distance and the deviation angle processed by quadrant through geometric back projection.

[0030] It should be noted that pixel position extraction should include confidence assessment and multi-frame tracking verification to reduce false recognition; deviation angle calculation should consider the mapping relationship between the image coordinate system and the heading reference and perform angle range normalization processing to avoid control misjudgment caused by angle jump.

[0031] Based on the above steps, this step couples the visual detection results with the UAV pose parameters to achieve geometric mapping from pixel space to three-dimensional space, thereby obtaining accurate horizontal distance and deviation angle, providing key measurements for visual servo control to approach the target, and improving the accuracy and robustness of positioning and trajectory adjustment.

[0032] S203. Calculate the actual position of the target vehicle based on the relative position and pixel position.

[0033] The actual location of the target vehicle refers to its spatial location in the geographic coordinate system.

[0034] In one possible implementation, the system substitutes the pixel position and the intrinsic and extrinsic parameters of the UAV camera into the pinhole camera model to complete the initial ground projection, thereby obtaining the estimated position of the vehicle in the airborne coordinate system. The airborne coordinates are then converted into geographic coordinates by combining the UAV's GNSS position and heading information. To improve accuracy, the geographic location is further matched and optimized with a high-precision road geographic information model or road boundary constraints to correct deviations caused by camera parameter errors or measurement noise.

[0035] As an example, if a vehicle is located in the emergency lane of a highway and the initial projected position is off-center from the lane, the projected point can be corrected by projecting it along the nearest road centerline. If the projection results of multiple frames are linearly distributed, weighted average and variance analysis can be used to remove outlier observations in order to obtain a stable actual position result.

[0036] Based on the above steps, this step combines pixel-level visual measurement with UAV 3D pose and road constraints to generate highly reliable geographic location results, thereby solving the problems of insufficient accuracy of GNSS alone or unstable single-frame visual projection, and providing a reliable geographic reference for subsequent accurate approximation, evidence collection and platform reporting.

[0037] S204. Using the actual location of the target vehicle as the final target point, plan the drone's approach route and control the drone to hover at a preset height above the final target point to perform the task operation corresponding to the event type.

[0038] The approach route refers to the set of continuous flight paths from the current position of the drone to a preset altitude above the actual position of the target vehicle, which must meet navigation safety constraints; the preset altitude is the hovering altitude set in advance to ensure a balance between shooting angle, resolution and safe distance; the task operations corresponding to the event type include taking photos for evidence, transmitting video streams, license plate OCR, high-altitude loudspeaker announcements and sound and light warnings, etc.

[0039] In one possible implementation, the system constructs a feasible flight space based on the actual position of the target and the current state of the UAV, and calls a global or local path search algorithm to generate a trajectory. The system decomposes the trajectory into a desired pose sequence, and adjusts the roll, pitch and yaw in real time through a PID or PD controller combined with visual servo error, so that the UAV approaches the target along the trajectory and enters hovering when the position error and altitude error thresholds are met to perform the predetermined task operation.

[0040] It should be noted that when vision is lost or confidence decreases, the control strategy should switch to safe return or maintain high alert for re-identification in order to avoid misoperation or accidents.

[0041] Based on the above steps, this step uses high-precision actual position for track generation and combines it with visual servo real-time closed-loop control, enabling the UAV to hover stably directly above the target and perform evidence collection and disposal tasks with high quality. This significantly improves disposal efficiency, evidence collection reliability, and the system's automation level, thereby effectively solving the problems of inaccurate positioning and unstable hovering in the existing technology.

[0042] This application's embodiments acquire the pixel position information of the target vehicle in real time based on target area image and video data. Furthermore, by combining the UAV's own attitude parameters and imaging model, the relative position between the UAV and the target vehicle is calculated, enabling the UAV to obtain high-precision horizontal distance and deviation angle data in dynamic scenarios. Based on this, the actual position of the target vehicle in the geographic coordinate system is inferred using the relative position and pixel parameters, providing reliable, continuous, and error-controllable position information input for subsequent path planning. Using the actual position of the target vehicle as the final target point, the UAV performs approach route planning, achieving autonomous, safe, and collision-free approach control in complex environments. Ultimately, the UAV can stably hover above the target vehicle at a preset altitude and perform event processing tasks such as violation evidence collection, warning broadcasts, or information feedback, reducing manual intervention costs, enhancing the automation and reliability of task execution, and solving the technical problem in existing technologies where UAVs cannot achieve precise hovering over specific targets in traffic management.

[0043] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S202 can be specifically implemented through the following S301 to S304, which are explained in detail below: S301. By performing target vehicle detection on image data and video data, the location region of the target vehicle in the image is identified.

[0044] In one possible implementation, the system extracts a mask point set of the highway based on image and video data, and uses the minimum bounding rectangle algorithm to obtain the minimum bounding rectangle, the long side of the minimum bounding rectangle, and the short side of the minimum bounding rectangle. The main direction of the highway is determined based on the direction of the long side of the minimum bounding rectangle, and the camera imaging direction is adjusted to be consistent with the direction of the long side of the minimum bounding rectangle. Vehicle search is performed on the image in the minimum bounding rectangle short side field of view mode. If no target vehicle is detected, the camera field of view is rotated 90 degrees to switch to the minimum bounding rectangle long side field of view mode to continue the search, obtaining candidate vehicle regions and determining the location region of the target vehicle in the image.

[0045] It should be noted that the confidence threshold for target vehicle detection should be set in combination with factors such as illumination, motion blur and occlusion, and multi-frame fusion or inter-frame correlation judgment can be introduced to reduce false detection and false negative rates. When the confidence of the detection result is low or multiple targets overlap, more intensive image sampling should be triggered or the zoom camera should be switched to obtain higher resolution images to assist in the judgment.

[0046] As an example, after the drone acquires a video stream of the covered area in the initial airspace using a wide-angle camera, it extracts keyframes every N frames of the video for target vehicle detection. If a vehicle candidate matching the alarm information type is detected, the candidate region is tracked in subsequent frames and a stable target location region is output.

[0047] Based on the above steps, this step determines the clear location region of the target vehicle in the image through reliable detection and inter-frame tracking, providing accurate two-dimensional input for subsequent feature point extraction, pixel position calculation and spatial mapping, thereby improving the stability and robustness of the entire localization process.

[0048] S302. Extract the feature point coordinates of the target vehicle based on the location region, and calculate the pixel position of the target vehicle.

[0049] Among them, feature point coordinates refer to several pixel-level key points used to represent the vehicle position within the identified vehicle position area, such as the vehicle center point, license plate center point, or vehicle outline vertex; pixel position refers to the two-dimensional pixel coordinates of the selected representative point in the image coordinate system.

[0050] In one possible implementation, the system calculates the center of the bounding box within the candidate vehicle region as the initial representative point, and further uses a mask or key point detector to extract feature points such as the center of the license plate, the front of the vehicle, and the midpoint of the rear of the vehicle; weighted fusion or confidence-based rules are used to select the final pixel position for multiple feature points, and smoothing filtering is performed over multiple frames to reduce the impact of noise.

[0051] It should be noted that the selection of feature points should take into account the task requirements. When the vehicle is partially occluded, a mask outer contour fitting or semantic segmentation recovery strategy should be used to estimate reasonable feature point positions. At the same time, the confidence of feature points should be recorded for weighted processing during subsequent localization and fusion.

[0052] Based on the above steps, this step transforms the detection results into stable and semantically meaningful pixel coordinate inputs, ensuring the consistency and usability of pixel-level observations. This provides reliable basic data for subsequent geometric back-projection and relative position calculations, thereby directly affecting positioning accuracy and evidence collection quality.

[0053] S303: Obtain the real-time flight altitude, attitude angle, and camera intrinsic and extrinsic parameters of the UAV.

[0054] In one possible implementation, the system periodically reads and reports the UAV's position information, altitude sensor data, and attitude angles output by the IMU, while simultaneously reading camera intrinsic and extrinsic parameters from the onboard parameter storage.

[0055] It should be noted that the timing synchronization and calibration accuracy of sensor data have a significant impact on the final positioning accuracy. Therefore, compensation for calibration time deviations of camera intrinsic and extrinsic parameters should be performed periodically. In the presence of vibration or temperature drift, online self-calibration or real-time correction mechanisms should be enabled to reduce parameter errors.

[0056] Based on the above steps, this step obtains and verifies all the spatiotemporal parameters required to complete the back projection, ensuring the reliability of the timing and values ​​of the input parameters when using pixel observations to solve the three-dimensional position, thereby reducing the positioning deviation caused by sensor errors and improving the accuracy and repeatability of the actual position calculation.

[0057] S304, based on pixel position, camera intrinsic and extrinsic parameters, UAV real-time flight altitude and attitude angle, calculates the horizontal distance and deviation angle between the UAV and the target vehicle through spatial geometric back projection.

[0058] In one possible implementation, the system back-calculates the pixel position into a normalized direction vector in the camera coordinate system using the camera intrinsic parameters, and then transforms the direction vector into the geographic coordinate system based on the camera extrinsic parameters and the UAV attitude. Subsequently, starting from the current position and altitude of the UAV, the system calculates the intersection point with the ground plane along the direction vector, calculates the horizontal distance between the UAV projection point and the intersection point, and calculates the deviation angle by comparing the projection line with the heading of the UAV.

[0059] Based on the above steps, this step achieves a stable mapping from pixel measurement to three-dimensional geometric quantities, directly outputting horizontal distance and deviation angle that can be used for visual servoing, supporting closed-loop approximation control of UAVs, thereby improving the positioning accuracy of specific targets and the control reliability of descent and hovering.

[0060] This application's embodiments construct a unified spatial mapping model based on parameters such as the target vehicle's pixel position, relative horizontal distance, and deviation angle. Combined with the UAV's flight attitude, focal length parameters, and imaging distortion compensation mechanism, it ensures the consistency and traceability of the spatial mapping calculation. By performing coordinate calculation and attitude calibration on the mapping results, the actual position of the target vehicle with low positioning error is obtained, providing an accurate spatial reference for subsequent path planning and approximation decisions. Furthermore, after completing the actual position calculation, this scheme can perform multi-source data verification and temporal smoothing processing on the results, effectively suppressing positioning offsets caused by high-speed UAV movement, lighting changes, or image jitter, making subsequent approximation route planning more accurate and the final control strategy safer and more reliable.

[0061] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, the above S204 can be implemented through the following S401, S402 and S403, which are explained in detail below: S401. Based on the actual location of the target vehicle and the current flight location of the UAV, construct the feasible flight space for the UAV.

[0062] Among them, the feasible flight space refers to the three-dimensional spatial area in which the UAV can safely fly and complete the mission under the current flight environment and mission constraints. Constructing this space requires consideration of constraints such as geographical boundaries, no-fly zones, altitude restrictions, static and dynamic obstacles, as well as aviation regulations and mission safety distances.

[0063] In one possible implementation, after the system obtains the actual geographical location of the target vehicle and the current location information of the UAV, it constructs an environmental model by combining airborne sensors and ground map data. Then, it extracts the mission-related space near the target with a preset radius and represents the space by rasterization or voxelization, marking inaccessible and accessible units to form a feasible flight space representation.

[0064] As an example, when the system performs tasks along the highway, it first reads the locations of bridges and overpasses from the cloud map and marks them as no-fly voxels. Then, based on the point cloud scanned by the UAV's forward-facing lidar, it writes dynamic obstacles into the voxel grid constructed in real time. When large maintenance machinery or other aircraft approach the road, the corresponding voxel is immediately marked as impassable.

[0065] Based on the above steps, this step generates a feasible flight space by uniformly modeling environmental constraints and mission objectives, providing an accurate solution domain for path search, reducing the probability of path planning generating infeasible or dangerous trajectories, thereby ensuring the safety and feasibility of subsequent approximation routes.

[0066] S402. Using the actual location of the target vehicle as the final target point, within the feasible flight space of the UAV, a path search algorithm is used to determine the approximation route of the UAV from its current flight position to the final target point.

[0067] In one possible implementation, in a gridded model that constructs a feasible flight space based on the UAV's current position, flight altitude limits, and obstacle distribution, a feasible path is calculated using a path search algorithm, starting from the current flight position and ending at the final target point. Based on the feasible path, the trajectory is smoothed to obtain an approximation route suitable for UAV flight control.

[0068] It should be noted that path search needs to take into account both global optimization and local safety. Therefore, a hierarchical planning strategy can be adopted: first obtain a low-resolution global path on a coarse grid, and then refine the local segments on a high-resolution local grid.

[0069] As an example, in this embodiment of the application, with a camera pitch angle of -90 degrees (vertically downwards), a height of 120 meters, and a focal length of 2x, a target segmentation algorithm is used to obtain a highway mask image. The maximum bounding rectangle of the highway is then obtained from the mask image. The angle values ​​between the long and short sides of the rectangle and the camera image are sent to the system to adjust the camera's yaw angle. Figure 5 As shown: Given that the long and short sides of the camera are fixed, the angle between the long side of the highway image and the long side of the camera is obtained, with the relevant angle being [-90, 90], which is converted to [0, 180]. After obtaining the angle, the flight control module adjusts it to finally obtain the correspondence between the long side of the camera and the long side of the highway. The highway mask image point set obtained after the target segmentation algorithm is then... The minimum bounding rectangle of the rotation is obtained using OpenCV's minAreaRect(P). The four vertices of the rectangle are, in order: Define the direction vector of the longer side of adjacent points. and the direction angle of the longer side Then the included angle between the longer and shorter sides can be calculated. The direction angle of the longer side is given. The minimum angle between the long side and the horizontal axis is given. The final required rotation angle is sent to the system and the angle is adjusted so that the long and short sides of the image correspond.

[0070] It should also be noted that the drone's yaw angle relative to the highway is calculated in real time, and the drone's control commands are used to adjust its flight direction. If the current yaw angle is less than 90 degrees, it is adjusted to a negative value and added to the reference angle; if the yaw angle is greater than or equal to 90 degrees, it is converted to 180 minus the yaw angle plus the reference angle. This design ensures the accuracy and efficiency of yaw angle adjustment. To avoid the yaw angle exceeding the effective range, this invention further normalizes the adjusted yaw angle. If the yaw angle is greater than 180 degrees, 360 degrees are subtracted; if the yaw angle is less than -180 degrees, 360 degrees are added. This step ensures that the yaw angle is always within the effective range of [-180, 180] degrees, avoiding control errors caused by angle overflow. Based on the adjusted yaw angle, the yaw angle of the drone relative to true north is further calculated, providing accurate azimuth reference for the drone's navigation and positioning. This method, through geometric relationships and algorithm optimization, enables rapid calculation and adjustment of the yaw angle of UAVs, significantly improving their target search efficiency in linear areas such as highways.

[0071] Based on the above steps, this step combines global and local planning strategies and performs path smoothing and dynamic constraint correction to generate an approach route that is both safe and conducive to visual evidence collection, thereby improving the ability of UAVs to reach the preset altitude above the target on time and reliably in complex environments.

[0072] S403. Control the UAV to fly to a preset altitude above the final target point based on the approach route from the current flight position to the final target point and hover.

[0073] In one possible implementation, the desired pose sequence of the UAV is generated according to the path points of the approximation route; the speed, acceleration and attitude angle of the UAV are controlled based on the desired pose sequence to drive the UAV to fly along the approximation route; during the approximation process, the altitude is adjusted according to the real-time altitude information of the UAV to make the UAV gradually approach the preset altitude; when both the position error and altitude error are less than the preset threshold, the UAV is controlled to enter the hovering mode and stabilize at the preset altitude above the final target point.

[0074] It should be noted that after the drone hovers at the designated location, it is necessary to verify the correspondence between the aircraft's yaw angle and the long and short sides of the highway in the visual image to expand the search range.

[0075] Based on the above steps, this step combines path tracking control with visual servo closed-loop control and implements a gradual convergence strategy of speed and altitude during the approach phase to ensure that the UAV can smoothly reach and hover stably above the target, thereby providing the necessary space and time stability for high-quality evidence collection and safe handling.

[0076] S404. In the hovering state, perform the corresponding task operation according to the event type, including evidence collection for violations, warning broadcast, or information feedback.

[0077] In one possible implementation, the system automatically selects the appropriate task combination based on the event category from the alarm information or detection module; during execution, it controls the camera pan-tilt to adjust the angle and focus to obtain clear evidence, triggers the OCR submodule to process the license plate image, and encapsulates the image, video, location information and timestamp into an evidence file, which is then reported to the cloud management platform or local emergency command center in real time through the communication link.

[0078] It should be noted that the execution of the mission should strictly record metadata to ensure the integrity and traceability of the chain of evidence; in addition, for shouts or warnings that may provoke emotional reactions from ground personnel, compliance policies should be set for volume, language and broadcast duration, and broadcast logs should be recorded in real time.

[0079] As an example, when the event type is determined to be illegal parking and the license plate is successfully recognized by OCR, the system will automatically generate an evidence package containing the license plate number, timestamp, latitude and longitude, drone ID and shooting evidence, and upload the package to the traffic management platform through an encrypted channel. At the same time, 10 high-resolution photos will be continuously taken on the local gimbal as redundant evidence.

[0080] Based on the above steps, this step combines hovering accuracy with task template operation to achieve automated, high-quality evidence collection and immediate handling, which not only meets the evidence chain requirements of law enforcement records, but also improves on-site handling efficiency and the ability to intervene in the public in a timely manner.

[0081] This application constructs a feasible flight space based on the actual location of the target vehicle and the real-time flight position of the UAV, enabling the UAV to effectively avoid obstacles and non-restricted areas in dynamic traffic environments. Subsequently, using the actual location of the target vehicle as the final target point, a path search algorithm is employed to plan an approximation route, ensuring that the UAV can obtain a convergent, smooth, and real-time executable approximation trajectory even in complex road environments. After approximation, the UAV hovers at a preset height above the target point according to the planned route, achieving stable visual overhead coverage of the target vehicle, thus improving the accuracy and reliability of evidence collection, warnings, or information transmission. Finally, by triggering matching task operations based on event types, the UAV can automatically perform evidence collection and photography, voice warnings, or data transmission according to the violation, forming a rapid-response, highly automated traffic violation handling chain. This not only improves the safety, accuracy, and autonomy of UAV task execution but also enhances the real-time nature and enforcement efficiency of violation handling, achieving a more intelligent and efficient traffic response capability.

[0082] The above primarily describes the solutions of the embodiments of this application from the perspective of device implementation. It is understood that each device, such as a drone-based traffic violation processing system, includes at least one of the hardware structures and software modules corresponding to the execution of each function in order to achieve the aforementioned functions. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] This application embodiment can divide the UAV-based traffic violation processing system into functional units according to the above method example. For example, each function can be divided into separate functional units, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0084] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0085] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0086] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A traffic incident handling method based on UAV visual servoing, characterized in that, include: Acquire image and video data of the target area; Based on the image data and video data of the target area, the pixel position of the target vehicle is calculated, and the relative position between the UAV and the target vehicle is calculated; the relative position includes the horizontal distance and the deviation angle. Based on the relative position and the pixel position, calculate the actual position of the target vehicle; Using the actual location of the target vehicle as the final target point, plan the drone approach route and control the drone to hover at a preset height above the final target point to perform the task operation corresponding to the event type.

2. The method for handling traffic violations based on unmanned aerial vehicles according to claim 1, characterized in that, Before acquiring image and video data of the target area, the drone-based traffic violation handling method further includes: Obtain traffic anomaly warning information; Based on the traffic anomaly warning information, the optimal route for the UAV to the initial target airspace is planned by combining optimal path search and local obstacle avoidance strategies. Based on the optimal route from the UAV to the initial target airspace, the UAV is dispatched to the initial target airspace, and image and video data are collected from the target area in the initial target airspace.

3. The method for handling traffic violations based on unmanned aerial vehicles according to claim 1, characterized in that, The step of calculating the pixel position of the target vehicle and the relative position of the drone and the target vehicle based on the image data and video data of the target area includes: By performing target vehicle detection on image and video data, the location region of the target vehicle in the image is identified; Based on the location region, the feature point coordinates of the target vehicle are extracted, and the pixel position of the target vehicle is calculated. Acquire the drone's real-time flight altitude, attitude angles, and camera intrinsic and extrinsic parameters; Based on the pixel position, the camera's intrinsic and extrinsic parameters, the UAV's real-time flight altitude, and the attitude angle, the horizontal distance and deviation angle between the UAV and the target vehicle are calculated through spatial geometric back projection.

4. The method for handling traffic violations based on unmanned aerial vehicles according to claim 3, characterized in that, The step of detecting target vehicles by performing target vehicle detection on the image data and video data, and identifying the location region of the target vehicle in the image, includes: Based on image and video data, a set of mask points for highways is extracted, and the minimum bounding rectangle, the long side of the minimum bounding rectangle, and the short side of the minimum bounding rectangle are obtained using the minimum bounding rectangle algorithm. The main direction of the highway is determined based on the direction of the long side of the minimum circumscribed rotation rectangle, and the camera imaging direction is adjusted to be consistent with the direction of the long side of the minimum circumscribed rotation rectangle. The image is searched for vehicles in the minimum circumscribed rotating rectangle short side field of view mode. If no target vehicle is detected, the camera field of view is rotated 90 degrees to switch to the minimum circumscribed rotating rectangle long side field of view mode to continue the search, obtain candidate vehicle regions, and determine the location region of the target vehicle in the image.

5. The method for handling traffic violations based on unmanned aerial vehicles according to claim 1, characterized in that, The process of planning a drone approach route using the actual location of the target vehicle as the final target point, and controlling the drone to hover at a preset height above the final target point to perform task operations corresponding to the event type includes: Based on the actual location of the target vehicle and the current flight location of the drone, a feasible flight space for the drone is constructed; Using the actual location of the target vehicle as the final target point, a path search algorithm is used in the feasible flight space of the UAV to determine the approach route of the UAV from the current flight position to the final target point; The drone is controlled to fly to a preset altitude above the final target point and hover according to the approximation route from the current flight position to the final target point; While hovering, perform corresponding tasks based on the event type, including collecting evidence of violations, issuing warnings, or transmitting information.

6. The method for handling traffic violations based on unmanned aerial vehicles according to claim 5, characterized in that, The step of using the actual location of the target vehicle as the final target point, and employing a path search algorithm within the feasible flight space of the UAV to determine the approximation route of the UAV from its current flight position to the final target point, includes: A gridded model of the feasible flight space is constructed based on the current location of the UAV, flight altitude limitations, and obstacle distribution. In the rasterized model, a feasible path is calculated using a path search algorithm, starting from the current flight position and ending at the final target point. Based on the feasible path, the trajectory is smoothed to obtain an approximation route suitable for UAV flight control.

7. The method for handling traffic violations based on unmanned aerial vehicles according to claim 5, characterized in that, The step of controlling the drone to hover at a preset altitude above the final target point according to the approximation route includes: Generate the desired pose sequence of the UAV based on the waypoints of the approximation route; The speed, acceleration, and attitude angle of the UAV are controlled based on the desired pose sequence, driving the UAV to fly along the approximation route; During the approach, the drone's altitude is adjusted based on its real-time altitude information, allowing the drone to gradually approach the preset altitude. When both the position error and altitude error are less than the preset threshold, the drone is controlled to enter hovering mode and stabilize at the preset altitude above the final target point.

8. A traffic incident handling system based on UAV visual servoing, characterized in that, include: Image acquisition module, target detection module, position calculation module, path planning module, and control module; The image acquisition module is used to acquire image data and video data of the target area; The target detection module is used to identify the location region of the target vehicle in the image based on the image data and video data; The position calculation module is used to determine the pixel position of the target vehicle based on the position area of ​​the target vehicle in the image, and then determine the actual position of the target vehicle. The path planning module is used to plan the approach route of the UAV using the actual position of the target vehicle as the final target point. The control module is used to control the UAV to fly along the approach route, hover at a preset height above the final target point, and perform task operations corresponding to the event type.

9. The traffic violation processing system based on unmanned aerial vehicles according to claim 8, characterized in that, The location calculation module includes: a relative location calculation module and an actual location calculation module; The relative position calculation module is used to determine the pixel position of the target vehicle based on the target vehicle's position area, and to calculate the relative position between the UAV and the target vehicle, wherein the relative position includes horizontal distance and deviation angle. The actual position calculation module is used to calculate the actual position of the target vehicle based on the pixel position and the relative position.

10. The traffic violation processing system based on unmanned aerial vehicles according to claim 8, characterized in that, The control module includes: a flight control module and a mission execution module; The flight control module is used to control the UAV to fly along the approach route and hover at a preset height above the final target point; The task execution module is used to perform tasks such as collecting traffic violation evidence, issuing warnings, or transmitting information based on the event type.

Images