A method and system for forensics based on unmanned aerial vehicle video stream and a control center

Abstract

This application provides a method, system, and control center for evidence collection based on UAV video streams, relating to the field of traffic engineering technology. The evidence collection method includes: acquiring a target video frame containing a target vehicle; acquiring the target time of the target video frame; and the first pose of the target vehicle in the target video frame; acquiring the position of the UAV and the gimbal attitude at the target time, as the first position and the target gimbal attitude; projecting the first pose onto a geographic coordinate system based on the target video frame, the first position, and the target gimbal attitude to obtain the second pose of the target vehicle in the geographic coordinate system; calculating candidate shooting positions based on the second pose, the first position, and the target gimbal attitude; controlling the UAV to fly to the candidate shooting position and take a picture towards the location of the target vehicle to obtain an evidence photo; performing license plate recognition on the evidence photo to obtain a recognition result; and outputting the second pose, the evidence photo, and the recognition result, thereby improving the efficiency of evidence collection.

Description

Technical Field

[0001] This application relates to the field of traffic engineering technology, and in particular to a method, system, and control center for obtaining evidence based on unmanned aerial vehicle (UAV) video streams. Background Technology

[0002] With the continuous growth of urban motor vehicle ownership, problems in urban traffic management, such as minor traffic accidents and illegal parking, are becoming increasingly prominent. Minor traffic accidents without injuries, such as scrapes and rear-end collisions, account for more than 70% of all traffic accidents. According to the "quick handling and quick compensation" policy, the parties involved should take photos and quickly leave the scene. However, in reality, they often fail to leave the scene in time due to reasons such as not knowing how to take photos for evidence, waiting for traffic police to arrive, or arguing about responsibility, thus causing traffic congestion. Vehicles illegally parked, occupying lanes and blocking fire lanes, are also common problems. Ground enforcement forces cannot cover all road sections, especially areas around schools, temporary parking areas in commercial districts, and back streets and alleys. Therefore, drones, due to their maneuverability and wide field of vision, are gradually being introduced into the field of traffic management. Combined with artificial intelligence (AI) target detection technology, drone patrols can detect illegally parked or accident vehicles.

[0003] In related technologies, inspection drones transmit video streams back to the cloud, where an AI system performs target detection. When an illegally parked or accident-damaged vehicle is detected, the drone operator is notified. The operator then manually controls the drone to fly near the vehicle to take photos as evidence. However, there is a certain delay in cloud-based video stream processing; the drone may have already passed the target area by the time the operator responds. Furthermore, the reliance on manual operation for photo collection results in low efficiency. Summary of the Invention

[0004] The purpose of this application is to provide a method, system, and control center for evidence collection based on drone video streams, so as to achieve end-to-end autonomous evidence collection, including vehicle positioning, autonomous drone flight, photography, and license plate recognition, thereby improving evidence collection efficiency. The specific technical solution is as follows:

[0005] A first aspect of this application provides a method for forensic investigation based on drone video streams, the method comprising:

[0006] Acquire a target video frame containing the target vehicle, capture the target time of the target video frame, and the first pose of the target vehicle in the target video frame;

[0007] The position and gimbal attitude of the UAV at the target time are obtained as the first position and the target gimbal attitude;

[0008] Based on the target video frame, the first position, and the target gimbal attitude, the first pose is projected onto the geographic coordinate system to obtain the second pose of the target vehicle in the geographic coordinate system.

[0009] Based on the second pose, the first position, and the target gimbal attitude, calculate the candidate shooting position;

[0010] The drone is controlled to fly to the candidate shooting location and take pictures towards the location of the target vehicle to obtain evidence photos;

[0011] The license plate was identified by performing license plate recognition on the evidence photos, and the recognition result was obtained;

[0012] Output the second pose, the evidence photo, and the recognition result.

[0013] In one possible embodiment, if the absolute value of the difference between the pitch angle and negative 90° of the target gimbal attitude does not exceed a preset angle, the step of projecting the first pose onto a geographic coordinate system based on the target video frame, the first position, and the target gimbal attitude to obtain the second pose of the target vehicle in the geographic coordinate system includes:

[0014] The ground sampling distance is calculated based on the screen width and height of the target video frame, the diagonal field of view of the UAV camera, and the flight altitude of the UAV.

[0015] Based on the screen width and screen height of the target video frame, the ground sampling distance, and the pixel coordinates in the first pose, calculate the first offset value of the camera coordinate system of the UAV camera relative to the ground coordinate system;

[0016] Based on the first offset value and the heading angle of the target gimbal attitude, calculate the second offset value of the UAV camera's camera coordinate system relative to the geographic coordinate system;

[0017] Based on the first position and the second offset value, the second position of the target vehicle in the geographic coordinate system is calculated;

[0018] The vehicle orientation of the target vehicle is estimated based on the aspect ratio of the target vehicle in the target video frame and the heading angle of the target gimbal attitude.

[0019] In one possible embodiment, when the absolute value of the difference between the pitch angle and negative 90° of the target gimbal attitude exceeds a preset angle, the step of projecting the first pose onto a geographic coordinate system based on the target video frame, the first position, and the target gimbal attitude to obtain the second pose of the target vehicle in the geographic coordinate system includes:

[0020] Based on the pixel coordinates in the first pose, the screen width and screen height of the target video frame, and the pixel focal length of the UAV camera, the first ray direction of the UAV camera in the camera coordinate system is calculated; the pixel focal length is determined based on the screen width and screen height of the target video frame and the field of view of the UAV camera, and is used to characterize the value of the equivalent focal length of the UAV camera on the imaging plane in pixels.

[0021] Transform the direction of the first ray to the East-North-Sky coordinate system to obtain the direction of the second ray in the East-North-Sky coordinate system;

[0022] Calculate the coordinates of the intersection point between the second ray direction and the ground;

[0023] Based on the first position and the coordinates of the intersection point, calculate the second position of the target vehicle in the geographic coordinate system;

[0024] The vehicle orientation of the target vehicle is estimated based on the aspect ratio of the target vehicle in the target video frame and the heading angle of the target gimbal attitude.

[0025] In one possible embodiment, calculating the candidate shooting position based on the second pose, the first position, and the target gimbal pose includes:

[0026] Based on the second position in the second pose and the flight altitude of the UAV, a first candidate shooting position is obtained; the first candidate shooting position is located at the flight altitude directly above the second position;

[0027] The position where the target vehicle's orientation is offset by a preset distance is used as the second candidate shooting position;

[0028] The position where the target vehicle is offset by a preset distance in the opposite direction is taken as the third candidate shooting position.

[0029] In one possible embodiment, there are multiple candidate shooting locations; controlling the drone to fly to the candidate shooting location includes:

[0030] Perform the following steps periodically:

[0031] Obtain the third position and heading angle of the UAV at the current moment;

[0032] Calculate the positional distance between the third position and the target candidate shooting position, and calculate the azimuth angle of the UAV; the target candidate shooting position is the candidate shooting position closest to the UAV.

[0033] Calculate the eastward distance and the northward distance based on the location distance and the azimuth angle, respectively;

[0034] Calculate the altitude difference between the current flight altitude of the UAV and the altitude of the target in the candidate shooting position, and calculate the heading error between the heading angle at the current moment and the heading angle in the candidate shooting angle;

[0035] Calculate the flight speed of the UAV in the North-East-Down coordinate system based on the eastward distance, the northward distance, and the altitude difference;

[0036] Based on the heading angle at the current moment, the flight speed of the UAV in the North-East-Down coordinate system is converted to the UAV's body coordinate system, and the heading angle of the UAV is determined based on the heading error;

[0037] Control the UAV to fly at the stated flight speed and the determined heading angle until the cycle ends;

[0038] At the end of the cycle, if the position distance, the altitude difference, and the heading error do not meet their respective preset conditions, the next cycle begins until the position distance, the altitude difference, and the heading error all meet their respective preset conditions.

[0039] In one possible embodiment, the method further includes:

[0040] If the drone's flight time exceeds a first preset time, the drone will be controlled to stop flying and take pictures of the location of the target vehicle.

[0041] In one possible embodiment, the method further includes:

[0042] If the identification result is a failure, switch the target candidate shooting position and execute the step of controlling the drone to fly to the target candidate shooting position and take pictures towards the location of the target vehicle to obtain evidence photos.

[0043] In one possible embodiment, the method further includes:

[0044] If the celestial component of the second ray direction in the East-North-Sky coordinate system is not less than a preset value, license plate recognition is performed on the target video frame to obtain the recognition result.

[0045] Output the target video frame and the recognition result;

[0046] And / or,

[0047] If the drone's photo-taking time exceeds the second preset time, license plate recognition is performed on the target video frame to obtain the recognition result;

[0048] Output the second pose, the target video frame, and the recognition result.

[0049] In a second aspect of this application, an evidence collection system based on drone video stream is also provided, the evidence collection system comprising: a control center and a drone; the control center comprising a main inference system and a traffic intelligent agent;

[0050] The main inference system is used to acquire real-time video streams from different drones and perform target detection on each real-time video stream. When an event requiring evidence collection is detected in a real-time video stream from a drone, an evidence collection task is generated and sent to the traffic intelligent agent.

[0051] The traffic intelligent agent is used to perform any of the methods described above.

[0052] In a third aspect of this application, a control center is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0053] Memory, used to store computer programs;

[0054] A processor, when executing a program stored in memory, implements any of the methods described above.

[0055] Beneficial effects of the embodiments in this application:

[0056] This application provides a method, system, and control center for evidence collection based on UAV video streams. It acquires a target video frame containing the target vehicle, the target time of the acquired video frame, the first pose of the target vehicle within the video frame, and the position of the UAV and the gimbal attitude at the target time as the first position and target gimbal attitude. Based on the target video frame, the first position, and the target gimbal attitude, the first pose is projected onto a geographic coordinate system to obtain the second pose of the target vehicle in the geographic coordinate system. Then, based on the second pose, the first position, and the target gimbal attitude, candidate shooting positions are calculated. The UAV is controlled to fly to the candidate shooting position and take a picture towards the location of the target vehicle to obtain an evidence photo. License plate recognition is performed on the evidence photo to obtain the recognition result. The second pose, the evidence photo, and the recognition result are output. Through autonomous vehicle positioning, autonomous control of UAV flight, photography, and license plate recognition, the entire chain of autonomous evidence collection is achieved without relying on manual operation, improving the efficiency of evidence collection.

[0057] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0059] Figure 1 A flowchart illustrating a method for obtaining evidence based on UAV video streams provided in an embodiment of this application;

[0060] Figure 2 This is a flowchart illustrating an implementation method for locating a target vehicle according to an embodiment of this application.

[0061] Figure 3 This is another schematic flowchart illustrating an implementation method for locating a target vehicle provided in this application embodiment;

[0062] Figure 4 A flowchart illustrating an implementation method for calculating candidate shooting positions provided in an embodiment of this application;

[0063] Figure 5 A flowchart illustrating an implementation method for controlling a drone to fly to a candidate shooting location, as provided in this application embodiment;

[0064] Figure 6 This is a schematic diagram illustrating the control of a drone to a target candidate shooting location provided in an embodiment of this application;

[0065] Figure 7 A schematic diagram illustrating a method for obtaining evidence based on UAV video streams provided in an embodiment of this application;

[0066] Figure 8 Another schematic diagram of the forensic method based on UAV video stream provided in the embodiments of this application.

[0067] Figure 9 A schematic diagram of the structure of an evidence collection system based on UAV video stream provided in an embodiment of this application;

[0068] Figure 10 A schematic diagram of the control center provided in an embodiment of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0070] With the continuous growth of urban motor vehicle ownership, problems in urban traffic management, such as minor traffic accidents and illegal parking, are becoming increasingly prominent. Drones, due to their maneuverability and wide field of view, are gradually being introduced into the field of traffic management. Combined with AI target detection technology, drones can detect illegally parked or accident-damaged vehicles during inspections. Among related technologies, the main methods of evidence collection for illegally parked or accident-damaged vehicles include:

[0071] 1. Manual Remote-Controlled Photography Method. This method involves a drone pilot flying the drone over the target vehicle (illegally parked or involved in an accident), manually adjusting the drone's gimbal attitude and camera focus, taking a photo of the vehicle, and then manually identifying the license plate information. In this method, drone photography for evidence collection relies entirely on the pilot's experience. Photographing a single vehicle takes 3-5 minutes and requires a certified pilot to operate the drone throughout the process, resulting in high labor costs. For minor traffic accidents, the time waiting for the pilot to arrive often exceeds the accident handling time, negating the purpose of rapid handling of minor traffic accidents.

[0072] 2. Fixed-route patrol photography. This method involves pre-setting a fixed route, along which the drone flies and takes photos at regular intervals. Afterwards, manual or AI systems identify the license plates of illegally parked vehicles and vehicles involved in accidents in batches. However, this method cannot provide close-up evidence of real-time illegal parking or sudden accidents, and the shooting angle and distance are uncontrollable, resulting in a low license plate recognition rate.

[0073] 3. Cloud-based AI detection + manual evidence collection. In this method, the drone transmits video streams to the cloud, where an AI system detects targets. When an illegally parked or accident-damaged vehicle is detected, the drone operator is notified, and then manually controls the drone to fly to the vicinity of the vehicle to take photos. In this method, there is a delay of several seconds in cloud-based video stream processing; by the time the operator responds, the drone may have already passed the target area, and the photo collection relies on manual operation.

[0074] 4. Traffic Management System Based on Fixed Cameras. This method uses fixed cameras installed at intersections or road sections to capture video footage. AI is used to identify illegally parked vehicles and automatically capture photos to recognize license plates, or to detect traffic accidents and notify traffic police. However, the coverage of fixed cameras at intersections or road sections is limited, failing to cover areas with high rates of temporary illegal parking and accident-prone sections of non-main roads. Furthermore, installation and maintenance costs are high.

[0075] However, all of the above methods of evidence collection suffer from a gap between the discovery of the target vehicle and the acquisition of a clear license plate photo, and have low efficiency.

[0076] Therefore, this application provides a method, system, and control center for evidence collection based on drone video streams, which is used to collect evidence of accident vehicles and illegally parked vehicles in scenarios such as minor traffic accidents and illegal parking. Through autonomous vehicle positioning, autonomous control of drone flight, photography, and license plate recognition, the entire chain of autonomous evidence collection is achieved without relying on manual operation, thus improving the efficiency of evidence collection.

[0077] One possible embodiment, see Figure 1 The evidence collection method based on drone video streams can be implemented by a traffic intelligent agent in the control center. This method may include:

[0078] Step S101: Acquire the target video frame containing the target vehicle, the target time of the target video frame, and the first pose of the target vehicle in the target video frame; Step S102: Acquire the position of the UAV and the gimbal attitude at the target time, as the first position and the target gimbal attitude; Step S103: Project the first pose onto the geographic coordinate system based on the target video frame, the first position, and the target gimbal attitude to obtain the second pose of the target vehicle in the geographic coordinate system; Step S104: Calculate the candidate shooting position based on the second pose, the first position, and the target gimbal attitude; Step S105: Control the UAV to fly to the candidate shooting position and take a picture towards the location of the target vehicle to obtain an evidence photo; Step S106: Perform license plate recognition on the evidence photo to obtain the recognition result; Step S107: Output the second pose, the evidence photo, and the recognition result. This embodiment achieves end-to-end autonomous evidence collection through autonomous vehicle positioning, autonomous UAV flight control, photography, and license plate recognition, realizing evidence collection without relying on manual operation and improving evidence collection efficiency.

[0079] In step S101 of this embodiment, the traffic intelligent agent can maintain a task queue containing multiple evidence collection tasks. These tasks can be generated and sent by the main inference system of the control center when it detects a real-time video stream uploaded by different drones and identifies an event requiring evidence collection within that stream. This event could be, for example, a vehicle illegally parked within a parking fence or a traffic accident. Alternatively, tasks can be manually sent by front-end personnel such as traffic management personnel. For example, the task queue can be a bounded queue, and its length can be configured according to actual conditions, such as 10, 20, or 30 characters.

[0080] The acquired target video frame containing the target vehicle, the target time of acquisition of the target video frame, and the first pose of the target vehicle in the target video frame can be included in an evidence collection task. The target video frame is the video frame in which an event requiring evidence collection is detected. The target vehicle can be an illegally parked vehicle or an accident vehicle. The first pose of the target vehicle in the target video frame can include the pixel coordinates of the target vehicle in the target video frame, such as the pixel coordinates of the detection box of the target vehicle in the target video frame.

[0081] In step S102 of this embodiment, the real-time uploaded location and gimbal attitude of the drone can be received to obtain the location and gimbal attitude of the drone at the target time when it captures the target video frame, which are used as the first location and target gimbal attitude. When there are multiple drones, the evidence collection task may also include drone identifiers. Then, the location and gimbal attitude of the drone with the same identifier as the drone in the evidence collection task at the target time when it captures the target video frame are obtained, which are used as the first location and target gimbal attitude. The first location may be, for example, GPS (Global Positioning System) latitude and longitude, altitude, etc.

[0082] For example, the traffic intelligent agent can deduplicate any two adjacent evidence collection tasks received. For instance, it can skip evidence collection tasks where the time difference between the target moments of the acquired target video frames is less than a preset time difference, and the distance between the drone positions at the target moments is less than a set distance. This prevents duplicate evidence collection for tasks where the time difference between target moments is less than the preset time difference and the distance between the drone positions is less than the set distance. The preset time difference and set distance can be configured as needed or according to actual conditions, such as configuring the preset time difference to 60 seconds (s) and the set distance to 5 meters.

[0083] In step S103 of this embodiment, the pixel coordinates of the target vehicle in the target video frame are projected onto the geographic coordinate system according to the screen width and screen height of the target video frame, the first position of the UAV and the attitude of the target gimbal to locate the target vehicle and obtain the second pose of the target vehicle in the geographic coordinate system. The specific implementation process is described in detail below.

[0084] In step S104 of this embodiment, the candidate shooting position is calculated based on the second position in the second pose of the target vehicle, the first position of the UAV, and the attitude of the target gimbal.

[0085] In step S105 of this embodiment, for the calculated candidate shooting positions, the proportional-integral-derivative (PID) control method can be used to control the drone to fly to the candidate shooting position and take pictures towards the location of the target vehicle, thereby obtaining evidence photos.

[0086] For example, there are multiple candidate shooting locations, and the drone can be controlled to fly to each candidate shooting location to take pictures and obtain multiple evidence photos.

[0087] In step S106 of this embodiment, license plate recognition is performed on the acquired evidence photos to obtain the recognition result. For example, when there are multiple evidence photos, the clear, frontal photo is used first for license plate recognition.

[0088] In step S107 of this embodiment, the evidence collection results, such as the second pose obtained by locating the target vehicle, the acquired evidence photos, and the recognition results of the license plate recognition, are output, for example, reported to the accident handling platform, so as to facilitate rapid accident handling using the obtained evidence collection results.

[0089] One possible embodiment is that the drone's video shooting mode can include top-down mode and oblique-view mode. For example, if the absolute value of the difference between the drone's target gimbal pitch angle and -90° does not exceed a preset angle, the shooting mode when the drone captures the target video frame is determined to be a top-down mode; if the absolute value of the difference between the drone's target gimbal pitch angle and -90° exceeds a preset angle, the shooting mode when the drone captures the target video frame is determined to be an oblique-view mode. The top-down mode is suitable for scenarios where the drone's gimbal is nearly vertically downward, while the oblique-view mode is suitable for scenarios where the drone's gimbal is not vertically downward, such as oblique-view inspection scenarios. For example, the preset angle can be 10°, 15°, or 20°, etc.

[0090] For example, if the preset angle is 15° and the pitch angle of the target gimbal attitude is represented as pitch, then when |pitch - (-90°)| ≤ 15°, the shooting mode when the drone captures the target video frame is determined to be the top-down mode; otherwise, it is the oblique-view mode.

[0091] One possible embodiment, see Figure 2 When it is determined that the shooting mode when the drone captures the target video frame is a top-down view, the process of locating the target vehicle can include:

[0092] Step S201: Calculate the ground sampling distance based on the screen width and height of the target video frame, the diagonal field of view of the UAV camera, and the flight altitude of the UAV. Step S202: Calculate the first offset value of the UAV camera's camera coordinate system relative to the ground coordinate system based on the screen width and height of the target video frame, the ground sampling distance, and the pixel coordinates in the first pose. Step S203: Calculate the second offset value of the UAV camera's camera coordinate system relative to the geographic coordinate system based on the first offset value and the heading angle of the target gimbal attitude. Step S204: Calculate the second position of the target vehicle in the geographic coordinate system based on the first position and the second offset value. Step S205: Estimate the vehicle's orientation based on the aspect ratio of the target vehicle in the target video frame and the heading angle of the target gimbal attitude. Using this embodiment, there is no need to calibrate ground control points. The target vehicle can be located by projecting its first pose in the target video frame onto the geographic coordinate system using only the position and attitude data of the UAV itself. This allows the UAV to automatically plan its flight path based on the second pose of the target vehicle obtained from the location. This solves the problem that existing methods cannot convert the pixel coordinates of the vehicle in the video frame into the vehicle's actual GPS latitude and longitude, which leads to the inability to automatically plan the UAV's flight path. This method is suitable for vehicle positioning in dynamic UAV flight scenarios.

[0093] In step S201 of this embodiment, for example, the diagonal field of view (DFOV) of the drone camera can be set to 82°. In actual applications, the diagonal field of view of different drone cameras is different. This embodiment does not specifically limit the diagonal field of view of the drone camera.

[0094] The aspect ratio of the target video frame is calculated based on its width and height. Then, based on the aspect ratio and the drone camera's DFOV, the horizontal field of view (FOV) and vertical FOV of the drone camera are calculated. Finally, the ground sampling distance (GSD) is calculated based on the drone's flight altitude, horizontal and vertical FOVs, and the width and height of the target video frame. The drone's flight altitude can be calculated based on its initial altitude or the relative ground height directly contained in its initial location.

[0095] For example, if the drone's flight altitude is denoted as H, and the target video frame's width and height are 1920×1080, then the aspect ratio of the target video frame is 1920 / 1080. Then, the horizontal FOV is calculated using the expression: Horizontal FOV = 2×arctan(tan(DFOV / 2) ×aspect / sqrt(1+aspect²)); the vertical FOV is calculated using the expression: Vertical FOV = 2×arctan(tan(DFOV / 2) / sqrt(1+aspect²)); and the eastward and northward components of GSD are calculated using the expressions GSD_x = 2×H×tan(Horizontal FOV / 2) / width and GSD_y = 2×H×tan(Vertical FOV / 2) / height. The units of GSD_x and GSD_y are meters per pixel (i.e., meters per pixel).

[0096] In step S202 of this embodiment, the pixel coordinates in the first pose can specifically be the pixel coordinates of the detection box of the target vehicle in the target video frame. For example, the eastward component cam_dx and the northward component cam_dy of the first offset value of the UAV camera's camera coordinate system relative to the ground coordinate system are calculated using the expressions: cam_dx = (px - screen width / 2) × GSD_x and cam_dy = (py - screen height / 2) × GSD_y. Here, (px, py) represents the center pixel coordinates of the detection box of the target vehicle in the target video frame, px represents the horizontal component, and py represents the vertical component. The units of cam_dx and cam_dy are meters. Strictly speaking, the northward component cam_dy is the offset of pixels in the vertical direction. In this direction, the area below the target video frame is represented as a positive value. At this time, the UAV camera's heading angle is 0°, and the northward component cam_dy should be mapped to -cam_dy, i.e., the negative northward component cam_dy.

[0097] In step S203 of this embodiment, for example, the heading angle of the target gimbal attitude is represented as yaw. The north component (north_m) and east component (east_m) of the second offset value of the UAV camera's camera coordinate system relative to the geographic coordinate system are calculated using the expressions: east_m = cam_dx × cos(yaw) + cam_dy × (-sin(yaw)) and north_m = cam_dx × (-sin(yaw)) + cam_dy × (-cos(yaw)). Here, yaw is in radians, with 0 for north and positive for clockwise.

[0098] In step S204 of this embodiment, for example, the longitude of the first position of the UAV is represented as lon and the latitude as lat. The vehicle longitude and vehicle latitude of the target vehicle are calculated by the expressions: vehicle longitude = lon + east_m / (111320 × cos(lat)) and vehicle latitude = lat + north_m / 111320, that is, the second position of the target vehicle in the geographic coordinate system is obtained.

[0099] In step S205 of this embodiment, the aspect ratio of the target vehicle in the target video frame is the aspect ratio of the detection box of the target vehicle in the target video frame. For example, the width of the detection box of the target vehicle in the target video frame is represented by bbox_w, the height by bbox_h, and the yaw angle of the target gimbal attitude is represented by yaw. If bbox_w > bbox_h × 1.3, it means that the target vehicle is horizontal, and the vehicle orientation is set to ≈ (yaw + 90°) mod 360°; if bbox_h > bbox_w × 1.3, it means that the target vehicle is vertical, and the vehicle orientation is set to ≈ yaw mod 360°; otherwise, the vehicle orientation is set to = yaw mod 360°.

[0100] One possible embodiment, see Figure 3 When it is determined that the shooting mode when the drone captures the target video frame is oblique view mode, the process of locating the target vehicle may include:

[0101] Step S301: Calculate the first ray direction of the UAV camera in the camera coordinate system based on the pixel coordinates in the first pose, the screen width and height of the target video frame, and the pixel focal length of the UAV camera; the pixel focal length is determined based on the screen width and height of the target video frame and the field of view of the UAV camera, and is used to characterize the equivalent focal length of the UAV camera on the imaging surface in pixels; Step S302: Transform the first ray direction to the East-North-Sky coordinate system to obtain the second ray direction in the East-North-Sky coordinate system; Step S303: Calculate the coordinates of the intersection point of the second ray direction and the ground; Step S304: Calculate the second position of the target vehicle in the geographic coordinate system based on the first position and the intersection point coordinates; Step S305: Estimate the vehicle orientation of the target vehicle based on the aspect ratio of the target vehicle in the target video frame and the heading angle of the target gimbal attitude. Using this embodiment, there is no need to calibrate ground control points. The target vehicle can be located by projecting its first pose in the target video frame onto the geographic coordinate system using only the position and attitude data of the UAV itself. This allows the UAV to automatically plan its flight path based on the second pose of the target vehicle obtained from the location. This solves the problem that existing methods cannot convert the pixel coordinates of the vehicle in the video frame into the vehicle's actual GPS latitude and longitude, which leads to the inability to automatically plan the UAV's flight path. This method is suitable for vehicle positioning in dynamic UAV flight scenarios.

[0102] In step S301 of this embodiment, the pixel coordinates in the first pose can specifically be the pixel coordinates of the detection box of the target vehicle in the target video frame. For example, the first ray direction ray_cam of the UAV camera in the camera coordinate system is calculated by the expression: ray_cam=[(px-screen width / 2) / fx, (py-screen height / 2) / fy, 1.0]. Where fx=(screen width / 2) / tan(horizontal FOV / 2), fy=(screen height / 2) / tan(vertical FOV / 2), the units of fx and fy are pixels, and fx and fy represent the pixel focal length of the UAV camera in the horizontal and vertical directions, respectively, that is, the equivalent focal length in pixels, that is, pixel focal length = (screen size of the target video frame / 2) / tan(field of view of the UAV camera / 2).

[0103] In step S302 of this embodiment, the coordinate transformation relationship from the UAV camera coordinate system to the East-North-Up (ENU) coordinate system can be calculated first, and then the first ray direction of the UAV camera in the camera coordinate system can be transformed to the ENU coordinate system using the calculated coordinate transformation relationship to obtain the second ray direction in the ENU coordinate system.

[0104] For example, the transformation relationship from the ENU coordinate system to the North-East-Down (NED) coordinate system is set as R_enu2ned=[[0,1,0],[1,0,0],[0,0,-1]]; the rotation transformation relationship from the NED coordinate system to the UAV body is R_body= R_roll× R_pitch×R_yaw, where R_roll is the rotation matrix corresponding to the roll angle, R_pitch is the rotation matrix corresponding to the pitch angle, and R_yaw is the rotation matrix corresponding to the yaw angle; the transformation relationship from the UAV body to the UAV camera coordinate system is R_body2cam= [[0,1,0],[0,0,1],[1,0,0]], and then the coordinate transformation relationship from the ENU coordinate system to the UAV camera coordinate system is R= R_body2cam×R_body×R_enu2ned. Then, since the inverse of an orthogonal matrix is ​​equal to its transpose, the expression ray_enu = transpose of R × ray_cam can be used to transform ray_cam in the camera coordinate system to the ENU coordinate system, thus obtaining the second ray direction ray_enu in the ENU coordinate system.

[0105] In step S303 of this embodiment, the intersection point of the second ray direction with the flight altitude in the vertical direction is first calculated by using the flight altitude of the UAV and the celestial component of the second ray direction ray_enu. Then, the coordinates of the intersection point between the second ray direction and the ground are calculated based on the intersection point and the east and north components of ray_enu.

[0106] For example, assuming the UAV is at the origin of the ENU coordinate system and its flight altitude is H, the ground is at the position up = -H in the celestial component. The celestial component of ray_enu is represented as ray_enu_up, the eastward component as ray_enu_east, and the northward component as ray_enu_north. The intersection of the second ray direction with the flight altitude in the vertical direction is denoted as t. The coordinates (hit_east, hit_north) of the intersection of the second ray direction with the ground can be calculated using the expressions t = -H / ray_enu_up, hit_east = ray_enu_east × t, and hit_north = ray_enu_north × t. Here, hit_east represents the intersection of the second ray direction with the ground in the eastward direction, and hit_north represents the intersection of the second ray direction with the ground in the northward direction.

[0107] In step S304 of this embodiment, for example, the longitude of the UAV's first position is represented as lon and the latitude as lat. The vehicle longitude and vehicle latitude of the target vehicle are calculated by the expressions: vehicle longitude = lon + hit_east / (111320×cos(lat)) and vehicle latitude = lat + hit_north / 111320, which gives the second position of the target vehicle in the geographic coordinate system.

[0108] In step S305 of this embodiment, for example, the width of the detection box of the target vehicle in the target video frame is represented as bbox_w, the height is represented as bbox_h, and the heading angle of the target gimbal attitude is represented as yaw. If bbox_w > bbox_h × 1.3, it means that the target vehicle is horizontal, and the vehicle orientation is set to ≈ (yaw + 90°) mod 360°; if bbox_h > bbox_w × 1.3, it means that the target vehicle is vertical, and the vehicle orientation is set to ≈ yaw mod 360°; otherwise, the vehicle orientation is set to = yaw mod 360°.

[0109] In one possible embodiment, after calculating the second position of the target vehicle in the geographic coordinate system, the second position can be corrected. For example, a fixed offset can be applied to the orientation of the target vehicle relative to the UAV to compensate for systematic deviations. This fixed offset can be calibrated using the actual flight data of the UAV; for example, the eastward offset can be set to ±3 meters, and the northward offset can be set to ±4 meters, etc.

[0110] In one possible embodiment, Figure 1 In the example shown, step S104 includes: obtaining a first candidate shooting position based on the second position in the second pose and the flight altitude of the drone; the first candidate shooting position is located at a flight altitude directly above the second position; the position where the vehicle's orientation is offset by a preset distance is used as the second candidate shooting position; the position where the vehicle's orientation is offset by a preset distance in the opposite direction is used as the third candidate shooting position. Using this embodiment, based on the second pose of the target vehicle in the geographic coordinate system, the positions directly above, in front of, and behind the target vehicle are automatically calculated as candidate shooting positions. This facilitates panoramic liability determination and license plate recognition of the target vehicle using evidence photos taken at the candidate shooting positions, ensuring that at least one angle of the evidence photo can identify the license plate.

[0111] In this embodiment, the latitude and longitude of the second position in the second pose are set to the latitude and longitude of the first candidate shooting position, and the flight altitude of the drone is set to the altitude of the first candidate shooting position, so that the first candidate shooting position is located at the drone's flight altitude directly above the second position, allowing for a panoramic view. Preferably, -90° can also be used as the pitch angle in the first candidate shooting angle corresponding to the first candidate shooting position, so that the drone's gimbal is vertically downward. This allows the drone to record the relative positional relationship of the vehicles involved in minor traffic accidents and the point of collision as a basis for determining liability, as well as the positional relationship between illegally parked vehicles and no-parking markings in illegal parking scenarios. For example, -90° ± a set angle can also be used as the pitch angle in the first candidate shooting angle corresponding to the first candidate shooting position. This set angle can be set according to actual conditions, such as 3°, 5°, or 10°.

[0112] The location where the target vehicle's orientation deviates by a preset distance is used as the second candidate shooting position, specifically the location in front of the target vehicle at a preset distance. This preset distance can be determined based on the UAV's flight altitude H and the target gimbal's pitch angle, for example, calculated using the expression H / tan(|pitch|). This ensures that the target vehicle's front license plate can be photographed at this second candidate shooting position. Preferably, the angle obtained by adding 180° to the target vehicle's orientation angle and then dividing by 360° (i.e., (vehicle orientation + 180°) mod 360°) can also be used as the heading angle in the second candidate shooting angle corresponding to the second candidate shooting position, and the pitch angle of the target gimbal's attitude can be used as the pitch angle in the second candidate shooting angle.

[0113] The position where the target vehicle is offset by a preset distance in the opposite direction is designated as the third candidate shooting position, that is, the position at a preset distance behind the target vehicle is designated as the third candidate shooting position, so that the rear license plate of the target vehicle can be photographed as evidence at this third candidate shooting position. Preferably, the angle of the target vehicle's orientation can also be used as the yaw angle in the third candidate shooting angle corresponding to the third candidate shooting position, and the pitch angle of the target gimbal attitude can be used as the pitch angle in the third candidate shooting angle.

[0114] In one possible embodiment, for each candidate shooting position, the focal length of the drone camera can be automatically adjusted based on the distance between the drone and the target vehicle when the drone is at each candidate shooting position, thereby achieving automatic selection of the drone camera focal length.

[0115] For example, the slant distance between the drone and the target vehicle when the drone is in the candidate shooting position can be calculated first. The slant distance represents the straight-line distance between the drone and the target vehicle when the drone is in the candidate shooting position. When the slant distance is less than 30 meters, the focal length is set to 24 mm wide-angle; when the slant distance is between 30 and 80 meters, the focal length is set to 70 mm medium telephoto; when the slant distance is greater than or equal to 80 meters, the focal length is set to 168 mm medium telephoto.

[0116] For example, see Figure 4 After calculating the target vehicle's second position in the geographic coordinate system (such as the vehicle's GPS position) and the vehicle's orientation, candidate shooting positions are calculated in combination with the drone's flight altitude. The calculated candidate shooting positions can include positions directly above, in front of, and behind the target vehicle. Then, combined with the target vehicle's orientation and the drone's target gimbal attitude, candidate shooting angles corresponding to each candidate shooting position are determined. Finally, the slant distance between the drone and the target vehicle when the drone is at each candidate shooting position is used to determine the focal length of the drone when it is at each candidate shooting position.

[0117] In one possible embodiment, there are multiple candidate shooting locations, see [link to relevant documentation]. Figure 5 The process of controlling a drone to fly to a candidate shooting location may include periodically performing the following steps:

[0118] Step S501: Obtain the third position and heading angle of the UAV at the current moment; Step S502: Calculate the positional distance between the third position and the target candidate shooting position, and calculate the azimuth angle of the UAV; the target candidate shooting position is the candidate shooting position closest to the UAV; Step S503: Calculate the eastward distance and northward distance based on the positional distance and azimuth angle; Step S504: Calculate the altitude difference between the current flight altitude of the UAV and the target altitude in the target candidate shooting position, and calculate the heading error between the current heading angle and the heading angle in the target candidate shooting angle; Step S505: Based on the eastward distance and northward distance, calculate the heading error between the current flight altitude and the target altitude in the target candidate shooting position; Calculate the UAV's flight speed in the North-East-Down coordinate system based on the heading angle, northward distance, and altitude difference; Step S506: Convert the UAV's flight speed in the North-East-Down coordinate system to the UAV's body coordinate system based on the current heading angle, and determine the UAV's heading angle based on the heading error; Step S507: Control the UAV to fly according to the flight speed and the determined UAV heading angle until the cycle ends; At the end of the cycle, if the position distance, altitude difference, and heading error do not meet their respective preset conditions, start the next cycle until the position distance, altitude difference, and heading error all meet their respective preset conditions. In this embodiment, the drone's flight speed is calculated and its heading angle is determined by periodically combining the drone's current third position and heading angle with the target candidate shooting position. The drone is then controlled to fly at the determined flight speed and heading angle until the preset conditions are met. In the process of calculating the drone's flight speed, the transformation from the north-east-down coordinate system to the drone's body coordinate system is realized. This ensures that the drone can correctly fly to the target candidate shooting position to take pictures and collect evidence, regardless of which direction it is facing.

[0119] In this embodiment, the period for calculating the drone's flight speed and determining its heading angle can be set as needed. For example, the period can be set to be consistent with the drone's GPS update frequency of 500 milliseconds (ms).

[0120] In step S501 of this embodiment, the third position of the UAV at the current moment may include latitude and longitude, altitude and flight altitude at the current moment, or the third position may include latitude and longitude and altitude, and then the flight altitude of the UAV at the current moment may be calculated from the altitude. All of these are feasible.

[0121] In step S502 of this embodiment, the target candidate shooting position is the candidate shooting position closest to the drone, which minimizes the drone's flight distance. Preferably, when the target candidate shooting position is directly above the target vehicle, the license plate cannot be identified in the photo taken at that position, so that position can be skipped, and the next closest candidate shooting position to the drone can be selected as the target candidate shooting position.

[0122] For example, the positional distance between the third position and the target candidate shooting position can be calculated using the Haversine formula, which is used to calculate the distance between two points on a sphere. For example, the positional distance dist = Haversine(third position, target candidate shooting position), and the azimuth angle (bearing) of the UAV can be calculated using the third position and the target candidate shooting position.

[0123] In step S503 of this embodiment, for example, the eastward distance error_east and the northward distance error_north are calculated respectively using the expressions error_north=dist×cos(bearing) and error_east=dist×sin(bearing).

[0124] In step S504 of this embodiment, for example, the altitude difference error_alt between the current flight altitude of the UAV and the target altitude in the target candidate shooting position is calculated by the expression error_alt = target altitude in the target candidate shooting position - the current flight altitude of the UAV. The heading error error_yaw between the heading angle at the current moment and the heading angle in the target candidate shooting position is calculated by the expression error_yaw = normalize(heading angle in the target candidate shooting angle - heading angle at the current moment). At the same time, error_yaw is normalized to [-180°, 180°].

[0125] In step S505 of this embodiment, three independent PID controllers can be used to control and calculate the three velocity components—northward, eastward, and altitude—to calculate the UAV's flight speed in the north-east-down coordinate system. For example, the proportional coefficient KP1 = 0.8, integral coefficient KI1 = 0.05, and derivative coefficient KD1 = 0.3 in the PID controller controlling the northward / eastward components, with an output range of [-8, 8] m / s; the proportional coefficient KP2 = 0.5, integral coefficient KI2 = 0.02, and derivative coefficient KD2 = 0.2 in the PID controller controlling the altitude component, with an output range of [-2, 2] m / s; and the heading component is set to proportional control: yaw_cmd = 2.0 × error_yaw, with a range of [-50, 50] ° / s.

[0126] Then, based on the calculated eastward distance `error_east`, northward distance `error_north`, and altitude difference `error_alt`, the three independent PID controllers are used to calculate the northward velocity component `vel_north`, the eastward velocity component `vel_east`, and the altitude velocity component `vel_up` in the north-east-down coordinate system, respectively, as the three components of the UAV's flight speed. For example, `vel_north`, `vel_east`, and `vel_up` can be calculated using the following expressions: `vel_north = PID_north.update(error_north, dt = 0.5)`, `vel_east = PID_east.update(error_east, dt = 0.5)`, `vel_up = PID_alt.update(error_alt, dt = 0.5)`, where `PID_north.update` is the PID controller controlling the northward velocity component, `PID_east.update` is the PID controller controlling the eastward velocity component, and `PID_alt.update` is the PID controller controlling the altitude velocity component. Among them, the PID controller for each component can be set with an integral limit, for example, by setting the integral term integral=clamp(integral,-output_max / KI,output_max / KI), where output_max represents the maximum output value and KI is the integral coefficient.

[0127] In step S506 of this embodiment, exemplarily, the heading angle at the current moment is represented as Yaw. The UAV's flight speed in the North-East-Down coordinate system can be converted to the UAV's body coordinate system using the following expressions, resulting in virtual joystick commands in the body coordinate system: forward pitch angle pitch_cmd = vel_north × cos(Yaw) + vel_east × sin(Yaw), right roll angle roll_cmd = -vel_north × sin(Yaw) + vel_east × cos(Yaw), throttle value throttle_cmd = vel_up, and the UAV's heading angle is calculated using the expression yaw_cmd = clamp(2.0 × error_yaw, -50, 50). This transformation ensures that regardless of the UAV's current orientation, the NED speed output by the PID controller can be correctly converted to virtual joystick commands in the body coordinate system.

[0128] In step S507 of this embodiment, virtual joystick commands (including pitch_cmd, roll_cmd, throttle_cmd, and yaw_cmd) in the body coordinate system can be sent to the drone via the full-duplex communication protocol WebSocket. This controls the drone to fly at its flight speed and a determined heading angle until the cycle ends. At the end of the cycle, it is determined whether the position distance, altitude difference, and heading error meet their respective preset conditions. If not, the next cycle of controlled flight begins until the position distance, altitude difference, and heading error all meet their respective preset conditions. At this point, it is determined that the drone has reached the target candidate shooting position.

[0129] For example, the preset conditions for position distance, altitude difference, and heading error to all meet can be: position distance < 3m, altitude difference < 2m, and heading error < 5°. Of course, this embodiment is only used as an example and does not limit the specific values ​​in the preset conditions; these specific data can be set according to actual conditions. When the position distance, altitude difference, and heading error all meet their respective preset conditions, it is determined that the UAV has reached the target candidate shooting position. Using three independent PID controllers and converting the UAV's flight speed in the North-East-Down coordinate system to the UAV's body coordinate system, flight accuracy control of less than 3m is achieved. Compared to the existing waypoint flight accuracy of 5-10m, the control accuracy is improved by more than 50%, and the problem of "the UAV can correctly fly towards the target regardless of its direction" is solved.

[0130] In one possible embodiment, in Figure 5 Based on the illustrated embodiment, if the drone's flight time exceeds a first preset time, the drone is controlled to abort its flight and take pictures of the target vehicle's location. By using this embodiment, when the drone's flight timeout occurs, it is directly controlled to abort its flight and take pictures of the target vehicle's location, thus achieving a downgraded processing of the evidence collection process. This avoids delays in evidence collection due to flight timeouts and further improves evidence collection efficiency.

[0131] In this embodiment, the flight duration can be the flight duration of the drone starting from the moment the first virtual joystick command is issued to the drone after the target candidate shooting position is determined. The first preset duration can be set according to the actual situation, such as 50 seconds, 60 seconds, etc.

[0132] In one possible embodiment, in Figure 5Based on the illustrated embodiment, in the event of a recognition failure, the target candidate shooting position is switched, and the drone is controlled to fly to the target candidate shooting position and take a picture of the target vehicle to obtain an evidence photo. This embodiment achieves the automatic control of the drone to fly to the other side of the target vehicle to take a supplementary picture when the license plate is not recognized, thereby improving the license plate recognition rate.

[0133] For example, in the case where candidate shooting locations are calculated, see Figure 6 The process of controlling a drone to fly to a target candidate shooting position can include: for the current execution cycle, obtaining the drone's current GPS position, calculating the positional distance between the drone's current GPS position and the target candidate shooting position, and the drone's azimuth angle; calculating the eastward distance and northward distance based on the positional distance and azimuth angle; calculating the altitude difference between the drone's current flight altitude and the target altitude in the target candidate shooting position, and the heading error between the drone's current heading angle and the heading angle in the target candidate shooting angle; then, based on the calculated eastward distance, northward distance, and altitude difference, using three independent PID controllers (i.e., three-axis PID controllers), calculating the northward velocity component, eastward velocity component, and altitude velocity component of the drone in the NED coordinate system as the three components of flight speed; then, using the drone's current heading angle, converting the drone's flight speed in the NED coordinate system to the drone's body coordinate system, and determining the drone's heading angle based on the heading error, obtaining virtual joystick commands, and sending the virtual joystick commands to the drone via WebSocket to control the drone to fly until the end of the current execution cycle. At the end of the current execution cycle, it is determined whether the calculated position distance, altitude difference, and heading error meet their respective preset conditions. If they do, the UAV is determined to have reached the target candidate shooting position and hovers to maintain stability. If they do not meet the conditions, the control flight of the next cycle begins, that is, the GPS position of the UAV at the current moment is reacquired until the position distance, altitude difference, and heading error all meet their respective preset conditions.

[0134] In one possible embodiment, in Figure 3Based on the illustrated embodiment, if the celestial component of the second ray direction in the East-North-Sky coordinate system is not less than a preset value, license plate recognition is performed on the target video frame to obtain the recognition result; the target video frame and the recognition result are output; and / or, if the drone's photography time exceeds a second preset time, license plate recognition is performed on the target video frame to obtain the recognition result; the second pose, the target video frame, and the recognition result are output. In this embodiment, if the celestial component of the second ray direction in the East-North-Sky coordinate system is not less than a preset value, it indicates that the ray is not pointing downwards. In this case, the position of the target vehicle cannot be calculated, leading to positioning failure. Therefore, license plate recognition is directly performed on the target video frame. Similarly, if the drone's photography time exceeds the preset time, license plate recognition is also directly performed on the target video frame. This multi-level downgraded license plate recognition strategy ensures efficient resource utilization and maximizes effective recognition results during the license plate recognition process.

[0135] In this embodiment, the preset values ​​can be set according to the actual situation. This embodiment assumes that the UAV is at the origin of the ENU coordinate system and the UAV's flight altitude is H. Then the ground is at the position of the celestial component up = -H. The celestial component of the second ray direction ray_enu is represented as ray_enu_up. The preset value can be 0.1, 0.01, or 0.001, etc. For example, if the preset value is 0.01, then if ray_enu_up ≥ 0.01, it means that the second ray is not pointing downwards. At this time, the intersection point t = -H / ray_enu_up of the second ray direction in the vertical direction with the flight altitude is a very large value. This causes the coordinates of the intersection point of the second ray direction with the ground calculated by hit_east = ray_enu_east × t and hit_north = ray_enu_north × t to be inaccurate, thus failing to locate the target vehicle. In this case, license plate recognition is directly performed on the target video frame. The corresponding output result is the target video frame and the recognition result.

[0136] The drone photo-taking time refers to the duration from when the drone hovers and stabilizes before starting the timer, or the duration from when the drone receives the photo-taking command after hovering and stabilizing. The second preset time can be set according to actual conditions, such as 5 seconds, 10 seconds, or 15 seconds. If the drone photo-taking timeout occurs, license plate recognition is directly performed on the target video frame. The output results are the target vehicle's second pose in the geographic coordinate system, the target video frame, and the recognition result.

[0137] For example, see Figure 7When the main inference system in the control center detects an event requiring evidence collection in the real-time video stream of a drone, it generates an evidence collection task and sends it to the traffic intelligent agent. Alternatively, front-end personnel can manually send evidence collection tasks to the traffic intelligent agent. The traffic intelligent agent maintains a task queue. Upon receiving an evidence collection task, it performs a deduplication check. For example, it checks whether the time difference between the current evidence collection task and its adjacent preceding evidence collection task is less than a preset time difference, and whether the distance between the locations of the drones corresponding to these two evidence collection tasks is less than a set distance. If both conditions are met, the evidence collection task is determined to be a duplicate task and is skipped; otherwise, it is determined not to be a duplicate task. Then, vehicle localization is performed for the evidence collection task. The specific localization method can be referred to the above. Figure 2 and 3 The illustrated embodiment implements a system that, in the event of a positioning failure, directly skips flight control and performs license plate recognition on the target video frame in the evidence collection task. For example, it performs license plate recognition on the detection box area within the target video frame and publishes the recognition results. In the event of a successful positioning, it calculates the shooting position based on the positioning results, such as calculating the position directly above, in front of, and behind the target vehicle. When taking evidence photos, it selects the position closest to the drone for taking the picture. After selecting the target shooting position, it periodically executes PID flight control with a 500ms cycle to control the drone to fly to the target shooting position. If the drone has reached the target shooting position, it hovers to maintain stability. If the drone's flight timeout is exceeded, it directly stops flying and takes a picture of the target vehicle's location in the target video frame. After hovering and stabilizing for approximately 2 seconds, gimbal control commands are sent to the drone to adjust its gimbal angle. Capture commands are also sent, such as waiting 3 seconds to take a picture, starting to take a picture, or stopping to take a picture. If the drone's capture timeout occurs, the process stops waiting for the capture result and directly performs license plate recognition on the target video frame. Otherwise, the captured evidence photo is retrieved, and the evidence photo returned by the drone via a Kafka queue is retrieved. License plate recognition is then performed on the evidence photo, and the recognition result is published. If the recognition result is that no license plate is recognized, the target shooting position is switched, and the PID flight control steps are executed again to achieve multi-angle re-capture. The photo acquisition mechanism can use a dedicated PhotoConsumer thread to subscribe to a Kafka topic to maintain the latest photo results for each device's serial number (SN), and provide the `wait_for_photo(device_sn, timeout, after_ts)` method, using an Event mechanism to implement wait / notification and avoid polling.

[0138] For example, see Figure 8The main inference system in the control center acquires real-time video streams from different drones (i.e., video streaming), performs AI object detection on each stream, and generates an evidence collection task when an event requiring evidence is detected in a drone's video stream. This task is then sent to the traffic intelligent agent. Otherwise, the acquired video stream is SEI encoded and then pushed to form a live stream. Front-end personnel can also manually send evidence collection tasks to the traffic intelligent agent, and the drone also transmits its real-time GPS data and gimbal attitude to the agent. The traffic intelligent agent maintains a task queue and deduplicates the evidence collection tasks within it. For each deduplicated task, it first uses the target video frame, the target vehicle's pose within the video frame, and the UAV's GPS position and gimbal attitude at the target time of the video frame acquisition to perform vehicle localization. Then, based on the vehicle's GPS position and orientation, it plans a shooting location (i.e., calculates candidate shooting positions). A PID controller then controls the UAV to fly to the candidate shooting position to take a picture. Vehicle recognition is performed on the captured photos. If the license plate is not identified, the candidate shooting position is switched for a retake. Vehicle recognition is then performed on the retaken photos. The recognition results are published as alarm information (including the identified license plate number, vehicle GPS position, and evidence photos) via a Kafka queue and as evidence photos via the MinIO protocol. The traffic intelligent agent can be adapted to UAVs from different manufacturers.

[0139] For example, a traffic intelligent agent can adapt to drones from different manufacturers through a unified interface, such as the abstract base class VendorAdapter. For instance, `build_gimbal_control(pitch, yaw) → (command_topic, data)` can control the attitude of the gimbal of drones from different manufacturers; `build_flight_control(roll, pitch, throttle, yaw) → (command_topic, data)` can control the flight control system of drones from different manufacturers; and `build_take_photo() → (command_topic, data)` can control the photography capabilities of drones from different manufacturers. Here, `command_topic` is a topic in the Message Queuing Telemetry Transport (MQTT) protocol used to receive control commands (such as on / off, setting values). The format of `command_topic` can be: / {vendor code} / {device SN} / {service type code} / {subtype code} / {user ID}.

[0140] For example, the adaptation parameters of different manufacturers' drones that have been adapted to the traffic intelligent agent in this application are shown in the following table:

[0141]

[0142] Among them, pitch_speed represents the angular velocity controlling the gimbal's vertical rotation, yaw_speed represents the angular velocity controlling the gimbal's horizontal rotation, clientId represents the client identifier, and pilotId represents the pilot identifier.

[0143] For example, a factory method can be used to adapt drones from different manufacturers according to the manufacturer code. For instance, by using create_adapter(vendor_code, device_sn, user_id) → VendorAdapter, where VendorAdapter represents the manufacturer adapter, the adaptation of drones from different manufacturers can be achieved. At runtime, the device can be dynamically switched using set_device().

[0144] In this embodiment of the application, all instruction information between the traffic intelligent agent and the drone is sent via WebSocket, and the connection management adopts a strategy of delayed connection + thread-safe lock + automatic reconnection.

[0145] This application employs a dual-mode vehicle GPS real-time positioning method. Based on real-time UAV attitude data (GPS, gimbal angle, flight altitude), the pixel coordinates of the detection box of the target vehicle in the target video frame are back-projected into the vehicle's GPS latitude and longitude for positioning. During vehicle positioning, two modes (front-view mode and oblique-view mode) are automatically switched (threshold ±15°) according to the UAV gimbal pitch angle. The front-view mode uses direct GSD mapping (pixel offset × GSD → yaw rotation → latitude and longitude), while the oblique-view mode uses ray-ground intersection (pixel → camera ray → inverse transformation of rotation matrix → ENU ray → ground intersection → latitude and longitude). The mode calculation does not depend on an external GIS library and includes GPS correction compensation. Compared to existing pixel-to-GPS conversion methods that rely on pre-calibrated ground control points or high-precision real-time dynamic carrier phase differential technology (RTK), this application embodiment only requires the UAV's own GPS and attitude data to achieve meter-level positioning without the need for pre-calibrated ground control points or high-precision RTK, making it suitable for dynamic flight scenarios.

[0146] This system implements multi-position shooting planning based on vehicle orientation. After successfully locating the target vehicle, it automatically calculates three candidate shooting positions—directly above (for panoramic liability determination / illegal parking evidence collection), in front (for capturing the front license plate), and behind (for capturing the rear license plate)—based on the vehicle's GPS location and estimated orientation. Each position includes complete flight parameters (latitude and longitude, altitude, heading angle, gimbal pitch angle, and focal length). The focal length is automatically selected based on the slant distance (24mm for close-range wide-angle, 70mm for mid-range, and 168mm for long-range telephoto). During the first shot, the nearest shooting position is selected based on the Haversine distance to minimize flight time. Compared to existing methods that rely on pilot experience to select shooting angles, this embodiment automatically plans based on vehicle orientation, ensuring that the license plate is visible from at least one angle.

[0147] Flight control is achieved using a three-axis PID controller based on real-time GPS feedback from the UAV. Upon obtaining candidate shooting positions, a three-axis independent PID controller (horizontal position + altitude) with a 500ms control cycle is used. In each control cycle, real-time flight parameters are calculated, and the NED velocity component output by the PID controller is rotated to the body coordinate system based on the UAV's current heading angle (pitch = forward, roll = rightward) to ensure the UAV can correctly fly towards the target regardless of its orientation. The PID control process includes a complete control strategy encompassing integral limiting, arrival determination (triple conditions: distance <3m + altitude <2m + heading <5°), 2-second hovering stability, and 60-second timeout protection. Compared to the existing waypoint flight accuracy of 5-10m, this embodiment achieves <3m accuracy through PID closed-loop control, and the NED → body coordinate transformation solves the problem of "correctly flying towards the target regardless of the UAV's direction."

[0148] A coordinated timing management mechanism for flight, gimbal, photography, and recognition has been implemented, featuring complete automated timing orchestration: PID flight positioning → hovering and stabilization → gimbal adjustment → waiting for stabilization → photography → waiting for operation → asynchronous Kafka photo acquisition → license plate recognition → optional multi-angle reshooting. Photo acquisition utilizes a dedicated Kafka consumer to implement an Event waiting / notification mechanism, avoiding polling, and includes a four-level degradation strategy (location failure (license plate recognition on the target video frame) → flight timeout (controlling the drone to stop flight and take a photo towards the target vehicle's location) → photography timeout (license plate recognition on the target video frame) → recognition failure), ensuring a fallback plan for each stage's failure. Compared to existing systems where flight and photography are separated and require manual coordination, this embodiment achieves fully automated timing management.

[0149] It can be adapted to drones from multiple manufacturers. Through a unified control command interface (gimbal control, flight control, photography), it adapts to the command formats and communication protocols of drones from different manufacturers, thus achieving unified control of drones from multiple brands.

[0150] The evidence collection process runs in parallel with the main inspection reasoning. It operates in an independent thread through the traffic intelligent agent and is decoupled from the main reasoning loop through a bounded task queue. The main reasoning system runs continuously during the evidence collection process, realizing "inspection and handling simultaneously".

[0151] The evidence collection deduplication and multi-level degradation strategy based on GPS location, through the deduplication mechanism of GPS location + time window (no repeated evidence collection within 5m and 60 seconds of the same location), the four-level degradation strategy can ensure efficient use of resources and maximize results.

[0152] In one possible embodiment, the above vehicle positioning can also be achieved through the following schemes: (1) Unified processing of pure oblique view mode. In this method, the mode switching of frontal view / oblique view in the above embodiment is not adopted, and the ray-ground intersection method is used uniformly. This is because the ray is close to vertical when looking from the front, and the ray-ground intersection point is still effective. Although this method can simplify the logic, the ray is close to vertical when looking from the front, and the numerical accuracy of the calculated value is lower than that of GSD direct mapping. (2) Deep learning positioning. In this method, a depth estimation network (such as MiDaS, ZoeDepth) is used to estimate the depth of the target vehicle, and the vehicle position is estimated by combining the camera intrinsic back projection. Although this method does not rely on the GPS and attitude data of the UAV, it requires an additional depth estimation model, which increases the GPU load, and the absolute accuracy is not as good as the geometric method. (3) Multi-frame triangulation. In this method, the parallax generated by the movement of the UAV is used to locate the ground target through multi-frame triangulation. Although this method does not rely on the altitude data of the UAV, it requires the UAV to have significant displacement, which is not suitable for hovering scenarios.

[0153] In one possible embodiment, the above flight control can also be implemented through the following schemes: (1) Waypoint flight mode, in which the waypoint flight interface of the UAV SDK (such as DJIFlightController.startGoTo) is directly called, and the flight is completed by the UAV's own flight control system. Although this method is simple to implement, the arrival accuracy is low (5-10m), and the interfaces of different brands of SDKs are not uniform. (2) Model predictive control (MPC), in which MPC is used to replace PID, and predictive control is performed considering the UAV dynamic model. Although this method is smoother and can handle constraints, the amount of calculation is large, an accurate dynamic model is required, and the implementation complexity is high. (3) Pure proportional control, in which the integral and derivative terms in PID are cancelled, and only proportional control is used. Although this method is the simplest to implement, there is steady-state error and the arrival accuracy is reduced.

[0154] In one possible embodiment, the above calculation of candidate shooting positions can also be achieved through the following schemes: (1) Surround shooting: In this method, the fixed front / back position is not calculated, but the drone flies around the target vehicle and takes a picture at certain angles, and selects the clearest evidence photo of the license plate. Although this method does not rely on the vehicle orientation estimation, it is time-consuming and consumes a lot of drone power. (2) Shooting only from directly above + digital zoom cropping: In this method, the drone flies only from directly above and takes pictures using a telephoto lens or digital zoom, and then crops the license plate area to obtain the evidence photo. Although this method has the shortest flight path, the license plate is almost invisible from a top-down angle.

[0155] In one possible embodiment, the above-mentioned adaptation to drones from different manufacturers can also be achieved through the following schemes: (1) Using the MAVLink unified protocol. In this method, the MAVLink open-source protocol is used as a unified control interface, and drones from different manufacturers access the network through the MAVLink gateway. Although this method is a protocol standardization, not all commercial drones support MAVLink. (2) Using the MQTT message bus. In this method, MQTT is used instead of WebSocket as the command transmission channel, and adapters from different manufacturers publish commands to the corresponding MQTT topic. Although this method supports offline messages and QoS, the latency is higher than that of WebSocket.

[0156] For example, applying the evidence collection method based on UAV video streams in this application embodiment, for the rapid handling of minor traffic accidents, the UAV patrols the airspace above urban roads. The AI ​​main inference system of the control center detects the accident scene (multiple vehicles parked close together, vehicles abnormally stopped in the middle of the lane, etc.) and generates an evidence collection task, which is sent to the traffic intelligent agent in the control center. Then, the traffic intelligent agent first takes a panoramic view of the accident (a top-down view, recording the relative positions of the vehicles and the point of collision), and then flies to the vicinity of each involved vehicle to take pictures of the license plates. All the evidence collection results (panoramic photos + license plate numbers + GPS locations of each vehicle) are pushed to the traffic police platform or insurance quick-handling system via Kafka messages, supporting remote liability determination and rapid claims settlement. The entire process is completed in 1-2 minutes, without the need for traffic police to be present. The parties involved can quickly leave the scene after receiving the handling results, reducing secondary congestion caused by the accident.

[0157] For example, applying the evidence collection method based on drone video streams from the embodiments of this application, automatic evidence collection is performed on illegally parked vehicles. The drone patrols along a preset route or within an electronic fence area. The main inference system AI in the control center detects the illegally parked vehicle and generates an evidence collection task, which is then sent to the traffic intelligent agent in the control center. The traffic intelligent agent then controls the drone to fly near the vehicle to take a photo of the license plate. After recognizing the license plate number, the image is combined with the vehicle's GPS location and on-site footage... Figure 1The system reports to the traffic police platform, and it takes 30-60 seconds to process a single vehicle's certificate. Multiple illegally parked vehicles can be processed in one inspection, making it suitable for areas with high rates of illegal parking, such as around schools, commercial districts, and fire lanes.

[0158] In one possible embodiment, the traffic agent in this application runs as a submodule of the UAV video stream AI inference control center, without changing the main flow of the existing control center's main inference system: pull → inference → push. Specifically, it can be integrated in the following ways: Trigger entry: Add traffic-related fence judgment and agent triggering logic to the detection result processing stage of the existing inference pipeline; Data acquisition: Reuse existing GPS consumers, gimbal consumers, and video frames through callback functions; Result output: Reuse existing Kafka alarm publishers and MinIO screenshot uploaders; Front-end interaction: Add manual triggering and result query APIs to the existing HTTP server.

[0159] Based on the same inventive concept, embodiments of this application also provide an evidence collection system based on drone video streams, see [link to relevant documentation]. Figure 9 The evidence collection system based on UAV video streams includes a control center 901 and a UAV 902. The control center 901 includes a main inference system 9011 and a traffic agent 9012. The main inference system 9011 acquires real-time video streams from different UAVs 902 and performs target detection on each stream. When an event requiring evidence collection is detected in a UAV's real-time video stream, an evidence collection task is generated and sent to the traffic agent 9012. The traffic agent 9012 executes the evidence collection method based on UAV video streams according to any of the above embodiments. This embodiment achieves end-to-end autonomous evidence collection, including vehicle positioning, autonomous UAV flight, photography, and license plate recognition. Furthermore, the photography-based evidence collection does not rely on manual operation, improving efficiency.

[0160] This application also provides a control center, see [link to relevant documentation] Figure 10 It includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002 and the memory 1003 communicate with each other through the communication bus 1004.

[0161] Memory 1003 is used to store computer programs;

[0162] When the processor 1001 executes the program stored in the memory 1003, it implements the evidence collection method based on UAV video stream of any of the above embodiments to achieve the same technical effect.

[0163] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0164] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0165] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0166] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0167] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements the evidence collection method based on UAV video stream of any of the above embodiments to achieve the same technical effect.

[0168] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the forensic method based on drone video stream of any of the above embodiments to achieve the same technical effect.

[0169] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0170] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0171] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system / control center embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0172] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for forensic investigation based on drone video streams, characterized in that, The method includes: Acquire a target video frame containing the target vehicle, capture the target time of the target video frame, and the first pose of the target vehicle in the target video frame; The position and gimbal attitude of the UAV at the target time are obtained as the first position and the target gimbal attitude; Based on the target video frame, the first position, and the target gimbal attitude, the first pose is projected onto the geographic coordinate system to obtain the second pose of the target vehicle in the geographic coordinate system. Based on the second pose, the first position, and the target gimbal attitude, calculate the candidate shooting position; The drone is controlled to fly to the candidate shooting location and take pictures towards the location of the target vehicle to obtain evidence photos; The license plate was identified by performing license plate recognition on the evidence photos, and the recognition result was obtained; Output the second pose, the evidence photo, and the recognition result.

2. The method according to claim 1, characterized in that, When the absolute value of the difference between the pitch angle and negative 90° of the target gimbal attitude does not exceed a preset angle, the step of projecting the first pose onto the geographic coordinate system based on the target video frame, the first position, and the target gimbal attitude to obtain the second pose of the target vehicle in the geographic coordinate system includes: The ground sampling distance is calculated based on the screen width and height of the target video frame, the diagonal field of view of the UAV camera, and the flight altitude of the UAV. Based on the screen width and screen height of the target video frame, the ground sampling distance, and the pixel coordinates in the first pose, calculate the first offset value of the camera coordinate system of the UAV camera relative to the ground coordinate system; Based on the first offset value and the heading angle of the target gimbal attitude, calculate the second offset value of the UAV camera's camera coordinate system relative to the geographic coordinate system; Based on the first position and the second offset value, the second position of the target vehicle in the geographic coordinate system is calculated; The vehicle orientation of the target vehicle is estimated based on the aspect ratio of the target vehicle in the target video frame and the heading angle of the target gimbal attitude.

3. The method according to claim 1, characterized in that, When the absolute value of the difference between the pitch angle and negative 90° of the target gimbal attitude exceeds a preset angle, the step of projecting the first pose onto the geographic coordinate system based on the target video frame, the first position, and the target gimbal attitude to obtain the second pose of the target vehicle in the geographic coordinate system includes: Based on the pixel coordinates in the first pose, the screen width and screen height of the target video frame, and the pixel focal length of the UAV camera, the first ray direction of the UAV camera in the camera coordinate system is calculated; the pixel focal length is determined based on the screen width and screen height of the target video frame and the field of view of the UAV camera, and is used to characterize the value of the equivalent focal length of the UAV camera on the imaging plane when expressed in pixels. Transform the direction of the first ray to the East-North-Sky coordinate system to obtain the direction of the second ray in the East-North-Sky coordinate system; Calculate the coordinates of the intersection point between the second ray direction and the ground; Based on the first position and the coordinates of the intersection point, calculate the second position of the target vehicle in the geographic coordinate system; The vehicle orientation of the target vehicle is estimated based on the aspect ratio of the target vehicle in the target video frame and the heading angle of the target gimbal attitude.

4. The method according to claim 1, characterized in that, The step of calculating candidate shooting positions based on the second pose, the first position, and the target gimbal pose includes: Based on the second position in the second pose and the flight altitude of the UAV, a first candidate shooting position is obtained; the first candidate shooting position is located at the flight altitude directly above the second position; The position where the target vehicle's orientation is offset by a preset distance is used as the second candidate shooting position; The position where the target vehicle is offset by a preset distance in the opposite direction is taken as the third candidate shooting position.

5. The method according to any one of claims 1-4, characterized in that, There are multiple candidate shooting locations; controlling the drone to fly to the candidate shooting location includes: Perform the following steps periodically: Obtain the third position and heading angle of the UAV at the current moment; Calculate the positional distance between the third position and the target candidate shooting position, and calculate the azimuth angle of the UAV; the target candidate shooting position is the candidate shooting position closest to the UAV. Calculate the eastward distance and the northward distance based on the location distance and the azimuth angle, respectively; Calculate the altitude difference between the current flight altitude of the UAV and the altitude of the target in the candidate shooting position, and calculate the heading error between the heading angle at the current moment and the heading angle in the candidate shooting angle; Calculate the flight speed of the UAV in the North-East-Down coordinate system based on the eastward distance, the northward distance, and the altitude difference; Based on the heading angle at the current moment, the flight speed of the UAV in the North-East-Down coordinate system is converted to the UAV's body coordinate system, and the heading angle of the UAV is determined based on the heading error; Control the UAV to fly at the stated flight speed and the determined heading angle until the cycle ends; At the end of the cycle, if the position distance, the altitude difference, and the heading error do not meet their respective preset conditions, the next cycle begins until the position distance, the altitude difference, and the heading error all meet their respective preset conditions.

6. The method according to claim 5, characterized in that, The method further includes: If the drone's flight time exceeds a first preset time, the drone will be controlled to stop flying and take pictures of the location of the target vehicle.

7. The method according to claim 6, characterized in that, The method further includes: If the identification result is a failure, switch the target candidate shooting position and execute the step of controlling the drone to fly to the target candidate shooting position and take pictures towards the location of the target vehicle to obtain evidence photos.

8. The method according to claim 3, characterized in that, The method further includes: If the celestial component of the second ray direction in the East-North-Sky coordinate system is not less than a preset value, license plate recognition is performed on the target video frame to obtain the recognition result. Output the target video frame and the recognition result; And / or, If the drone's photo-taking time exceeds the second preset time, license plate recognition is performed on the target video frame to obtain the recognition result; Output the second pose, the target video frame, and the recognition result.

9. A forensic system based on UAV video stream, characterized in that, The evidence collection system includes a control center and a drone; the control center includes a main inference system and a traffic intelligent agent. The main inference system is used to acquire real-time video streams from different drones and perform target detection on each real-time video stream. When an event requiring evidence collection is detected in a real-time video stream from a drone, an evidence collection task is generated and sent to the traffic intelligent agent. The traffic intelligent agent is used to perform the method described in any one of claims 1-8.

10. A control center, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-8.

Images