An unmanned aerial vehicle traffic incident closed-loop forensics method and system based on a multi-modal intelligent agent and cross-network transfer

Abstract

The application discloses a UAV traffic incident closed-loop evidence collection method and system based on a multi-modal intelligent agent and cross-network transfer, wherein an alarm is generated by a monitoring system in a traffic special network after an abnormal event is identified, and the alarm is transferred to a UAV scheduling system on the public network side through a cellular SMS channel; the multi-modal intelligent agent analyzes the alarm, filters events that need to be responded by a UAV, selects a patrol route, and triggers the UAV to automatically take off; the UAV collects video frames along the route point by point, the intelligent agent judges the source through a two-layer gate mechanism, and actively searches for a real event source outside a fixed monitoring field of view; after a hit, the source geographical coordinates are calculated from the pixel coordinates, the UAV locks the target, gradually approaches it, and performs multi-directional surrounding zoom shooting to collect evidence; the evidence images, videos and state messages are returned and stored with the event unique code as the key; compared with the prior art, the application realizes full automation of the closed loop from event discovery to evidence fixation, and has the characteristics of cross-network compliance, rapid response and clear evidence.

Description

Technical Field

[0001] This invention relates to the field of autonomous drone inspection, specifically to a closed-loop evidence collection method and system for drone traffic incidents based on multimodal intelligent agents and cross-network relay. Background Technology

[0002] With the advancement of intelligent transportation infrastructure, automatic identification of traffic anomalies based on roadside surveillance videos has been widely deployed, capable of detecting anomalies such as congestion, accidents, and debris spills. However, in practical applications, two technical obstacles remain in advancing the process from event detection to on-site arrival, verification, characterization, and evidence preservation.

[0003] 1. Fixed surveillance cameras have limited fields of view, often only observing the "result" or "appearance" of traffic anomalies, rather than the event itself. For example, a surveillance camera on a certain road segment may identify "severe slow traffic / congestion," but the traffic accident causing the congestion may have occurred on a ramp, curve, or adjacent road segment outside the camera's field of view. In this case, the fixed camera cannot confirm the true nature of the event, locate the actual source of the event, or obtain license plate-level evidence; while relying on manual dispatch of police officers to the scene suffers from slow response and limited coverage.

[0004] 2. Network isolation hinders the coordination between detection systems and response resources. Road surveillance video streams and event recognition algorithms are typically deployed on dedicated transportation networks (private networks, physically or logically isolated from the public network), while drone dispatch systems, drone nesting sites, and public-facing AI services are deployed on the public network; the two are not interconnected. Using dedicated lines to connect them or establishing conventional application interfaces and message buses between the two networks often faces high construction costs and compliance restrictions, making it difficult to reliably and in real-time trigger event detection results from the dedicated transportation network to drone assets on the public network.

[0005] In addition, existing drone inspections mostly adopt timed patrols along preset routes, lacking a closed loop of "event-driven, on-demand takeoff, and autonomous source identification and evidence collection by intelligent agents"; a few solutions that attempt to process the entire video directly with large models face the problems of high computing power costs and failure to form a closed loop linkage with execution links such as flight control, gimbal alignment, zoom and panoramic shooting. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing video surveillance, which is mostly fixed and has a limited field of view; road monitoring and drone operation are located on different networks and require manual coordination, resulting in low efficiency. To address these shortcomings, this invention proposes a closed-loop evidence collection method and system for drone traffic incidents based on multimodal intelligent agents and cross-network relay.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A closed-loop evidence collection method for drone traffic incidents based on multimodal intelligent agents and cross-network relay includes the following steps:

[0009] S1. The road monitoring video analysis system deployed in the traffic private network identifies abnormal events in the road video stream, generates alarm information containing event location information, and transfers the alarm information from the traffic private network to the drone dispatch system deployed in the public network via the cellular SMS channel.

[0010] S2. The multimodal intelligent agent of the UAV dispatch system performs event analysis on alarm information to determine whether UAV response is required. When it is determined that UAV response is required, the multimodal intelligent agent selects the corresponding patrol route based on the event location information.

[0011] S3. Trigger the drone to automatically take off with the location indicated by the event location information as the target point, and fly along the selected patrol route. During the flight, video frames are collected, and the multimodal intelligent agent judges the source of each video frame to identify the source of the event.

[0012] S4. When the source of the incident is determined, the drone locks onto the source and takes multi-directional surround shots to collect evidence.

[0013] S5. After associating the obtained on-site images and videos with the alarm information, save them and send them back to the business management platform.

[0014] As a further preferred embodiment of the present invention, in S1,

[0015] The SMS sending terminal within the dedicated transportation network encodes the alarm information into an SMS message and sends it out.

[0016] The SMS receiving gateway on the public network side receives SMS messages via a cellular communication module connected through a serial port using AT commands. It decodes the UCS2 encoded text, reassembles long SMS messages into fragments according to the sender, and parses them into structured data containing at least a unique event code, WGS84 coordinates arranged in longitude first and latitude last, station number and coordinate mapping, event text, and sending time. The data is then published via message middleware using the unique event code as the key. The event location coordinates are arranged in a longitude first and latitude last format.

[0017] As a further preferred embodiment of the present invention, the multimodal intelligent agent is configured as an intelligent agent based on a large language model. The multimodal intelligent agent analyzes the alarm information, confirms and outputs the event type, event severity, event confidence level, whether a drone response is required, and the reason for the analysis. The event confidence level represents the degree of credibility of the alarm information corresponding to the real traffic event.

[0018] The conditions for determining that a drone response is required are: the severity of the event reaches a preset level, the alarm information contains valid location coordinates, the alarm information is not an obvious false alarm, and the event needs to be verified or evidence collected on-site. All four conditions must be met simultaneously; otherwise, no drone response is required. Among them, when the confidence level of the event is not lower than the preset false alarm judgment threshold and the event type is not judged as a false alarm, it is judged as a non-obvious false alarm; otherwise, it is judged as an obvious false alarm.

[0019] When a drone response is required, the corresponding patrol route is selected. Specifically, this involves: extracting the road number, station number, and direction from the event description and structured data, and assembling them into a route selection key; querying the preset route mapping table using the route selection key to obtain the patrol route; if the patrol route is not matched, the drone refuses to take off and issues an alarm, instead of returning to the default route.

[0020] As a further preferred embodiment of the present invention, before triggering the automatic takeoff of the UAV in S3, a coordinate safety verification is first performed, including:

[0021] Reorder the event location coordinates from longitude first and latitude last to latitude first and longitude last, and verify whether the reordered coordinates fall within the preset geographical range;

[0022] The distance between the verification event point and the drone nest does not exceed the preset operating radius;

[0023] Limit the target's flight altitude to a preset safe altitude range;

[0024] If any of the above conditions fail to be verified, takeoff will be refused.

[0025] As a further preferred embodiment of the present invention, the source determination adopts a two-layer gate mechanism, specifically including:

[0026] In the first layer, the multimodal agent makes judgments on the video frames of each waypoint and outputs the original judgment and confidence level.

[0027] The second layer determines a valid hit if and only if the original determination is the source and the confidence level is not lower than the preset threshold. A valid hit stops subsequent point-by-point source determination.

[0028] As a further preferred embodiment of the present invention, in step S3, when the source of the event is determined to be the source, a spatial ray determined by pixel coordinates is constructed based on the position of the UAV, the pitch and azimuth angles of the gimbal, the field of view angle of the camera, and the flight altitude when the frame is acquired, and the intersection point of the ray with the ground is calculated to back-calculate the geographical coordinates of the source; for frames at no less than two different observation positions, rays are constructed respectively, and triangulation is performed when the horizontal convergence angle of each ray is not less than a preset angle threshold, and the result of the triangulation intersection is used as the geographical coordinates of the event source.

[0029] As a further preferred embodiment of the present invention, in S4, during the locking and approaching phase, the position of the source target pixel in the hit frame is aligned and adjusted to the center of the image by the camera pointing, and the drone flies along the current gimbal orientation with a preset step size, re-detecting and re-aligning the position of the source target pixel to the center of the image at each step, until the gimbal tilt angle reaches a preset tilt angle threshold, at which point the drone stops, collects evidence on the spot, and returns to base; if during the approaching process, the approaching distance limit is reached or the source target is continuously lost, and the preset number of retries is reached, then the drone collects evidence on the spot and returns to base;

[0030] During the surround shooting phase, multiple surround shooting waypoints are generated at preset angles and with the source as the center. The drone takes pictures of each point in a surround shooting manner. The drone first points to the center of the source, then re-detects and aligns the target with the center of the image, and then switches to telephoto zoom to take pictures, thus obtaining evidence images from multiple directions.

[0031] As a further preferred embodiment of the present invention, in step S5, the on-site evidence images and videos of each stage are uploaded to the object storage using the event unique code as the key, and assembled into an accessible resource address;

[0032] Throughout the mission, the drone's real-time location trajectory is collected and transmitted back at a preset frequency. When the drone's telemetry remains silent for more than a preset duration, the drone's location telemetry is switched to be the trajectory source.

[0033] At each stage of acceptance, takeoff, point-by-point source identification, surrounding evidence collection and mission completion, the system sends back analysis, source identification and mission status messages carrying the unique event code to the business control platform.

[0034] The obtained on-site images and videos are then associated with the event identification information, stored, and sent back to the business management platform.

[0035] A closed-loop evidence collection system for drone traffic incidents based on multimodal intelligent agents and cross-network relay includes:

[0036] The SMS cross-network relay module is set up on the public network side. It is used to receive event SMS messages from the traffic private network via the cellular communication module on the public network side, parse them into structured alarms carrying unique event codes and location coordinates, and publish them through message middleware.

[0037] The event analysis intelligent agent module has a built-in multimodal intelligent agent for filtering false alarms, classifying events, and determining whether a drone response is needed for structured alarms.

[0038] The route selection module is used to extract road number, station number and direction from event information, and select patrol routes by looking up the combination of these information in a table.

[0039] The flight control module is used to perform coordinate safety verification before the drone takes off, control the drone to take off automatically, fly point by point along the patrol route, align the gimbal and zoom to take pictures, and return to the cabin.

[0040] The point-to-point source identification module is used to identify the source of video frames at each waypoint, and to calculate the geographic coordinates of the event source from the pixel coordinates of the hit frames.

[0041] The approximation and surround evidence collection module is used to locate the source of the event, gradually approach it, and perform multi-directional surround zoom evidence collection on the source.

[0042] The evidence retention and transmission module is used to upload and retain on-site evidence at each stage, and transmit real-time trajectory and status messages at each stage.

[0043] The road monitoring video analysis system is set up on a dedicated traffic network, while the cross-network relay module, event analysis intelligent agent module, flight control module, and UAV are set up on the public network. The two are connected through the cross-network relay channel to trigger events across networks.

[0044] As a further preferred embodiment of the present invention, the cross-network relay channel is a cellular SMS channel; the cross-network relay module receives SMS messages via AT commands through a cellular communication module connected via a serial port, completes UCS2 decoding and long SMS concatenation, and parses it into a structured alarm carrying a unique event code and positioning coordinates, which is then published via a message middleware using the unique event code as the key; the flight control module adopts a single-machine serial lock to ensure that only one flight mission is executed at any given time, and performs three safety guardrails before takeoff: coordinate validity verification, operation radius verification, and flight altitude clamping.

[0045] The present invention proposes a closed-loop evidence collection method and system for unmanned aerial vehicle (UAV) traffic incidents based on multimodal intelligent agents and cross-network relay, which has the following advantages compared with the prior art:

[0046] 1. This invention uses cellular SMS as an event triggering channel that bridges the gap between the dedicated transportation network and the public network. It utilizes existing cellular infrastructure and eliminates the need for dedicated lines or open interfaces. While ensuring network isolation compliance, it significantly reduces construction costs and solves the practical problem of the difficulty in linking detection systems with drone disposal resources.

[0047] 2. This invention utilizes a multimodal intelligent agent throughout the entire evidence collection process, enabling the system to understand the semantics of unstructured alarms and to perform autonomous reasoning based on visual content. It replaces traditional hard-coded rules, allowing the system to proactively search for the sources of various events such as congestion, accidents, and spilled objects outside the fixed monitoring field of view. This overcomes the limitation of fixed cameras that can only observe appearances but cannot locate the event itself, significantly improving generalization ability and discrimination accuracy.

[0048] 3. This invention performs coordinate safety verification before takeoff and conducts multiple safety confirmations on the key parameters output by the intelligent agent, effectively preventing safety accidents caused by coordinate format errors, overflight, or altitude exceeding limits, and ensuring flight safety during the UAV launch process.

[0049] 4. This invention employs a two-layer gate mechanism for point-by-point source identification, separating semantic discrimination from quantitative risk control. This not only preserves the intelligent agent's advantage in understanding complex scenarios but also uses a confidence threshold to suppress false alarms triggered by low confidence, thus effectively controlling the false alarm rate while ensuring recall.

[0050] 5. After the source is hit, the present invention constructs a spatial ray from the target pixel coordinates and intersects it with the ground to calculate the source's geographical coordinates. The source can be accurately located without the need for additional loads such as laser ranging. In addition, by constructing rays for hit frames at no less than two different observation positions, and performing triangulation and intersection when the horizontal convergence angle of each ray meets the standard, the systematic bias introduced by terrain and pitch angle errors in single-frame ranging is effectively eliminated, and the positioning robustness is significantly improved.

[0051] 6. This invention uses the unique event code as the key throughout the process, storing on-site evidence images and videos in object storage, and transmitting real-time trajectories and event analysis data from each stage back to the business management platform, forming a complete evidence chain and task process that is traceable, visualized, and archiveable; and ensuring flight safety through safety barriers such as coordinate reordering verification, operating radius, and maximum flight altitude, providing a reliable basis for emergency response and event archiving by traffic management departments.

[0052] 7. This invention employs a point-and-click aiming and locking strategy, gradually approaching along the gimbal's orientation, adaptive stopping at the tilt angle threshold, and four-way surround zoom shooting to overcome the problems of fixed cameras having a single perspective and being unable to see clearly due to excessive distance. It can obtain clear evidence from multiple directions, including license plate level, around the source of the event. The upper limit of the approach distance and the on-site evidence collection and return-to-base measures ensure the safety and controllability of the evidence collection process. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the overall closed-loop process provided in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the system architecture for isolating the private transportation network from the public network and for cross-network SMS relay in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of pointing and aligning for evidence collection in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of four-directional surround evidence collection in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram illustrating a positive example of the structured source identification result output by the system in an embodiment of the present invention.

[0058] Figure 6 This is a reverse example diagram of the structured source identification results output by the system in this embodiment of the invention.

[0059] Figure 7 This is a block diagram of the system module logic architecture provided for an embodiment of the present invention. Detailed Implementation

[0060] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0061] The present invention relates to a closed-loop evidence collection method and system for unmanned aerial vehicle (UAV) traffic incidents based on multimodal intelligent agents and cross-network relay. It enables cross-network triggering without breaking through isolated networks, and allows the intelligent agent to autonomously drive the UAV to the scene, actively find the real source of the event outside the fixed camera's field of view, and obtain clear evidence, forming a complete closed loop.

[0062] Example 1, combined with Figure 1 and 2 This paper presents a closed-loop evidence collection method for drone traffic incidents based on multimodal intelligent agents and cross-network relay. A single multimodal intelligent agent drives the entire process from cross-network triggering to event analysis, automatic takeoff, point-by-point source identification, close-up panoramic imaging, and evidence collection and transmission. The specific steps are as follows:

[0063] A closed-loop evidence collection method for drone traffic incidents based on multimodal intelligent agents and cross-network relay includes the following steps:

[0064] S1: The road monitoring video analysis system, set up within the dedicated transportation network, identifies abnormal events in the road video stream and generates alarm information that includes event description and location information.

[0065] In this step, a road monitoring video analysis system (such as a video AI event recognition system) deployed within the dedicated transportation network performs persistent traffic anomaly event identification on the video streams from roadside monitoring cameras. This system often identifies the surface anomalies; for example, it might only output "severe traffic slowdown / congestion" for a monitoring point, even though the actual accident causing the slowdown might not be within the camera's field of view. After identifying an anomaly, the system encodes the event information into a text message via a text message sent from within the dedicated transportation network. An example text message might read: "[XX Highway] At 14:55 on May 9, 2026, a severe traffic slowdown occurred at K309+900 on XX Road, downhill…", accompanied by the event location indicated by station number and coordinates (e.g., station number "309+900" corresponds to coordinates [X, Y]).

[0066] S2: The alarm information is transferred from the traffic network to the drone dispatch system set up on the public network via a preset cross-network transfer channel; the public network parses the alarm information and generates structured data.

[0067] The cross-network relay channel is set as a cellular SMS channel; the SMS sending end in the dedicated transportation network encodes the alarm information into SMS and sends it; the SMS receiving gateway on the public network side receives the SMS via the cellular communication module connected by the serial port using AT commands, decodes the UCS2 encoded text, splices and reconstructs long SMS messages according to the sender, and parses them into structured data that includes at least the event unique code, WGS84 coordinates arranged in longitude first and latitude last, station number and coordinate mapping, event text and sending time, and publishes it through the message middleware with the event unique code as the key.

[0068] like Figure 2 As shown, the dedicated transportation network and the public network are isolated and do not communicate with each other. This invention uses a cellular SMS channel as the event triggering and location information transmission channel between the two. The SMS receiving gateway on the public network side (denoted as djsms in this embodiment) is connected to the GSM / 4G cellular communication module via a serial port (e.g., 115200 baud rate) and operates in AT command mode: configured as a new message callback mode (AT+CNMI=2,1,0,0,0) and supplemented by a listening thread polling the serial port buffer. After receiving a new message notification (+CMTI), the SMS is read (AT+CMGR). For text messages using UCS2 encoding, they are decoded and restored to Chinese characters using UTF-16BE; for long SMS messages sent by the operator in segments, they are fragmented and buffered according to the sender's number, and merged within a short window (e.g., 5 seconds) or forcibly output after a timeout (e.g., 30 seconds). The integrity is determined by the header identifier "[XXX Highway]" and the end of the date and time, and then the fragments are reassembled. The SMS receiving gateway parses the restored alarm into structured data and publishes it via message middleware (MQTT, QoS=1). In this embodiment, it is first published to the topic sms / received, and then normalized by the backend to the authoritative event topic sms / event. This structured data includes at least: the event unique code evtId, the WGS84 coordinates lnglat arranged in longitude first and latitude last, the event text evt, the sender, the sending time timestamp, and the pils mapping from station number to coordinates.

[0069] S3. Based on structured data, the multimodal intelligent agent built into the UAV scheduling system analyzes the event to determine whether a UAV response is needed. When a UAV response is needed, the corresponding patrol route is selected simultaneously based on the event location coordinates.

[0070] The multimodal agent is configured as an agent based on a large language model. The multimodal agent analyzes the alarm information, confirms and outputs the event type, event severity, event confidence level, whether a drone response is required, and the reason for the analysis. The event confidence level represents the credibility of the alarm information corresponding to the real traffic event.

[0071] The conditions for determining that a drone response is required are: the severity of the event reaches a preset level, the alarm information contains valid location coordinates, the alarm information is not an obvious false alarm, and the event needs to be verified or evidence collected on-site. All four conditions must be met simultaneously; otherwise, no drone response is required. Among them, when the confidence level of the event is not lower than the preset false alarm judgment threshold and the event type is not judged as a false alarm, it is judged as a non-obvious false alarm; otherwise, it is judged as an obvious false alarm.

[0072] When a drone response is required, the corresponding patrol route is selected. Specifically, this involves: extracting the road number, station number, and direction from the event description and structured data, and assembling them into a route selection key; querying the preset route mapping table using the route selection key to obtain the patrol route; if the patrol route is not matched, the drone refuses to take off and issues an alarm, instead of returning to the default route.

[0073] In this step, the multimodal agent (based on a large language model, this embodiment uses Qwen3.6-plus, temperature=0) performs event analysis on the alarm and outputs the event type event_type (values ​​such as traffic_accident, traffic_congestion, road_obstacle, false_alarm, other), severity event_severity (high / medium / low), whether a drone response is required needs_drone, and the reason for the analysis. The conditions for determining that a drone response is required are: severity is high or medium, it contains valid latitude and longitude coordinates, it is not an obvious false alarm (confidence level not less than 0.5), and it requires on-site verification or evidence collection. In terms of route selection, the adaptation process uses regular expressions to extract road numbers (such as GXX, SXX), station numbers (such as K309+900 being unified as 309+900), and directions (up / down) from the event text and monitoring point names. The route selection key is then assembled using the combination of "road number-station number-direction" (such as Gxx-309+900-down). The corresponding patrol route file is obtained by querying the preset route mapping table (camera_routes.yaml). If the selected route key is not matched or the downgraded match is not unique, the system will refuse to deploy and issue an alarm, and will never return to the default route to avoid flying the wrong route.

[0074] S4: When the analysis determines that a drone response is required, the multimodal agent triggers the drone to automatically take off with the location indicated by the event location information as the target point and fly along the selected patrol route.

[0075] Before triggering automatic takeoff of the drone, a coordinate safety check is performed, including:

[0076] Reorder the event location coordinates from longitude first and latitude last to latitude first and longitude last, and verify whether the reordered coordinates fall within the preset geographical range;

[0077] The distance between the verification event point and the drone nest does not exceed the preset operating radius;

[0078] Limit the target's flight altitude to a preset safe altitude range;

[0079] If any of the above conditions fail to be verified, takeoff will be refused.

[0080] The bridging service on the UAV scheduling system side subscribes to the event topic sms / event (QoS=1). To ensure reliability and single-machine security, the bridging service performs persistent idempotent deduplication of evtId across restarts (the processed evtId is written to disk, and the "mark first and then process" strategy is adopted to prevent re-deployment and re-flight after crash); and a single "flight lock" is obtained in a non-blocking manner to realize task serialization. If it is busy, it will directly reject and send back a "rejected" notification with the unique event code as the key. In the adaptation process, lnglat is reordered in the order of "longitude first, latitude second" to latitude=lnglat[1] and longitude=lnglat[0], and it is verified that it falls within the geographical range. Otherwise, an error is reported and the flight is rejected. This reordering and range verification is the key security defense to prevent the silent flight to the wrong point due to writing the reverse coordinates.

[0081] In this step, the multimodal agent invokes the task orchestration entry point (dispatch→run_mission) to trigger flight, and a single-machine serial lock ensures that only one flight occurs at any given time. Before takeoff, three safety barriers are executed sequentially: coordinate validity verification (within the territory), operation radius verification (the event point is no more than the operation radius from the nest, which is 10km in this embodiment), and target flight altitude constraint (limited to a safe takeoff altitude of 30m to a maximum flight altitude of 120m to prevent large model illusions from causing out-of-bounds travel). Takeoff uses the event location coordinates as the target point (takeoff_to_point), and its target ellipsoidal height is obtained by superimposing the relative ground height on the nest ground ellipsoidal height (in this embodiment, the measured nest ground ellipsoidal height is approximately 17m). After takeoff, the camera is forcibly switched to video recording mode (CAMERA_MODE_VIDEO) to ensure that the aspect ratio of the image transmission and frame extraction is 16:9 (1920×1080) instead of the 4:3 of the photo mode; then the gimbal is reset to face downwards, and a wide-angle image transmission source is selected (select_video_id('wide')) for subsequent frame extraction source determination.

[0082] S5: The UAV collects video frames along the patrol route, point by point, and transmits them back to the multimodal agent. The multimodal agent analyzes the video frames at each waypoint to identify the true source of the event. When the source is identified, a spatial ray is constructed based on the UAV's pose information and camera parameters at the time the frame was collected. The geographic coordinates of the event source are then calculated based on the spatial ray. The UAV's pose information and camera parameters include: UAV position, gimbal pitch and azimuth angles, camera field of view, and flight altitude. Spatial rays are constructed for frames from at least two different observation positions. Triangulation is performed to find the intersection of the rays when the horizontal convergence angle of each ray is not less than a preset angle threshold. The result of the triangulation intersection is used as the geographic coordinates of the event source.

[0083] The source determination adopts a two-layer gate mechanism, specifically including:

[0084] In the first layer, the multimodal agent makes judgments on the video frames of each waypoint and outputs the original judgment and confidence level.

[0085] The second layer determines a valid hit if and only if the original determination is the source and the confidence level is not lower than the preset threshold. A valid hit stops subsequent point-by-point source determination.

[0086] In this step, the drone flies along the selected patrol route, point by point: upon reaching each waypoint, the event cache is cleared, and then the drone flies to the waypoint in a manner that involves "waiting to return to manual mode, then issuing flight commands, and retrying a preset number of times if rejected" (_fly_to_point_ready→fly_to_point→wait_fly_to_done, flight timeout 120s). After reaching the waypoint, the drone waits for the image transmission to stabilize (e.g., 5 seconds), and then extracts real-time frames from the backend proxy (path in the form of / manage / api / v1 / live / streams / whep / {drone SN-payload index}) via the WHEP protocol. The multimodal vision model determines the source of this frame and outputs the original judgment is_source, confidence, scene description scene_description, evidence, recommendation, and target pixel position target_position{pixel_x, pixel_y, confidence}. This embodiment employs a two-layer gate mechanism: the first layer is the model's original determination `raw_is_source`; the second layer is the valid determination `is_source_found` after passing a confidence threshold (where `is_source` and `confidence` ≥ the source confidence threshold). In this embodiment, the threshold is set to 0.90. The source of the event is determined to be found and subsequent point-by-point source determination stops only when there is a valid hit (confidence ≥ 0.90). The source determination prompts emphasize "only looking at directly visible evidence, not guesswork." For example, a car crash requires collision marks and that location must be the starting point of the accident; spilled material must be the initial accumulation point leading to subsequent congestion; and congestion requires a clear boundary between sparse and dense areas and the upstream blocking point. It also follows the principle of "better to miss a report than to make a false one"—it is precisely through this semantic-level discrimination that the UAV can proactively locate the true source of the event along the flight path, outside the field of view of the fixed monitoring camera.

[0087] After a source hit is detected, the system calculates the ground geographic coordinates of the event source from the pixel coordinates of the target in the hit frame. The geometric model is as follows: In the camera's East-North-Sky (ENU) coordinate system, the optical axis direction forward = (cos(pitch)·sin(head), cos(pitch)·cos(head), sin(pitch)) is constructed based on the gimbal's pitch and azimuth angles. A spatial ray is constructed by combining the offset of the target pixel relative to the image center with the camera's effective field of view. This ray intersects the ground (z=0), and the intersection parameter t = flight altitude / (−ray.z) yields the ground location of the source. The camera's effective field of view is restored through zoom cropping and then calculated using actual ground coverage calibration at a height of 100m. To improve robustness, spatial rays can be constructed for frames acquired from at least two different observation locations. When the convergence angle of each ray in the horizontal direction is not less than a preset threshold (e.g., 8°) and the intersection residual does not exceed a preset threshold (e.g., 15m), the source coordinates are determined by triangulation of each ray. Since triangulation directly finds the intersection point of the spatial rays, it is naturally independent of the relative terrain height and pitch constant, thus immune to the same-direction offset introduced by terrain and pitch angle in single-frame ranging.

[0088] S6: When the source is determined to be the real source of the event, the drone locks onto the source and gradually approaches it. After approaching, it takes multi-directional surround shots to collect evidence. The collected on-site images and videos are then associated with the event identification information, stored, and sent back to the business management platform.

[0089] During the locking and approach phase, the position of the source target pixel in the hit frame is aligned with the center of the image by the camera. The drone flies along the current gimbal orientation with a preset step size, re-detecting and re-aligning the position of the source target pixel with the center of the image at each step, until the gimbal tilt angle reaches the preset tilt angle threshold. The drone then stops, collects evidence on the spot, and returns to base. If, during the approach process, the approach distance limit is reached or the source target is lost continuously, and the preset number of retries is reached, the drone collects evidence on the spot and returns to base.

[0090] During the surround shooting phase, multiple surround shooting waypoints are generated at preset angles and with the source as the center. The drone takes pictures of each point in a surround shooting manner. The drone first points to the center of the source, then re-detects and aligns the target with the center of the image, and then switches to telephoto zoom to take pictures, thus obtaining evidence images from multiple directions.

[0091] The retention and transmission uses the event unique code as the key: uploading on-site evidence images and videos from each stage to object storage and assembling them into an accessible resource address;

[0092] Throughout the mission, the drone's real-time location trajectory is collected and transmitted back at a preset frequency. When the drone's telemetry remains silent for more than a preset duration, the drone's location telemetry is switched to be the trajectory source.

[0093] At each stage of acceptance, takeoff, point-by-point source identification, surrounding evidence collection, and mission completion, the system sends back analysis, source identification, and mission status messages carrying the unique event code to the business control platform.

[0094] The obtained on-site images and videos are then associated with the event identification information, stored, and sent back to the business management platform.

[0095] In this step, see Figure 3 and 4 This invention replaces the traditional "circling around the geometric center" with "pointing and locking → gradually approaching along a reliable orientation → stopping when the depression angle reaches the target → shooting around in four directions with a small radius". The approach phase is driven by a state machine (default parameters: pitch threshold −50°, single step size 40m, maximum approach distance 200m, 3 retries if target is lost): First, the camera is pointed to and aligned (camera_aim, x, y∈[0,1]) using the normalized pixel coordinates of the target in the hit frame to center the target in the frame and lock it; then, the process is repeated in a loop—realigning and centering, reading the measured gimbal pitch and azimuth angles in the UAV OSD and making a decision: if the target is visible and the gimbal pitch reaches the threshold (pitch≤−50°), the target is determined to be in position and the approach stops; if the target is visible but not in position and the cumulative approach distance has not reached the maximum, the target moves forward one step along the current gimbal azimuth (altitude remains unchanged) and is re-detected; if the target is lost, the approach is retried, and if the approach fails after 3 retries, a set of evidence images is taken on the spot and the return process is returned, and the blind flight is stopped; if the cumulative approach distance reaches the maximum, the approach stops.

[0096] Once in position, a reliable panoramic center is recalculated using a new frame indicating "target centered, tilt angle steep" (the geometric center of the source waypoint is retrieved when effective pixel backprojection is unavailable). This yields the panoramic center coordinates, flight ellipsoid height (the current flyable ellipsoid height is used, defaulting to 30m when relative ground height is unknown), and ground observation height (flight ellipsoid height minus relative ground height; if too low, it is downgraded to ensure gimbal pointing is legitimate). Subsequently, four panoramic waypoints (east, south, west, and north) are generated at 90° intervals around the event source, based on the panoramic radius (30m in this embodiment). The drone flies to each point sequentially: first, it calls the camera pointing center (camera_look_at, to lock yaw alignment) for coarse pointing; then, it extracts frames for re-detection and precisely centers the camera using camera_aim; then, it switches to a telephoto lens and zooms (7x zoom) to capture evidence images. After capturing, it resets to a wide-angle lens for the next point detection, thus obtaining clear license plate-level evidence in four directions around the event source.

[0097] Throughout the entire process, the unique event identifier evtId is used as the key for data retention and transmission. The transmission service sends messages carrying evtId to the business control platform at each stage: acceptance (including rejection during busy hours), takeoff, point-by-point source determination, completion of surround-sight evidence collection, landing and cabin entry, and mission completion. Specifically, during the acceptance / rejection stage, a rejection notification is sent; during takeoff, a stage message containing telemetry snapshots is sent; during point-by-point source determination, a source determination event (including waypoint number, valid source determination conclusion is_source_found, confidence level, and source determination image URL) is sent; during surround-sight evidence collection completion, a surround-sight completion event is sent; and upon mission completion, a final message containing complete telemetry snapshots, all evidence image URLs, video URLs, and a mission summary is sent. The source image and panoramic image are uploaded to object storage (MinIO, endpoint http: / / 192.168.0.1, bucket dji-cloud, object key in the form of missions / {evtId} / {filename}, signed and authenticated), and assembled into an accessible URL (in the form of http: / / 192.168.0.1 / dji-cloud / missions / {evtId} / {filename}) and written to the imageurl field. Throughout the mission, the real-time location trajectory of the UAV is transmitted back at a frequency of approximately 2Hz. When the UAV telemetry is silent for more than a preset time (e.g., 5 seconds), it automatically switches to using the pod telemetry as the trajectory source, and continues to transmit during the grace period after landing and entering the cabin (e.g., 10 seconds) to cover the UAV power outage process. The above three types of transmissions use the topics uav / mission / report (stage / final message), uav / mission / track (real-time trajectory), and uav / mission / event (analysis / source event), respectively. Finally, the drone performs a return-to-home operation and polls for confirmation of entry into the cabin (return_home→wait_drone_in_dock, return-to-home timeout 240s). The mission serial lock is then released, and the system can accept the next event.

[0098] Example 2, combined with Figure 7 A closed-loop evidence collection system for drone traffic incidents based on multimodal intelligent agents and cross-network relay includes:

[0099] The SMS cross-network relay module, located on the public network side, is used to receive event SMS messages from the dedicated transportation network via a cellular communication module on the public network side, parse them into structured alarms carrying unique event codes and location coordinates, and publish them via message middleware. The cross-network relay channel is a cellular SMS channel. The cross-network relay module receives SMS messages via AT commands through a cellular communication module connected via a serial port, completes UCS2 decoding and long SMS concatenation, parses them into structured alarms carrying unique event codes and location coordinates, and publishes them via message middleware using the unique event code as the key.

[0100] The event analysis intelligent agent module has a built-in multimodal intelligent agent for filtering false alarms, classifying events, and determining whether a drone response is required for structured alarms.

[0101] The route selection module is used to extract road number, station number and direction from event information, and select patrol routes by looking up the combination of these information in a table.

[0102] The flight control module is used to perform coordinate safety verification before the UAV takes off, control the UAV to take off automatically, fly point by point along the patrol route, align the gimbal and zoom to take pictures, and return to the cabin. The flight control module adopts a single-machine serial lock to ensure that only one flight mission is executed at a time, and performs three safety guardrails before takeoff: coordinate legality verification, operation radius verification and flight altitude clamping.

[0103] The point-to-point source identification module is used to extract frames from the wide-angle image transmission of each waypoint, and the multimodal vision model performs source identification through two layers of gates. It also calculates the geographic coordinates of the event source from the pixel coordinates of the hit frame (and can improve robustness through multi-frame triangulation).

[0104] The approach and surround evidence collection module is used to lock onto the source of the event by pointing the camera, gradually approaching it along the pan-tilt-zoom direction until the downward angle reaches the target, and perform four-way surround zoom evidence collection on the source; it includes guardrails such as approach limit and on-site evidence collection and return to home if the tracking is lost.

[0105] The evidence retention and transmission module is used to store the on-site evidence uploaded at each stage and assemble an accessible URL, transmit the real-time trajectory at approximately 2Hz (including the source switching between UAV and nest telemetry), and transmit judgment, source determination and task status messages carrying the unique event code at each stage.

[0106] The road monitoring video analysis system is set up on a dedicated traffic network, while the cross-network relay module, event analysis intelligent agent module, flight control module, and UAV are set up on the public network. The two are connected through the cross-network relay channel to trigger events across networks.

[0107] Example 3: This example demonstrates in detail how the present invention outputs specific source identification results, scenario descriptions, and reasoning basis through structured (JSON) data in practical applications, and explains how the two-layer gate mechanism suppresses false alarms.

[0108] Scenario 1: Identification of the source of the event (positive example)

[0109] See Figure 5When the drone hits the actual accident source point by point along the flight path, the multimodal vision model returns `is_source` as true for that frame, and the original judgment `raw_is_source` is also true, with a confidence level as high as 0.95. The scene description is "two vehicles rear-end collision, right lane", the evidence is "obvious collision marks, vehicle damage", the recommendation is "source found", and the target pixel position is {pixel_x: 2800, pixel_y: 900, confidence: 0.85}. Since the confidence level is not lower than the threshold of 0.90, it is determined by the second-level gate to be a valid hit of the event source (`is_source_found` is true). Based on this, the system enters the approximation and four-way panoramic evidence collection process, and uploads the panoramic evidence image to the object storage and writes it to the imageurl using the event's unique code as the key.

[0110] Scenario 2: False Alarm Elimination (Counterexample)

[0111] See Figure 6 After the drone extracts frames at a certain waypoint, the model returns `is_source` as false and `confidence` as 0.40 for a suspected target. The scene description is "construction enclosure, vehicles parked in an orderly manner," the evidence is "no collision traces, temporary control of the construction area," the suggestion is "continue tracing the source," and the target pixel position `target_position` is null. Since this frame does not meet the second-layer gate condition of "originally determined as the source and confidence ≥ 0.90," the system determines `is_source_found` as false, automatically blocking the alarm and not triggering evidence collection. This effectively solves the technical problem of easily misjudging scenes such as parked vehicles in construction areas as accidents. Therefore, the two-layer gate mechanism separates the "original model judgment" from the "effective judgment after confidence threshold," reducing the false alarm rate while preserving semantic understanding capabilities.

[0112] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A closed-loop evidence collection method for unmanned aerial vehicle (UAV) traffic incidents based on multimodal intelligent agents and cross-network relay, characterized in that, Includes the following steps: S1. The road monitoring video analysis system deployed in the traffic private network identifies abnormal events in the road video stream, generates alarm information containing event location information, and transfers the alarm information from the traffic private network to the drone dispatch system deployed in the public network via the cellular SMS channel. S2. The multimodal intelligent agent of the UAV dispatch system performs event analysis on alarm information to determine whether UAV response is required. When it is determined that UAV response is required, the multimodal intelligent agent selects the corresponding patrol route based on the event location information. S3. Trigger the drone to automatically take off with the location indicated by the event location information as the target point, and fly along the selected patrol route. During the flight, video frames are collected, and the multimodal intelligent agent judges the source of each video frame to identify the source of the event. S4. When the source of the incident is determined, the drone locks onto the source and takes multi-directional surround shots to collect evidence. S5. After associating the obtained on-site images and videos with the alarm information, save them and send them back to the business management platform.

2. The method according to claim 1, characterized in that, In S1, The SMS sending terminal within the dedicated transportation network encodes the alarm information into an SMS message and sends it out. The SMS receiving gateway on the public network side receives SMS messages via a cellular communication module connected through a serial port using AT commands. It decodes the UCS2 encoded text, reassembles long SMS messages into fragments according to the sender, and parses them into structured data containing at least a unique event code, WGS84 coordinates arranged in longitude first and latitude last, station number and coordinate mapping, event text, and sending time. The data is then published via message middleware using the unique event code as the key. The event location coordinates are arranged in a longitude first and latitude last format.

3. The method according to claim 1, characterized in that, The multimodal agent is configured as an agent based on a large language model. The multimodal agent analyzes the alarm information, confirms and outputs the event type, event severity, event confidence level, whether a drone response is required, and the reason for the analysis. The event confidence level represents the credibility of the alarm information corresponding to the real traffic event. The conditions for determining that a drone response is required are: the severity of the event reaches a preset level, the alarm information contains valid location coordinates, the alarm information is not an obvious false alarm, and the event needs to be verified or evidence collected on-site. All four conditions must be met simultaneously; otherwise, no drone response is required. Among them, when the confidence level of the event is not lower than the preset false alarm judgment threshold and the event type is not judged as a false alarm, it is judged as a non-obvious false alarm; otherwise, it is judged as an obvious false alarm. When a drone response is required, the corresponding patrol route is selected. Specifically, this involves: extracting the road number, station number, and direction from the event description and structured data, and assembling them into a route selection key; querying the preset route mapping table using the route selection key to obtain the patrol route; if the patrol route is not matched, the drone refuses to take off and issues an alarm, instead of returning to the default route.

4. The method according to claim 1, characterized in that, Before triggering the automatic takeoff of the drone in S3, a coordinate safety check is performed, including: Reorder the event location coordinates from longitude first and latitude last to latitude first and longitude last, and verify whether the reordered coordinates fall within the preset geographical range; The distance between the verification event point and the drone nest does not exceed the preset operating radius; Limit the target's flight altitude to a preset safe altitude range; If any of the above conditions fail to be verified, takeoff will be refused.

5. The method according to claim 1, characterized in that, The source determination adopts a two-layer gate mechanism, specifically including: In the first layer, the multimodal agent makes judgments on the video frames of each waypoint and outputs the original judgment and confidence level. The second layer determines a valid hit if and only if the original determination is the source and the confidence level is not lower than the preset threshold. A valid hit stops subsequent point-by-point source determination.

6. The method according to claim 1, characterized in that, In step S3, when the source of the event is determined, a spatial ray determined by pixel coordinates is constructed based on the position of the UAV, the pitch and azimuth angles of the gimbal, the field of view of the camera, and the flight altitude when the frame is acquired, and the intersection point of the ray with the ground is calculated to back-calculate the geographical coordinates of the source. Rays are constructed for frames at no less than two different observation positions, and triangulation is performed to find the intersection when the horizontal convergence angle of each ray is not less than a preset angle threshold. The result of the triangulation intersection is used as the geographical coordinates of the event source.

7. The method according to claim 1, characterized in that, In S4, during the locking and approaching phase, the position of the source target pixel in the hit frame is aligned and adjusted to the center of the screen by the camera pointing. The drone flies along the current gimbal orientation with a preset step size, re-detects and re-aligns the position of the source target pixel to the center of the screen at each step, until the gimbal tilt angle reaches the preset tilt angle threshold, at which point the drone stops, collects evidence on the spot, and returns to base. If, during the approach process, the approach distance limit is reached or the source target is lost continuously, and the preset number of retries is reached, then evidence is collected on-site and the aircraft returns to base. During the surround shooting phase, multiple surround shooting waypoints are generated at preset angles and with the source as the center. The drone takes pictures of each point in a surround shooting manner. The drone first points to the center of the source, then re-detects and aligns the target with the center of the image, and then switches to telephoto zoom to take pictures, thus obtaining evidence images from multiple directions.

8. The method according to claim 1, characterized in that, In step S5, using the event unique code as the key, on-site evidence images and videos from each stage are uploaded to the object storage and assembled into an accessible resource address. Throughout the mission, the drone's real-time location trajectory is collected and transmitted back at a preset frequency. When the drone's telemetry remains silent for more than a preset duration, the drone's location telemetry is switched to be the trajectory source. At each stage of acceptance, takeoff, point-by-point source identification, surrounding evidence collection and mission completion, the system sends back analysis, source identification and mission status messages carrying the unique event code to the business control platform. The obtained on-site images and videos are then associated with the event identification information, stored, and sent back to the business management platform.

9. A closed-loop evidence collection system for unmanned aerial vehicle (UAV) traffic incidents based on multimodal intelligent agents and cross-network relay, used to execute the method as described in any one of claims 1 to 8, characterized in that, include: The SMS cross-network relay module is set up on the public network side. It is used to receive event SMS messages from the traffic private network via the cellular communication module on the public network side, parse them into structured alarms carrying unique event codes and location coordinates, and publish them through message middleware. The event analysis intelligent agent module has a built-in multimodal intelligent agent for filtering false alarms, classifying events, and determining whether a drone response is needed for structured alarms. The route selection module is used to extract road number, station number and direction from event information, and select patrol routes by looking up the combination of these information in a table. The flight control module is used to perform coordinate safety verification before the drone takes off, control the drone to take off automatically, fly point by point along the patrol route, align the gimbal and zoom to take pictures, and return to the cabin. The point-to-point source identification module is used to identify the source of video frames at each waypoint, and to calculate the geographic coordinates of the event source from the pixel coordinates of the hit frames. The approximation and surround evidence collection module is used to locate the source of the event, gradually approach it, and perform multi-directional surround zoom evidence collection on the source. The evidence retention and transmission module is used to upload and retain on-site evidence at each stage, and transmit real-time trajectory and status messages at each stage. The road monitoring video analysis system is set up on the dedicated traffic network, while the cross-network relay module, the event analysis intelligent agent module, the flight control module, and the drone are set up on the public network. The two are connected through the cross-network relay channel to trigger events across networks.

10. The system according to claim 9, characterized in that, The cross-network relay channel is a cellular SMS channel; the cross-network relay module receives SMS messages via AT commands through a cellular communication module connected via a serial port, completes UCS2 decoding and long SMS concatenation, and parses it into a structured alarm carrying a unique event code and location coordinates, which is then published via a message middleware using the unique event code as the key; the flight control module adopts a single-machine serial lock to ensure that only one flight mission is executed at a time, and performs three safety guardrails before takeoff: coordinate validity verification, operation radius verification, and flight altitude clamping.

Images