An unmanned aerial vehicle based method and system for autonomous reconstruction of an incident scene

Abstract

This invention relates to the field of unmanned aerial vehicle (UAV) technology, and more particularly to a method and system for autonomous reconstruction of incident scenes based on UAVs. The method includes: the UAV collecting visual images and sparse point clouds while patrolling along a preset route; inputting the visual images and sparse point clouds into an accident recognition model to generate feature snapshots; the UAV generating a layered surround acquisition path based on the feature snapshots and a preset layered acquisition strategy; the UAV flying along the layered surround acquisition path to collect multimodal data frames; adaptively extracting frames from the multimodal data frames to obtain keyframes; acquiring the inter-frame velocity vector and semantic segmentation confidence of each keyframe, performing weighted fusion to obtain dynamic target regions; removing LiDAR point clouds within the dynamic target regions to obtain keyframes with removed dynamic targets, and inputting these keyframes into a 3D Gaussian splash model to generate a three-dimensional accident scene model. This invention solves the problems of traditional methods, such as reliance on manual labor, long processing times, insufficient data integrity, and inadequate modeling accuracy.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a method and system for autonomous reconstruction of event scenes based on UAVs. Background Technology

[0002] Rapid and accurate reconstruction of accident / incident scenes is crucial for liability determination, damage assessment, and subsequent handling. Traditional methods rely on manual measurement and camera photography, which suffer from low efficiency, strong subjectivity, and limited data dimensions. In recent years, drones have been gradually applied to on-site investigation, using aerial photography to assist in mapping or modeling. However, current technologies still require manual intervention in path planning, data screening, and post-modeling processing, failing to achieve full-process autonomy. Furthermore, they are susceptible to interference from moving targets (such as pedestrians and vehicles) in point clouds, leading to decreased reconstruction accuracy.

[0003] For example, patent application publication number CN121746608A (classification number G06T) provides a method and device for reconstructing traffic accident scenes; patent application publication number CN117893690A (classification number G06T) provides a device and method for three-dimensional reconstruction of UAV sequence images that integrates deep learning; and patent application publication number CN121999181A (classification number G06T) provides a method and related equipment for rapid three-dimensional reconstruction and decision support of multiple UAVs for disaster scenes.

[0004] Therefore, there is an urgent need for a method and system for autonomous reconstruction of event scenes based on drones. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the existing technology, the present invention provides an autonomous reconstruction method and system for event sites based on UAVs, which solves the problems of traditional methods relying on manual labor, long time consumption, and insufficient data integrity and modeling accuracy.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0009] In a first aspect, embodiments of the present invention provide an autonomous reconstruction method for an event scene based on unmanned aerial vehicles (UAVs), comprising:

[0010] When the UAV is inspecting along the preset route, it collects visual images and sparse point clouds at the first sampling rate, and inputs the visual images and sparse point clouds into the trained accident recognition model to generate a feature snapshot containing the accident type and the accident spatial range.

[0011] The drone generates a layered surround acquisition path, including a panoramic layer, a detail layer, and a close-up layer, based on the feature snapshot and a preset layered acquisition strategy.

[0012] The UAV flies along the layered surrounding acquisition path and acquires multimodal data frames at a second sampling rate; and performs adaptive frame extraction processing on the multimodal data frames based on changes in motion saliency and geometric complexity to obtain key frames;

[0013] The inter-frame velocity vector and semantic segmentation confidence of each key frame are obtained, and the inter-frame velocity vector and semantic segmentation confidence are weighted and fused to obtain the dynamic target region; the lidar point cloud in the dynamic target region is removed to obtain the key frame of dynamic target removal.

[0014] The keyframes for removing dynamic targets are input into the 3D Gaussian splash model to generate a three-dimensional accident scene model.

[0015] Optionally, in some embodiments of this application, the multimodal data frame includes:

[0016] RGB images, LiDAR point clouds, and IMU pose data.

[0017] Optionally, in some embodiments of this application, inputting the visual image and sparse point cloud into a trained accident recognition model to generate a feature snapshot containing the accident type and the spatial extent of the accident includes:

[0018] Based on the visual image and sparse point cloud, the accident recognition model identifies at least one abnormal target at the accident scene and outputs the accident type of the abnormal target.

[0019] Based on the visual image and sparse point cloud, the accident recognition model is used to perform spatial aggregation and boundary estimation on the abnormal target, and outputs the accident spatial range, which includes the center coordinates and spatial boundary of the core area of ​​the accident.

[0020] The accident type and the accident spatial range are fused together and encapsulated to generate the feature snapshot.

[0021] Optionally, in some embodiments of this application, the accident identification model includes:

[0022] A parallel dual-branch feature encoder includes a visual branch and a point cloud branch. The visual branch uses a lightweight network based on temporal convolution to extract two-dimensional semantic features and motion change features of the visual image. The point cloud branch uses a sparse convolutional network to extract three-dimensional structural features and local ensemble distortion features of the sparse point cloud.

[0023] A cross-modal dynamic fusion module is used to associate the two-dimensional semantic features with the three-dimensional structural features under spatial coordinate mapping and generate a fused feature map based on an attention mechanism.

[0024] The temporal memory aggregation unit is used to receive fused feature maps from N consecutive time points, model the evolution of the accident scenario on the time axis through a gated loop mechanism, and output the accident probability field and dynamic change field.

[0025] The multi-task output head includes an accident type classification branch, a spatial range regression branch, and a confidence assessment branch, which are used to obtain the accident type, the accident spatial range, and the comprehensive confidence based on the accident probability field and the dynamic change field, respectively.

[0026] Optionally, in some embodiments of this application, the preset layered acquisition strategy includes a panoramic layer strategy, a detail layer strategy, and a close-up layer strategy.

[0027] The panoramic layer strategy is to determine the first acquisition height and the first orbital radius based on the accident space range in the feature snapshot, and generate a panoramic layer flight path.

[0028] The detailed layer strategy adaptively determines multiple hovering points based on the accident type of the feature snapshot in order to generate a detailed layer flight path.

[0029] The close-up layer strategy is to use the center coordinates of the core area of ​​the accident as the acquisition point and generate a close-up layer approach path.

[0030] The panoramic layer strategy, detail layer strategy, and close-up layer strategy are executed sequentially.

[0031] Optionally, in some embodiments of this application, the multimodal data frames are subjected to adaptive frame extraction processing based on changes in motion saliency and geometric complexity to obtain keyframes, including:

[0032] Acquire the RGB images of the current frame and the previous frame, and calculate the optical flow field between the two frames; statistically analyze the optical flow amplitude of each pixel in the optical flow field, and use the average or median of the optical flow amplitude as the first motion saliency measure of the current frame;

[0033] Acquire IMU pose data of the current frame and the previous frame, calculate the change in angular velocity between the two frames, and use the change in angular velocity as the second motion saliency measure.

[0034] The lidar point cloud of the current frame is acquired, the lidar point cloud is divided into voxel grids, the distribution of the number of points in each voxel grid is statistically analyzed, the information entropy of the distribution of the number of points is calculated, and the information entropy is used as the geometric complexity measure of the current frame.

[0035] Keyframes are obtained based on the first motion saliency measure, the second motion saliency measure, the geometric complexity measure, and preset judgment rules.

[0036] Optionally, in some embodiments of this application, obtaining keyframes based on the first motion saliency measure, the second motion saliency measure, the geometric complexity measure, and preset judgment rules specifically includes:

[0037] The first motion saliency measure and the second motion saliency measure are weighted and fused to obtain a comprehensive motion saliency measure;

[0038] Construct a two-dimensional state space with the comprehensive motion saliency measure and the geometric complexity measure as the horizontal and vertical axes, respectively;

[0039] A keyframe trigger region is preset in the two-dimensional state space, and the boundary of the keyframe trigger region is defined by a nonlinear function.

[0040] Calculate the coordinates of the current frame in the two-dimensional state space. When the coordinates fall within the keyframe trigger area, mark the current frame as a keyframe.

[0041] Optionally, in some embodiments of this application, obtaining the inter-frame velocity vector and semantic segmentation confidence of each keyframe, and weightedly fusing the inter-frame velocity vector and semantic segmentation confidence to obtain the dynamic target region includes:

[0042] A three-dimensional voxel grid is constructed, and the LiDAR point cloud of each key frame is projected onto the three-dimensional voxel grid. The average velocity vector is obtained by averaging the inter-frame velocity vectors of the points in each voxel grid, and the highest confidence score is obtained by taking the maximum value of the semantic segmentation confidence score of the points in each voxel grid.

[0043] The average velocity vector is normalized and then weighted and summed with the highest confidence level to obtain the dynamic probability of each voxel grid, thereby generating a dynamic target probability field.

[0044] The region growing algorithm is used to take grids with dynamic probabilities exceeding a preset seed threshold as seed points and expand them outward to the point where the dynamic probabilities are below the preset boundary threshold, thereby generating one or more dynamic target regions.

[0045] Optionally, in some embodiments of this application, removing the lidar point cloud within the dynamic target area and obtaining the keyframe for removing the dynamic target specifically includes:

[0046] The lidar point cloud within each dynamic target area is clustered into the corresponding dynamic target. Motion pattern consistency analysis is performed on the inter-frame velocity vectors of all points within the dynamic target. If the velocity vectors are radially distributed, they are identified as expansion-type dynamic targets; if they are vortex-type distributed, they are identified as rotation-type dynamic targets; if the directions are consistent, they are identified as translation-type dynamic targets.

[0047] The lidar point clouds of the dilatational and rotational dynamic targets are removed by morphological erosion, and the lidar point clouds of the translational targets are removed by edge feathering, thus obtaining the key frames for removing the dynamic targets.

[0048] Optionally, in some embodiments of this application, inputting the keyframes for removing dynamic targets into the 3D Gaussian splash model to generate a three-dimensional accident scene model specifically includes:

[0049] Using the RGB image, LiDAR point cloud, and IMU pose data from the keyframes for removing dynamic targets as input, a 3D Gaussian ellipsoid is initialized; the 3D Gaussian ellipsoid is then projected onto the image plane using a differentiable renderer to generate a rendered image.

[0050] Calculate the loss between the rendered image and the input RGB image, and iteratively optimize the parameters of the 3D Gaussian ellipsoid based on backpropagation of the loss;

[0051] When the iteration converges or reaches the preset number of iterations, a three-dimensional accident scene model is output.

[0052] Secondly, embodiments of the present invention provide an autonomous event scene reconstruction system based on unmanned aerial vehicles (UAVs), comprising:

[0053] Unmanned aerial vehicles (UAVs), multi-sensor modules, edge computing units, and ground control terminals;

[0054] The ground control terminal is electrically connected to the UAV, the multi-sensor module, and the edge computing unit.

[0055] The multi-sensor module and edge computing unit are integrated on the UAV, enabling the UAV to perform any of the above-described UAV-based autonomous event scene reconstruction methods.

[0056] (III) Beneficial Effects

[0057] This invention provides an autonomous event scene reconstruction method based on unmanned aerial vehicles (UAVs). Its advantages lie in the following: by organically combining accident identification during UAV patrols along a preset route with a layered surround acquisition strategy, intelligent, refined, and efficient 3D reconstruction of the accident scene is achieved. First, the method uses an accident identification model to generate feature snapshots, guiding the UAV to automatically adjust its acquisition path, avoiding the blindness and repetitive work of traditional manual flight. Second, by using layered surround acquisition (panoramic, detailed, and close-up layers) and adaptive frame extraction to reduce redundant data, the method significantly improves the targeting of data acquisition and storage / computing efficiency. Third, by weighted fusion of inter-frame velocity vectors and semantic segmentation confidence, dynamic targets (such as vehicles and pedestrians) are accurately eliminated, effectively eliminating the interference of moving objects on static scene reconstruction and ensuring the purity of point cloud data. Finally, a high-fidelity 3D accident scene model is generated using a 3D Gaussian splash model. This model can quickly output structurally complete, textured, and geometrically accurate static scene reconstruction results without relying on dynamic targets, providing reliable 3D digital support for on-site investigation, situation assessment, and subsequent processing. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating an embodiment of an unmanned aerial vehicle (UAV)-based autonomous event scene reconstruction method according to this application.

[0059] Figure 2 This is a schematic diagram illustrating the process of generating a feature snapshot according to one embodiment of this application;

[0060] Figure 3 This is a schematic diagram of the process of obtaining keyframes according to an embodiment of this application;

[0061] Figure 4 This is a schematic diagram of the process of obtaining keyframes for proposing dynamic targets according to one embodiment of this application;

[0062] Figure 5 This is a schematic diagram of removing dynamic targets according to one embodiment of this application. Detailed Implementation

[0063] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] In current related technologies, accident scene investigation mainly relies on manual operation of drones or handheld devices for taking photos, measuring and labeling. Although some solutions introduce drone aerial photography to assist in mapping, it is still necessary to manually plan flight paths, select shooting angles, filter data and manually stitch together models. The whole process takes several hours or even days. Moreover, the collection perspective is singular and is easily affected by point cloud interference from moving targets (such as pedestrians and temporary vehicles), resulting in insufficient accuracy of the 3D reconstruction model and fragmented process, which makes it difficult to meet the needs of rapid response and full autonomy in emergency scenarios.

[0065] The core technical problem this invention aims to solve is: how to achieve full automation of the entire process from autonomous accident scene localization, multi-view data acquisition, intelligent keyframe filtering, dynamic target point cloud removal to high-precision 3D modeling, thereby eliminating reliance on manual intervention and improving survey efficiency and reconstruction quality. To address this problem, the technical solution proposed in this invention includes: when a UAV patrols along a preset route, it actively identifies the accident type and spatial range through joint visual and radar analysis, generating feature snapshots; automatically generating panoramic, detail, and close-up surround paths based on a layered acquisition strategy; during the acquisition process, adaptive frame extraction based on motion saliency and geometric complexity changes is performed on multimodal data frames to obtain keyframes; further, by weighted fusion of inter-frame velocity vectors and semantic segmentation confidence, the LiDAR point cloud of dynamic target areas is accurately identified and removed; finally, the keyframes after removing dynamic targets are input into a 3D Gaussian splash model to generate a high-fidelity 3D accident scene model.

[0066] By adopting the above scheme, the present invention can realize end-to-end autonomous accident scene reconstruction: no need for manual path planning, data screening or dynamic interference processing, the time for on-site investigation and modeling can be shortened; information integrity is guaranteed by layered multi-view acquisition; dynamic target point cloud elimination technology reduces static scene reconstruction error, significantly improves reconstruction accuracy and automation level, and provides reliable technical support for rapid, objective and refined on-site investigation.

[0067] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0068] Figure 1 This invention provides an embodiment of an autonomous event scene reconstruction method based on unmanned aerial vehicles (UAVs), such as... Figure 1 As shown, the autonomous reconstruction method includes:

[0069] Step S100: When the UAV is inspecting along the preset route, it collects visual images and sparse point clouds at the first sampling rate, and inputs the visual images and sparse point clouds into the trained accident recognition model to generate a feature snapshot containing the accident type and the accident spatial range.

[0070] Specifically, when performing inspection missions, the drone flies along a pre-set trajectory. During the inspection process, the drone simultaneously acquires visual image data and sparse point cloud data at a first sampling rate. The visual image acquisition system uses a high-definition camera to continuously capture images of the work area during the drone's flight. The sparse point cloud acquisition system uses an airborne LiDAR, which emits laser pulses and receives reflected signals to obtain spatial three-dimensional point distribution information of the work area. The visual image acquisition and sparse point cloud acquisition use a unified clock for time synchronization, ensuring that the visual images and sparse point clouds acquired at the same time are consistent in the time dimension.

[0071] Furthermore, the acquired visual images and sparse point cloud data are preprocessed before being input into the trained accident recognition model. Preprocessing operations include: size normalization, pixel value standardization, and data augmentation for the visual images; spatial downsampling, coordinate normalization, and wavelet denoising for the sparse point clouds. The preprocessed multimodal data are then fed into different processing branches of the accident recognition model.

[0072] Further, see Figure 2 Step S100 specifically includes:

[0073] Step S110: Based on the visual image and sparse point cloud, identify at least one abnormal target in the accident scene using the accident recognition model, and output the accident type of the abnormal target;

[0074] Step S120: Based on the visual image and sparse point cloud, the abnormal target is spatially aggregated and its boundaries are estimated using the accident recognition model, and the accident spatial range is output. The accident spatial range includes the center coordinates and spatial boundaries of the core area of ​​the accident.

[0075] Step S130: Merge the accident type with the accident spatial range and encapsulate to generate the feature snapshot.

[0076] Furthermore, the accident identification model includes:

[0077] A parallel dual-branch feature encoder includes a visual branch and a point cloud branch. The visual branch uses a lightweight network based on temporal convolution to extract two-dimensional semantic features and motion change features of the visual image. The point cloud branch uses a sparse convolutional network to extract three-dimensional structural features and local set distortion features of the sparse point cloud.

[0078] The visual branch employs a temporal convolutional network, taking as input the most recent N consecutive frames (N=3~5) of grayscale or color images. Each frame is downsampled to a fixed resolution (e.g., 320×240). The temporal convolutional network consists of multiple layers of dilated convolutions, each layer performing convolution operations on different receptive fields along the temporal dimension, thereby simultaneously capturing short-term motion abrupt changes (e.g., sudden vehicle stops) and long-term scene evolution (e.g., smoke diffusion). Compared to common 3D convolutions, temporal convolutional networks have fewer parameters. This branch outputs two feature maps: a 2D semantic feature map (1 / 8 the size of the original image) and a motion change feature map (a heatmap representing the change in optical flow amplitude between adjacent frames).

[0079] The point cloud branch employs a sparse convolutional network, with the input being the sparse point cloud corresponding to the current time step (the number of points is controlled within 1000). Since critical deformations in accident scenarios often occur in localized areas, this embodiment introduces local geometric distortion detection into the sparse convolutional network: for each point, the deviation of its normal vector from that of other points in its neighborhood and the rate of curvature change are calculated. When the local curvature of a point is significantly greater than that of its surrounding points (exceeding a preset multiple), it is marked as a potential distortion point. The point cloud branch outputs a geometric distortion feature vector of the same dimension as the point cloud, which is concatenated with the 3D structural features extracted by the sparse convolution, thereby enhancing the model's sensitivity to geometric anomalies such as collisions, rollovers, and road surface cracking.

[0080] A cross-modal dynamic fusion module is used to associate the two-dimensional semantic features with the three-dimensional structural features under spatial coordinate mapping and generate a fused feature map based on an attention mechanism.

[0081] Specifically, a stationary-guided attention fusion mechanism is employed: First, using a known camera projection matrix, the 3D coordinates of each point in the point cloud are projected onto the image plane to obtain the corresponding "stationary point" pixel coordinates. For each stationary point, a 2D feature vector within its neighborhood is extracted from the image feature map, and simultaneously, a 3D structural feature vector for that point is extracted from the point cloud feature map. These two vectors are then input into a lightweight cross-attention unit to calculate the fusion weights of the cross-modal features at that stationary point. Non-stationary regions obtain fused features through interpolation or propagation. Finally, a fused feature map of the same size as the original image feature map is generated, where each location incorporates available 3D structural information (if a stationary point exists nearby) or purely 2D semantic information (if no corresponding depth exists).

[0082] To improve fusion efficiency, this embodiment also introduces adaptive sparse attention: First, it combines the accident area of ​​the previous frame (expanded by 1.5 to 2 times), the area with significant motion changes in the current frame, and the area with significant geometric distortion of the point cloud (projected onto the image and expanded), and takes the union to obtain candidate accident areas. Then, stationary-guided cross-modal attention fusion is performed only within the candidate areas; outside the areas, visual features are used directly to skip attention calculation, thereby reducing computational load. If the area of ​​the candidate region exceeds 70% of the entire image, it automatically degenerates into dense attention across the entire image; if it is less than 10%, it is further simplified to a lightweight fusion method such as linear weighting, achieving dynamic adaptive control.

[0083] The temporal memory aggregation unit is used to receive fused feature maps from N consecutive time points, model the evolution of the accident scenario on the time axis through a gated loop mechanism, and output the accident probability field and dynamic change field.

[0084] This embodiment employs a variant of the gated recurrent unit—the residual gating mechanism—to perform temporal modeling on fused feature maps from multiple consecutive time points. This unit receives fused feature maps from N consecutive time points (N=3~5) as input and uses a gated recurrent mechanism (such as convolutional GRU or ConvLSTM) to independently update the temporal state at each spatial location, thereby modeling the evolution of the accident scene along the time axis. Specifically, the fused feature map at the current time point and the hidden state passed from the previous time point pass through a reset gate and an update gate to generate a new hidden state. This new hidden state is then passed through an output convolutional layer to produce two results: one is an accident probability field (with values ​​ranging from 0 to 1, representing the probability that each pixel belongs to the accident area), and the other is a dynamic change field (representing the cumulative difference between the probability fields of adjacent time points, used to indicate dynamic processes such as fire spread and vehicle displacement). This unit can capture the complete temporal pattern of an accident from its inception, outbreak, to its decay, and output a spatiotemporal representation that can be used for downstream tracking or early warning.

[0085] The multi-task output head includes an accident type classification branch, a spatial range regression branch, and a confidence assessment branch, which are used to obtain the accident type, the accident spatial range, and the comprehensive confidence based on the accident probability field and the dynamic change field, respectively.

[0086] Specifically, the accident type classification branch: the fused features (after global average pooling) corresponding to the highest probability region in the accident probability field are input into a multilayer perceptron, and a multidimensional vector is output. After softmax, the probability distribution of various accident types is obtained, and the highest probability category is taken as the accident type.

[0087] Spatial extent regression branch: Using the accident probability field as spatial weights, it calculates the probability-weighted first moment (center coordinates) and second moment (spatial scatter matrix), thereby deriving the center coordinates of the accident core region, the major and minor axes of the elliptical boundary, and the orientation angle. This branch does not rely on a preset anchor frame and can adapt to accident extents of arbitrary shapes.

[0088] Confidence assessment branch: Taking into account the predicted probability entropy of the classification branch, the boundary convergence of the spatial regression branch, and the consistency of the hidden state of the sequence in the temporal memory unit, a comprehensive confidence score between 0 and 1 is output. When the comprehensive confidence score is lower than a preset threshold (e.g., 0.6), the model does not generate a feature snapshot, but marks it as an "event to be confirmed," triggering a subsequent secondary fine-grained data collection process.

[0089] Through the above end-to-end model design, UAVs can analyze images and sparse point clouds in real time at the first sampling rate during routine inspections, and directly output the accident type, spatial range and confidence level. The whole process does not require dense 3D reconstruction or high-precision map assistance.

[0090] Specifically, step S100 above can be explained in detail through the following steps:

[0091] The currently acquired visual image and its corresponding sparse point cloud are input into the trained accident recognition model. The model first processes the image and point cloud separately using a parallel dual-branch feature encoder. Specifically, the visual branch employs a lightweight temporal convolutional network, simultaneously inputting a sequence of several consecutive frames (e.g., 5 frames). This extracts two-dimensional semantic features from each frame (such as the category and contour of targets like vehicles, people, and road damage), and uses inter-frame temporal convolution to capture motion change features (such as sudden vehicle deceleration, crowd gathering, and smoke diffusion). Simultaneously, the point cloud branch uses a sparse convolutional network to process the corresponding 3D point cloud, extracting 3D structural features (such as ground feature height, slope, and obstacle size) and local geometric distortion features (such as vehicle deformation after a collision and irregular geometric changes like guardrail bending). The output features of both branches are then mapped using spatial coordinates—projecting the 3D point cloud onto the image plane using camera intrinsics and UAV pose, so that each pixel region in the image can be associated with corresponding depth information—and then stitched together or aligned element-wise in the image coordinate system to form initial fused features. Subsequently, the cross-modal dynamic fusion module in the accident recognition model utilizes an attention mechanism to correlate and weight stationary points (i.e., key points corresponding to the same physical location in the image and point cloud) in two-dimensional semantic features and three-dimensional structural features: for spatially well-aligned regions, the attention weight is higher, and the fused features are strengthened; for regions with large spatial alignment deviations (possibly due to sparse point clouds or motion blur), the attention weight is reduced, and the fused features are dominated by the more reliable side. In this way, the model can accurately identify at least one abnormal target at the accident scene and output its specific accident type, such as "rear-end collision," "person falling," "road collapse," or "initial fire."

[0092] After identifying anomalous targets, the model further performs spatial aggregation and boundary estimation. First, the pixel positions of each anomalous target in the image are back-projected into 3D space using depth information to obtain a set of candidate coordinate points in the global coordinate system. Then, a density-based spatial clustering algorithm is used to aggregate the point set, removing outlier noise points to obtain a point cluster of the accident core area. The minimum convex hull or rotated rectangle boundary of this point cluster is estimated as the accident spatial range. The model outputs two parts: the center coordinates of the accident core area (e.g., 3D geographic coordinates) and the spatial boundary (e.g., the major axis, minor axis, and orientation angle of an ellipse with the center as the origin, or the six face parameters of an axis-aligned bounding box). To improve the robustness of boundary estimation, this embodiment introduces multi-frame temporal smoothing: the boundary estimated in the current frame is weighted and averaged with the boundary results of the previous three frames, with the weight proportional to the detection confidence of each frame, thereby effectively suppressing boundary jitter caused by single-frame point cloud fluctuations.

[0093] The accident type identified in step S110 and the accident spatial range estimated in step S120 are fused and encapsulated into a structured feature snapshot. This feature snapshot includes: an accident type label, a geometric description of the spatial range, a timestamp of the current moment, the current position and attitude of the UAV, and a comprehensive confidence score (provided by the subsequent multi-task output head). The feature snapshot has a very small data size (typically less than 1KB) and can be transmitted in real time to the ground command center or adjacent UAVs via the UAV data link, providing immediate alerts and preliminary situational awareness for subsequent emergency response.

[0094] This embodiment achieves real-time accident detection and spatial range estimation during routine UAV inspections through a lightweight accident recognition model and a dual-mode sampling strategy, without relying on post-processing at a ground station. The parallel dual-branch encoder integrates two-dimensional motion changes and three-dimensional geometric distortions, significantly improving the sensitivity to physical anomalies such as collisions and deformations. The cross-modal dynamic fusion module, combined with stationary point-guided attention, solves the feature alignment problem between sparse point clouds and images. The temporal memory aggregation unit utilizes residual gating and motion saliency factors to ensure rapid response and smooth inference of accident evolution. The multi-task output head directly generates structured feature snapshots, facilitating real-time transmission and decision-making. Simultaneously, the adaptive secondary fine-tuning acquisition mechanism intelligently switches between low-cost conventional mode and high-precision reconstruction mode, balancing endurance and reconstruction quality. In summary, this embodiment provides an efficient, autonomous, and robust event scene reconstruction solution, particularly suitable for low-altitude UAV applications such as urban security, traffic patrol, and disaster emergency response.

[0095] Step S200: The UAV generates a layered surround acquisition path including a panoramic layer, a detail layer, and a close-up layer based on the feature snapshot and the preset layered acquisition strategy.

[0096] The preset layered acquisition strategies include panoramic layer strategy, detail layer strategy, and close-up layer strategy.

[0097] The panoramic layer strategy is to determine the first acquisition height and the first orbital radius based on the accident space range in the feature snapshot, and generate a panoramic layer flight path.

[0098] The detailed layer strategy adaptively determines multiple hovering points based on the accident type of the feature snapshot in order to generate a detailed layer flight path.

[0099] The close-up layer strategy is to use the center coordinates of the core area of ​​the accident as the acquisition point and generate a close-up layer approach path.

[0100] The panoramic layer strategy, detail layer strategy, and close-up layer strategy are executed sequentially.

[0101] In this embodiment of the invention, step S200 is further refined to the UAV automatically generating a layered surround acquisition path including a panoramic layer, a detail layer, and a close-up layer based on feature snapshots and a preset layered acquisition strategy, so as to cover the accident scene from coarse to fine. The specific implementation method is as follows:

[0102] Step S210: Analyze the feature snapshot and extract planning parameters;

[0103] The system extracts the accident type, the center coordinates of the core accident area, the spatial extent (ellipse or bounding box size), and the overall confidence level from the feature snapshot. It then dynamically adjusts subsequent data acquisition parameters based on the drone's current status (location, remaining battery power, wind speed). If the confidence level is low, the safety margin of the panoramic layer automatically increases to cover estimation bias.

[0104] Step S220: Generate the panoramic flight path;

[0105] The panoramic layer strategy aims to acquire the overall situation of the accident scene and its surrounding environment. First, based on the accident spatial range (major axis length) and the camera's field of view, the first acquisition height and the first orbital radius are calculated to ensure the accident area is centered in the image with at least a 50% margin. If the spatial range is elongated, an elliptical orbital trajectory is generated. The flight speed is set to slow cruising (2~3 m / s) to ensure an inter-frame overlap of at least 60%. During execution, the ground coverage percentage is calculated in real time; if the coverage falls below 95% after one orbit, a rescan is automatically triggered. After the panoramic layer is completed, a panoramic orthophoto is generated as the base map for subsequent detail layers.

[0106] Step S230: Generate the flight path of the detail layer;

[0107] The detailed layer strategy performs multi-angle, refined data collection on key targets at the accident scene. Based on the accident type, a pre-set "Accident Type - Data Collection Template Library" is invoked to automatically determine the number, azimuth, and pitch angles of hovering points. For example, in a rear-end collision, two hovering points facing different directions are needed around each vehicle; in a road collapse, 3-5 hovering points are set at equal intervals along the edge. Subsequently, the shortest path sequence to all hovering points is approximated and planned. After the UAV reaches each hovering point, it automatically performs exposure bracketing and focus fine-tuning, continuously capturing multiple high-resolution images and evaluating image quality (sharpness, contrast) in real time. If the image quality is unsatisfactory, the attitude is fine-tuned and the image is retaken. If new suspicious targets are discovered through real-time images during execution, online expansion is supported—1-2 new hovering points are immediately inserted to the front of the task queue for priority collection, achieving adaptive path adjustment.

[0108] Step S240: Generate the flight path for the close-up layer;

[0109] The close-up layer strategy enables extremely close-up photography of the core area of ​​the accident. Using the accident center coordinates as the target acquisition point, a minimum safe distance is preset based on the accident type (2 meters for static collapse areas, 5 meters for fire areas). The drone decelerates and enters the field from its current position along the direction of the light source or at the azimuth angle with the best effect in the detail layer. Simultaneously, the front-facing binocular camera detects obstacles in real time. If an obstacle is detected within the safe margin, the drone automatically shifts 0.3 meters and replans the final segment of its path. If it still cannot reach the obstacle after two shifts, it retreats to a safe distance and uses a telephoto lens for shooting. For ground targets, it can also descend vertically to a height of 0.5 meters to obtain a top-down close-up. This layer supports multispectral simultaneous acquisition (near-infrared, thermal imaging) to obtain detailed information beyond visible light.

[0110] Step S250: Hierarchical collaboration and adaptive scheduling;

[0111] The three-layer paths are executed sequentially by default, but can be automatically interrupted and a larger orbital radius recalculated if the scale of the incident is found to be far greater than estimated in the panoramic layer. If a misjudgment of the incident type is found in the detail layer, the template can be dynamically switched and hovering points can be added. All paths are uploaded to the flight controller in the form of a waypoint list, supporting breakpoint resumption and task saving.

[0112] For example, the panoramic layer is collected at a height of 40-50m, with a circumference of 15-20m, a flight speed of ≤2m / s, and a pitch angle of 30°-60° to obtain the complete scene layout; the detail layer is collected at a height of 20-30m, with a circumference of 8-12m, hovering in four fixed directions (front, back, left, and right) to collect data, with a pitch angle of 0°-30° to obtain collision traces, vehicle posture, and debris distribution; the close-up layer is at a height of 5-10m, using close-up zoom photography or laser point cloud scanning to capture high-precision collision details at the center collision point or key debris.

[0113] Furthermore, for severely obstructed areas (such as under vehicles, corners of walls, and inside guardrails), additional acquisition angles or low-altitude oblique observation points are automatically added; for areas affected by dynamic obstacles, acquisition is automatically paused and a local reshoot path is replanned after the obstacles are removed; the sampling focus is automatically adjusted according to different accident types (collision accidents prioritize acquisition of contact deformation parts, and leakage or combustion accidents prioritize acquisition of leakage boundaries and diffusion directions).

[0114] Through the aforementioned layered acquisition path, the UAV achieved autonomous, safe, and efficient acquisition from the overall situation to local details and then to core close-ups, taking into account both the integrity of the coverage and high-resolution observation of key parts, significantly improving the data quality and acquisition adaptability of accident site reconstruction.

[0115] Step S300: The UAV flies along the layered surround acquisition path and acquires multimodal data frames at the second sampling rate; and performs adaptive frame extraction processing on the multimodal data frames based on changes in motion saliency and geometric complexity to obtain key frames;

[0116] The second sampling rate is determined as follows: based on a baseline sampling rate (e.g., 10Hz), it is dynamically adjusted according to the following three factors: the higher the intensity of motion in the accident area, the faster the sampling; the denser the point cloud geometric distortion, the faster the sampling; and the smaller the UAV's flight radius (the closer it is to the ground), the faster the sampling. The final sampling rate is limited to between 2 and 20Hz to ensure that it does not exceed the sensor's and processing capabilities.

[0117] The multimodal data frame includes:

[0118] RGB images, LiDAR point clouds, and IMU pose data.

[0119] In the specific implementation process, see Figure 3 Step S300, which involves performing adaptive frame extraction processing on the multimodal data frames based on changes in motion saliency and geometric complexity to obtain keyframes, includes:

[0120] Step S310: Obtain the RGB images of the current frame and the previous frame, and calculate the optical flow field between the two frames; statistically analyze the optical flow amplitude of each pixel in the optical flow field, and use the average or median of the optical flow amplitude as the first motion saliency measure of the current frame.

[0121] Specifically, after acquiring the RGB image captured by the UAV in the current frame and the RGB image in the previous frame, the two images are preprocessed to improve the robustness of optical flow calculation: the color image is converted to a grayscale image, and a Gaussian filter is used to remove high-frequency noise. Subsequently, the dense optical flow field is calculated using the pyramid-based Lucas-Kanade optical flow method to obtain the motion vector of each pixel on the image plane.

[0122] Optionally, this embodiment also introduces a spatial weighting and region filtering mechanism. Specifically, based on the "accident probability field" output by the accident recognition model at the previous moment, pixel regions with probability values ​​greater than a preset threshold (e.g., 0.5) are extracted as regions of interest. The average optical flow amplitude is calculated only within this region of interest, and this average value is used as the first motion saliency measure. If there is no historical accident probability field in the current frame (e.g., the first frame), the calculation is performed across the entire image, but stationary background points with excessively low optical flow amplitudes (less than 0.5 pixels / frame) must be excluded.

[0123] After the above processing, the resulting first motion significance measure can truly reflect the intensity of motion in the core area of ​​the accident.

[0124] Step S320: Obtain the IMU pose data of the current frame and the previous frame, calculate the change in angular velocity between the two frames, and use the change in angular velocity as the second motion saliency measure.

[0125] Specifically, the attitude data output by the UAV's onboard IMU at the current frame and the previous frame are acquired, including pitch angle, roll angle, yaw angle, and three-axis angular velocities. In this embodiment, the original IMU pose data is first low-pass filtered to eliminate high-frequency vibration noise, and then the absolute values ​​of the changes in each angular velocity component between the two frames are calculated. To comprehensively characterize the UAV's own motion intensity, the changes in pitch angular velocity, roll angular velocity, and yaw angular velocity are weighted and summed to obtain a second motion significance metric. Among these, the change in yaw angular velocity is most sensitive to the UAV's steering and is given a high weight (e.g., 0.5), while the pitch and roll weights are each 0.25.

[0126] Optionally, considering the IMU integral drift problem, the embodiment further introduces sliding window differential verification: maintaining an angular velocity sequence of 5 frames, and calculating the deviation of the current frame from the average angular velocity of the previous three frames within the window. If the deviation exceeds a preset threshold, it indicates that the UAV has performed a sudden maneuver (such as a sharp turn or emergency climb). In this case, the second motion saliency metric should increase significantly to indicate that subsequent frame extraction processing needs to retain more keyframes to capture scene changes during the maneuver.

[0127] Step S330: Obtain the lidar point cloud of the current frame, divide the lidar point cloud into voxel grids, count the distribution of the number of points in each voxel grid, calculate the information entropy of the distribution of the number of points, and use the information entropy as the geometric complexity measure of the current frame.

[0128] In this embodiment, an adaptive voxel mesh generation method is adopted. First, the initial side length of the voxel mesh is dynamically determined according to the current flight altitude of the UAV: ​​the lower the flight altitude (closer to the ground), the smaller the voxel side length (e.g., from 0.5 meters to 0.1 meters) to ensure the ability to resolve details; the higher the flight altitude, the larger the voxel side length (e.g., 2 meters) to control the amount of computation.

[0129] After dividing the point cloud space into a 3D voxel mesh, the number of points contained in each non-empty voxel is counted. To measure the dispersion of the point cloud distribution, the information entropy of the number of points in all voxels is calculated. The larger the information entropy value, the more uniform the distribution of the point cloud among different voxels, and the more complex the geometric structure (e.g., scattered debris, deformed vehicle parts); the smaller the information entropy, the more concentrated the point cloud is in a few voxels, and the simpler the scene structure (e.g., a flat road surface).

[0130] To further improve sensitivity to local geometric distortions (such as road surface cracks and vehicle dents), the embodiment performs curvature pre-filtering on the point cloud before calculating the information entropy: for each point, the rate of change of the angle between its angle and the normal vector of its neighboring points is calculated, and points with curvature exceeding three times the neighborhood mean are marked as "distorted interest points". Then, only these distorted interest points and auxiliary points within a radius of 0.2 meters are retained to construct a sparse subset of the distorted point cloud. Next, the voxel mesh information entropy calculation described above is performed on this subset. The advantage of this approach is that even if most areas of the overall scene are flat roads, as long as there are a few obvious local geometric distortions, the information entropy will increase significantly, thereby triggering the subsequent frame extraction process to prioritize the retention of that frame.

[0131] Finally, the calculated information entropy is used as a measure of the geometric complexity of the current frame, and together with the aforementioned motion saliency measure, it determines whether the frame is selected as a keyframe.

[0132] In the specific implementation process, the above steps S310 to S330 integrate the region-weighted motion statistics of visual optical flow, the sliding window differential verification of IMU pose, and the curvature pre-filtering and adaptive voxel partitioning of point cloud information entropy. This method can accurately evaluate the motion saliency and geometric complexity of each frame of data in complex accident scenarios, effectively eliminate the interference caused by background noise and the UAV's own maneuvering, and enhance the sensitivity to local geometric distortions (such as collision indentation and debris scattering). This guides the adaptive frame extraction process to prioritize the retention of frames containing key dynamic changes and geometric details, reduce redundant data, and improve the efficiency and accuracy of subsequent accident recognition and spatiotemporal modeling without losing important information.

[0133] Step S340: Obtain keyframes based on the first motion saliency measure, the second motion saliency measure, the geometric complexity measure, and the preset judgment rules.

[0134] Furthermore, step S340 specifically includes:

[0135] Step S341: Weightedly fuse the first motion saliency measure and the second motion saliency measure to obtain a comprehensive motion saliency measure.

[0136] In this embodiment, the weight allocation is not a fixed value, but is dynamically adjusted according to the current flight mode of the UAV. Specifically, when the UAV is in a horizontal high-speed cruise or turning maneuver, the reliability of IMU data is high, and the weight of the second motion saliency measure is increased to 0.7; when the UAV is hovering or moving slowly, the visual optical flow can better reflect the real motion of the target in the scene, and the weight of the first motion saliency measure is increased to 0.7. The flight mode can be automatically identified by IMU acceleration integral and velocity threshold. In addition, in order to prevent misjudgment caused by instantaneous anomalies of a single sensor, the embodiment introduces median filtering: the weighted results of the current frame and the frame before and after are smoothed by median before being used as the final comprehensive motion saliency measure output.

[0137] Step S342: Construct a two-dimensional state space with the comprehensive motion saliency measure and the geometric complexity measure as the horizontal and vertical axes, respectively. To ensure comparability between the measures, both are normalized: the comprehensive motion saliency measure is mapped to the interval between 0 and 1, using the maximum value within a historical sliding window (e.g., retaining the maximum value of the most recent 50 frames as the normalization denominator); the geometric complexity measure is also normalized using information entropy. After normalization, each frame corresponds to a point in the two-dimensional state space. The area near the origin of this space represents a stable scene with low motion and low complexity, while the area far from the origin represents a critical scene with high motion or high complexity.

[0138] Step S343: Preset a keyframe trigger region in the two-dimensional state space, wherein the boundary of the keyframe trigger region is defined by a nonlinear function;

[0139] This embodiment employs a combination of elliptical arcs and exponential decay curves: specifically, the trigger region is the intersection of a sector expanding outward from the origin and the elliptical boundary. The major axis of the ellipse points to the horizontal coordinate direction, and the minor axis points to the vertical coordinate direction. The length of its semi-axis is dynamically adjusted by the accident level parameter (the higher the accident level, the shorter the semi-axis, and the stricter the triggering conditions, avoiding excessive frame skipping). Simultaneously, an exponential decay curve is superimposed on the elliptical boundary as an auxiliary judgment line—when the comprehensive motion saliency metric is higher than a high threshold (e.g., 0.6), even if the geometric complexity metric is low (>0.1), it is considered part of the trigger region; conversely, when the geometric complexity metric is higher than a high threshold (e.g., 0.7), even if the motion saliency is low, it is included in the trigger region. The advantage of this nonlinear boundary is that it can capture purely dynamic events (e.g., vehicles moving at high speed but with simple geometry) and purely static complex events (e.g., debris scattered after a rollover but no longer moving).

[0140] Step S344: Calculate the coordinates of the current frame in the two-dimensional state space. When the coordinates fall within the key frame trigger area, mark the current frame as a key frame; otherwise, mark it as a non-key frame. It can be discarded or downsampled and stored in subsequent frame extraction processing.

[0141] Ultimately, all multimodal data frames marked as keyframes will be sent to the subsequent accident identification and snapshot generation module, while non-keyframes will be stored in a lightweight manner or discarded directly, thereby achieving efficient data filtering.

[0142] In step S340, a comprehensive motion saliency is obtained by dynamically weighting and fusing the first and second motion saliency measures. This comprehensive motion saliency is then combined with a geometric complexity measure to construct a two-dimensional state space. An elliptic-exponential composite nonlinear boundary is used to define the keyframe trigger region. By combining a continuous frame memory mechanism and longest interval protection, joint sensitive judgment of dynamic changes and geometric distortions in the accident scenario is achieved. This method can adaptively distinguish between the UAV's own maneuvering and the actual motion of the scene, avoiding missed detections or oversampling caused by a single threshold. Simultaneously, it ensures the temporal continuity of the accident evolution and the minimum sampling coverage, significantly improving the accuracy, robustness, and computational efficiency of keyframe selection.

[0143] Step S400: Obtain the inter-frame velocity vector and semantic segmentation confidence of each key frame; weight and fuse the inter-frame velocity vector and semantic segmentation confidence to obtain the dynamic target region; remove the lidar point cloud in the dynamic target region to obtain the key frame with the dynamic target removed.

[0144] See Figure 4 Step S400 specifically includes:

[0145] Step S410: Construct a three-dimensional voxel grid, project the LiDAR point cloud of each key frame onto the three-dimensional voxel grid, take the average value of the inter-frame velocity vector of each voxel grid to obtain the average velocity vector, and take the maximum value of the semantic segmentation confidence of each voxel grid to obtain the highest confidence.

[0146] Step S420: Normalize the average velocity vector and then sum it with the highest confidence level to obtain the dynamic probability of each voxel grid and generate a dynamic target probability field.

[0147] Furthermore, the normalization method employs adaptive saturation normalization: the magnitude distribution of the average velocity vector of all voxels in the current keyframe is statistically analyzed, and the 95th percentile of the magnitude is taken as the upper limit of saturation. Velocities greater than this upper limit are normalized to 1; those less than this upper limit are linearly mapped to a range between 0 and 1. This approach avoids normalization inaccuracies caused by individual high-speed motion points while preserving the effective distinguishability of low-speed motion.

[0148] Step S430: Using a region growing algorithm, grids with dynamic probabilities exceeding a preset seed threshold are used as seed points and expanded outwards until the dynamic probabilities are below a preset boundary threshold, generating one or more dynamic target regions.

[0149] Specifically, first, a high seed threshold (e.g., dynamic probability greater than 0.7) and a low boundary threshold (e.g., dynamic probability less than 0.3) are set. All voxels are traversed, and voxels with dynamic probabilities exceeding the seed threshold are marked as candidate seed points, sorted by probability value from highest to lowest. Starting from each seed point, expansion proceeds sequentially in six neighborhood directions in 3D space. As long as the dynamic probability of a neighboring voxel is not lower than the boundary threshold, it is included in the current dynamic region, and expansion continues outward. The process stops when a voxel with a dynamic probability lower than the boundary threshold is encountered during expansion.

[0150] To avoid oversegmentation or undersegmentation, this embodiment introduces a dynamic region merging strategy: if the minimum Euclidean distance between two expanded dynamic regions is less than twice the voxel side length, and the difference between the mean dynamic probabilities of the two regions is less than 0.2, then they are merged into a single dynamic target region. Ultimately, each dynamic target region corresponds to an independent dynamic target (such as a vehicle or a pedestrian).

[0151] Step S440: For each dynamic target region extracted in step S430, cluster the lidar point cloud within each dynamic target region into the corresponding dynamic target (e.g., ...). Figure 5 As shown in the figure, motion pattern consistency analysis is performed on the inter-frame velocity vectors of all points within the dynamic target; if the velocity vectors are radially distributed, they are determined to be expansion type dynamic target; if they are vortex-shaped distributed, they are determined to be rotation type dynamic target; if the directions are consistent, they are determined to be translation type dynamic target.

[0152] In this embodiment, the motion pattern consistency analysis employs a local neighborhood orientation histogram method: the velocity direction of each point is divided into eight main direction intervals, and the number of points within each interval is counted. If the number of points is evenly distributed and covers multiple directions, and the velocity vector radiates outward from its starting point to its ending point (i.e., diverging outward from a central point), it is identified as an expansion-type dynamic target, typically seen in scenarios such as an explosion shockwave, scattered debris, or airbag deployment. If the velocity vector shows a tendency to rotate around a central point (i.e., the direction change exhibits a continuous circular distribution), it is identified as a rotation-type dynamic target, typically seen in scenarios such as a rolling vehicle or a spinning wheel. If the velocity direction of the vast majority of points is concentrated in one main direction interval (accounting for more than 70%), it is identified as a translational dynamic target, typically seen in scenarios such as a normally moving car.

[0153] Step S450: Remove the lidar point clouds of the dilatation-type dynamic target and the rotation-type dynamic target by morphological erosion, and remove the lidar point clouds of the translation-type target by edge feathering, and obtain the key frame for removing the dynamic target.

[0154] Specifically, for expanding dynamic targets: due to their divergent motion direction, the point cloud is often distributed along the target boundary and has a loose shape. Direct deletion may lead to the accidental deletion of static background points at the target edge. Therefore, a morphological erosion operation is used: first, the dynamic target region is binarized (1 within the region, 0 outside the region), and then a three-dimensional structuring element (e.g., a spherical kernel with a radius twice the side length of the voxel) is used to erode the binary mask, reducing the region's boundary. Only the lidar point cloud within the eroded region is removed, retaining the point cloud in the edge ring (these points are likely to belong to the static background). This allows for the removal of spatter debris while preserving the background integrity to the maximum extent.

[0155] For rotating dynamic targets: the point cloud of a rotating target has a lower velocity near the axis of rotation (such as the center of a wheel) and may be mistakenly classified as static; while the velocity is higher at the edges. To ensure complete removal without leaving a trail, an erosion operation similar to that used for dilatation is employed, but with a larger erosion core (radius three times the side length of the voxel) to ensure that both the outer edge and internal residue of the rotating target are removed.

[0156] For translational dynamic targets: These targets (such as moving vehicles) typically have clear boundaries, but the point cloud between the vehicle's bottom and the road surface may have a velocity transition zone (where the tires contact the ground). Directly deleting these points with hard boundaries would create holes in the road surface point cloud. Therefore, edge feathering culling is used: instead of directly deleting all points within the region, a gradient probability is assigned to each point based on its distance from the boundary of the dynamic target region. Points closer to the boundary have a higher retention probability; points closer to the region center have a lower retention probability. Finally, points with a retention probability higher than 0.8 are randomly sampled or directly deleted. This achieves a smooth transition and avoids artificially created boundary cracks in subsequent static mapping.

[0157] After the above differential removal process, the dynamic target point cloud in each keyframe is effectively removed, and the remaining LiDAR point cloud mainly consists of static background (such as roads, buildings, trees, etc.). These keyframes after removing dynamic targets will be stored in a static map database for subsequent disaster assessment or 3D reconstruction.

[0158] Optionally, step S400 can also fuse the point cloud with the image based on the improved FAST-LIVO2 algorithm to generate an initial dense point cloud; identify dynamic targets (such as pedestrians, vehicles, rescue equipment, etc.) through the point cloud velocity features and image semantic segmentation results; remove moving target point clouds and retain static scene point clouds (such as road surface, accident vehicle wreckage). Utilize the contextual relationship between consecutive frames and the point cloud to synchronously apply the segmentation model to remove dynamic target regions on the image.

[0159] Step S500: Input the keyframes of the dynamic target removal into the 3D Gaussian splash model to generate a three-dimensional accident scene model.

[0160] Specifically, step S500 includes:

[0161] Step S510: Using the RGB image, LiDAR point cloud, and IMU pose data from the keyframe for removing dynamic targets as input, initialize a 3D Gaussian ellipsoid (each point corresponds to a Gaussian ellipsoid, including position, scaling, rotation, opacity, and color parameters); project the 3D Gaussian ellipsoid onto the image plane using a differentiable renderer to generate a rendered image;

[0162] Specifically, firstly, a 3D Gaussian ellipsoid is initialized using the LiDAR point cloud: each LiDAR point is used as the center of a Gaussian ellipsoid. Each Gaussian ellipsoid is defined by the following learnable parameters: 3D spatial position, three-axis scaling factor (determining the extent to which the ellipsoid stretches in the length, width, and height directions), quaternion rotation (determining the orientation of the ellipsoid), opacity (determining the weight of the ellipsoid's contribution to the final image), and spherical harmonic coefficients (used to express color variations with the viewing direction). For sparse areas of the point cloud, additional Gaussian ellipsoids are interpolated between adjacent points to ensure the integrity of scene coverage.

[0163] Next, a differentiable renderer is constructed. Given the camera's precise pose (obtained by fusing IMU data with visual odometry), the renderer sorts all Gaussian ellipsoids by depth from the camera, from farthest to nearest, and projects them sequentially onto the image plane. Each Gaussian ellipsoid, after projection, forms a two-dimensional elliptical patch, the color of which is calculated from the spherical harmonic coefficients based on the current viewpoint direction, and the opacity controls the blending strength of the patch. The renderer uses a front-to-back alpha blending method to synthesize the final color of each pixel, outputting a complete rendered image. The entire process is differentiable, meaning that the gradient of the rendered image with respect to the parameters of each Gaussian ellipsoid can be calculated, providing a basis for subsequent optimization.

[0164] To accelerate the convergence speed, this embodiment introduces depth-leader initialization: the lidar point cloud is directly used as the initial position of the Gaussian ellipsoid, and the initial scaling factor is estimated based on the local point density of the point cloud—the scaling factor is smaller where the points are dense and larger where the points are sparse, so that the Gaussian ellipsoid adaptively covers the cavity area.

[0165] Step S520: Calculate the loss between the rendered image and the input RGB image, and iteratively optimize the parameters of the 3D Gaussian ellipsoid based on backpropagation of the loss; after each iteration, the Gaussian ellipsoid can be adaptively added or deleted according to the gradient. When the loss converges or the preset number of iterations is reached, output a three-dimensional accident scene model.

[0166] By employing differentiable rendering and iterative optimization based on 3D Gaussian ellipsoids, a high-fidelity 3D accident scene model can be efficiently reconstructed from keyframes after removing dynamic targets. The final output 3D model supports real-time rendering from any viewpoint, providing an intuitive and interactive visualization basis for accident investigation and other related tasks.

[0167] This invention discloses an autonomous event scene reconstruction method based on unmanned aerial vehicles (UAVs). Through fully automated processes—autonomous identification and positioning, layered surround data acquisition, multimodal adaptive frame extraction, dynamic target differential removal, and 3D Gaussian splash reconstruction—it achieves rapid 3D reconstruction of accident scenes without human intervention. Compared to existing technologies, this solution reduces on-site investigation and modeling time from several hours to less than 30 minutes. It reduces data redundancy through keyframe selection driven by motion saliency and geometric complexity, and utilizes velocity vector pattern classification to perform morphological erosion or edge feathering removal on dynamic targets such as dilation, rotation, and translation, ensuring that static scene modeling errors are ≤5cm. The final output is an interactive and quantifiable high-fidelity scene model, significantly improving accident response efficiency, data integrity, and reconstruction accuracy in emergency scenarios.

[0168] Furthermore, the drone-based autonomous event scene reconstruction system of this embodiment also includes:

[0169] Unmanned aerial vehicles (UAVs), multi-sensor modules, edge computing units, and ground control terminals;

[0170] The ground control terminal is electrically connected to the UAV, the multi-sensor module, and the edge computing unit.

[0171] The multi-sensor module and edge computing unit are integrated on the UAV, enabling the UAV to perform any of the above-described UAV-based autonomous event scene reconstruction methods.

[0172] Specifically, the UAV is equipped with a power system and a flight control system, supporting autonomous flight and obstacle avoidance. The multi-sensor module includes: a lidar (e.g., 16-line, 0.1°×0.1° resolution) for acquiring environmental point cloud data; an RGB camera (20 megapixels) and a telephoto camera (5x optical zoom) for acquiring global scene and detailed close-up shots, respectively; and a GNSS / IMU module providing real-time pose (latitude, longitude, altitude, and attitude angle). The edge computing unit uses a Jetson AGX NX host, integrating multi-sensor synchronization interfaces and computing resources, running real-time data processing algorithms for multimodal data synchronization, adaptive frame extraction, moving target point cloud culling, and 3DGS modeling. The ground control terminal receives the 3D model results transmitted back by the UAV, supporting manual verification and secondary editing.

[0173] This embodiment of the UAV-based autonomous event scene reconstruction system achieves fully autonomous and highly timely 3D reconstruction of event scenes by using UAVs equipped with multi-sensor modules and edge computing units, significantly reducing reliance on manual operation and on-site work time. Utilizing adaptive frame extraction and dynamic target point cloud culling technology, it effectively filters out motion interference (such as personnel and vehicles), and combines this with a 3D Gaussian splash (3DGS) model to generate a high-precision static 3D accident scene model, solving the problems of poor data integrity and insufficient modeling accuracy in traditional methods. Simultaneously, the layered surround acquisition strategy and multimodal data synchronization mechanism ensure complete coverage from global to detailed data, providing reliable and interactive digital twin evidence for accident evidence collection, emergency command, and remote analysis.

[0174] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0175] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0176] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that they are in indirect contact through an intermediate medium. Furthermore, "above," "over," or "on top" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," or "beneath" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0177] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0178] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for autonomous reconstruction of an event scene based on unmanned aerial vehicles (UAVs), characterized in that, include: When the UAV is inspecting along the preset route, it collects visual images and sparse point clouds at the first sampling rate, and inputs the visual images and sparse point clouds into the trained accident recognition model to generate a feature snapshot containing the accident type and the accident spatial range. The drone generates a layered surround acquisition path, including a panoramic layer, a detail layer, and a close-up layer, based on the feature snapshot and a preset layered acquisition strategy. The drone flies along the layered surrounding acquisition path and acquires multimodal data frames at a second sampling rate; The multimodal data frames are then subjected to adaptive frame extraction processing based on changes in motion saliency and geometric complexity to obtain keyframes. The multimodal data frame includes: RGB images, LiDAR point clouds, and IMU pose data; The multimodal data frames are subjected to adaptive frame extraction processing based on changes in motion saliency and geometric complexity to obtain keyframes, including: Acquire the RGB images of the current frame and the previous frame, and calculate the optical flow field between the two frames; statistically analyze the optical flow amplitude of each pixel in the optical flow field, and use the average or median of the optical flow amplitude as the first motion saliency measure of the current frame; Acquire IMU pose data of the current frame and the previous frame, calculate the change in angular velocity between the two frames, and use the change in angular velocity as the second motion saliency measure. The lidar point cloud of the current frame is obtained, the lidar point cloud is divided into voxel grids, the distribution of the number of points in each voxel grid is statistically analyzed, the information entropy of the distribution of the number of points is calculated, and the information entropy is used as the geometric complexity measure of the current frame. Keyframes are obtained based on the first motion saliency measure, the second motion saliency measure, the geometric complexity measure, and the preset judgment rules. The inter-frame velocity vector and semantic segmentation confidence of each key frame are obtained, and the inter-frame velocity vector and semantic segmentation confidence are weighted and fused to obtain the dynamic target region; the lidar point cloud in the dynamic target region is removed to obtain the key frame of dynamic target removal. The keyframes for removing dynamic targets are input into the 3D Gaussian splash model to generate a three-dimensional accident scene model.

2. The method for autonomous reconstruction of event scenes based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The visual image and sparse point cloud are input into the trained accident recognition model to generate a feature snapshot containing the accident type and the spatial extent of the accident, including: Based on the visual image and sparse point cloud, the accident recognition model identifies at least one abnormal target at the accident scene and outputs the accident type of the abnormal target. Based on the visual image and sparse point cloud, the accident recognition model is used to perform spatial aggregation and boundary estimation on the abnormal target, and outputs the accident spatial range, which includes the center coordinates and spatial boundary of the core area of ​​the accident. The accident type and the accident spatial range are fused together and encapsulated to generate the feature snapshot.

3. The method for autonomous reconstruction of event scenes based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The accident identification model includes: A parallel dual-branch feature encoder includes a visual branch and a point cloud branch. The visual branch uses a lightweight network based on temporal convolution to extract two-dimensional semantic features and motion change features of the visual image. The point cloud branch uses a sparse convolutional network to extract three-dimensional structural features and local ensemble distortion features of the sparse point cloud. A cross-modal dynamic fusion module is used to associate the two-dimensional semantic features with the three-dimensional structural features under spatial coordinate mapping and generate a fused feature map based on an attention mechanism. The temporal memory aggregation unit is used to receive fused feature maps from N consecutive time points, model the evolution of the accident scenario on the time axis through a gated loop mechanism, and output the accident probability field and dynamic change field. The multi-task output head includes an accident type classification branch, a spatial range regression branch, and a confidence assessment branch, which are used to obtain the accident type, the accident spatial range, and the comprehensive confidence based on the accident probability field and the dynamic change field, respectively.

4. The method for autonomous reconstruction of event scenes based on unmanned aerial vehicles (UAVs) according to claim 2, characterized in that, The preset layered acquisition strategies include panoramic layer strategy, detail layer strategy, and close-up layer strategy; The panoramic layer strategy is to determine the first acquisition height and the first orbital radius based on the accident space range in the feature snapshot, and generate a panoramic layer flight path. The detailed layer strategy adaptively determines multiple hovering points based on the accident type of the feature snapshot in order to generate a detailed layer flight path. The close-up layer strategy is to use the center coordinates of the core area of ​​the accident as the acquisition point and generate a close-up layer approach path. The panoramic layer strategy, detail layer strategy, and close-up layer strategy are executed sequentially.

5. The method for autonomous reconstruction of event scenes based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, Based on the first motion saliency measure, the second motion saliency measure, and the geometric complexity measure, as well as the preset judgment rules, the keyframes are obtained specifically including: The first motion saliency measure and the second motion saliency measure are weighted and fused to obtain a comprehensive motion saliency measure; Construct a two-dimensional state space with the comprehensive motion saliency measure and the geometric complexity measure as the horizontal and vertical axes, respectively; A keyframe trigger region is preset in the two-dimensional state space, and the boundary of the keyframe trigger region is defined by a nonlinear function. Calculate the coordinates of the current frame in the two-dimensional state space. When the coordinates fall within the keyframe trigger area, mark the current frame as a keyframe.

6. The method for autonomous reconstruction of event scenes based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The inter-frame velocity vector and semantic segmentation confidence score of each keyframe are obtained, and the inter-frame velocity vector and semantic segmentation confidence score are weighted and fused to obtain the dynamic target region, including: A three-dimensional voxel grid is constructed, and the LiDAR point cloud of each key frame is projected onto the three-dimensional voxel grid. The average velocity vector is obtained by averaging the inter-frame velocity vectors of the points in each voxel grid, and the highest confidence score is obtained by taking the maximum value of the semantic segmentation confidence score of the points in each voxel grid. The average velocity vector is normalized and then weighted and summed with the highest confidence level to obtain the dynamic probability of each voxel grid, thereby generating a dynamic target probability field. The region growing algorithm is used to take grids with dynamic probabilities exceeding a preset seed threshold as seed points and expand them outward to the point where the dynamic probabilities are below the preset boundary threshold, thereby generating one or more dynamic target regions.

7. The method for autonomous reconstruction of event scenes based on unmanned aerial vehicles (UAVs) according to claim 6, characterized in that, The process of removing LiDAR point clouds within the region of dynamic targets and obtaining keyframes for removing dynamic targets specifically includes: The lidar point cloud within each dynamic target area is clustered into the corresponding dynamic target. Motion pattern consistency analysis is performed on the inter-frame velocity vectors of all points within the dynamic target. If the velocity vectors are radially distributed, they are identified as expansion-type dynamic targets; if they are vortex-type distributed, they are identified as rotation-type dynamic targets; if the directions are consistent, they are identified as translation-type dynamic targets. The lidar point clouds of the dilatational and rotational dynamic targets are removed by morphological erosion, and the lidar point clouds of the translational targets are removed by edge feathering, thus obtaining the key frames for removing the dynamic targets.

8. The method for autonomous reconstruction of event scenes based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The process of inputting the keyframes for removing dynamic targets into the 3D Gaussian splash model to generate a 3D accident scene model specifically includes: Using the RGB image, LiDAR point cloud, and IMU pose data from the keyframes for removing dynamic targets as input, a 3D Gaussian ellipsoid is initialized; the 3D Gaussian ellipsoid is then projected onto the image plane using a differentiable renderer to generate a rendered image. Calculate the loss between the rendered image and the input RGB image, and iteratively optimize the parameters of the 3D Gaussian ellipsoid based on backpropagation of the loss; When the iteration converges or reaches the preset number of iterations, a three-dimensional accident scene model is output.

9. An autonomous event scene reconstruction system based on unmanned aerial vehicles (UAVs), characterized in that, include: Unmanned aerial vehicles (UAVs), multi-sensor modules, edge computing units, and ground control terminals; The ground control terminal is electrically connected to the UAV, the multi-sensor module, and the edge computing unit. The multi-sensor module and the edge computing unit are integrated on the UAV, enabling the UAV to perform the UAV-based autonomous event scene reconstruction method as described in any one of claims 1 to 8.

Images