Disaster scene real scene situation multi-mode reconstruction method and system

Abstract

The present application belongs to the technical field of multi-sensor fusion calibration, and specifically relates to a disaster scene real scene situation multi-modal reconstruction method and system, wherein the method comprises the following steps: converting an emergency stop maneuver into an external parameter calibration signal source: monitoring the flight state of a UAV in real time and detecting an emergency stop event; constructing a data acquisition time window based on the time when the emergency stop is triggered; using the registration residual error of the point cloud at the emergency stop instant from motion to stillness and the image optical flow residual error as observation information to estimate the external parameter rotation matrix drift and the translation vector drift and to perform online compensation and update on the external parameter matrix; generating a three-dimensional model based on the fused point cloud and image data after calibration and performing dynamic update; identifying disaster elements using an AI lightweight semantic segmentation network to generate an analysis report and a three-dimensional situation map. The present application realizes external parameter self-adaptive calibration in flight without calibration objects and without landing.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor fusion calibration technology, and in particular to a method and system for multimodal reconstruction of real-world disaster scene conditions. Background Technology

[0002] The use of drones equipped with LiDAR and cameras for 3D reconstruction of disaster sites has become a key technology for emergency response. LiDAR provides high-precision 3D spatial geometric information, while cameras provide texture and color information; their fusion can significantly improve scene perception capabilities. The spatial transformation relationship between the LiDAR coordinate system and the camera coordinate system is represented by an extrinsic parameter matrix (including rotation matrices and translation vectors), and an accurate extrinsic parameter matrix is ​​a prerequisite for effective data fusion.

[0003] Current extrinsic parameter calibration primarily employs offline methods, involving calibration using targets such as calibration boards before takeoff and assuming that the extrinsic parameters remain constant during flight. However, in actual UAV flight, vibrations, temperature changes, and mechanical shocks from emergency maneuvers (such as sudden stops and hovering transitions) can all cause slight drifts in the extrinsic parameters. Once this drift occurs, the accuracy of point cloud-image fusion will significantly decrease. Furthermore, UAVs cannot land and return for recalibration after takeoff, and existing online calibration schemes rely on regular structures such as line features and corner features present in the scene. However, disaster sites (such as landslides and breach areas) often lack these features, making it impossible to guarantee calibration accuracy. Summary of the Invention

[0004] This invention proposes a multimodal reconstruction method and system for real-time situation at disaster sites, aiming to solve the technical problem in the prior art where the external parameters of UAVs drift in real time due to the impact of sudden stops and maneuvers during disaster site operations, resulting in a decrease in the accuracy of point cloud and image fusion.

[0005] In a first aspect, the present invention provides a method for multimodal reconstruction of real-world disaster scene conditions, including: The system acquires point cloud data collected by the lidar on the drone, pose data collected by navigation, and image data collected by the camera; wherein the spatial transformation relationship between the lidar coordinate system and the camera coordinate system is represented by an extrinsic parameter matrix, which includes a rotation matrix and a translation vector. During the flight of the UAV, the flight status is monitored. When an emergency stop event is detected, a data acquisition time window is triggered. Within the data acquisition time window, the point cloud data frame sequence of the lidar and the image frame sequence of the camera are acquired. Based on the point cloud data frame sequence and image frame sequence within the acquisition time window, determine the drift amount of the extrinsic parameter matrix between the lidar and the camera; The current extrinsic parameter matrix is ​​updated and compensated based on the drift of the extrinsic parameter matrix. Based on the updated extrinsic parameter matrix, the point cloud data and the image data are fused to generate a 3D model; The disaster elements in the three-dimensional model are identified, and three-dimensional situation analysis results of the disaster site are generated.

[0006] Furthermore, the detection of the emergency stop event includes: when the UAV's flight speed drops from greater than a first speed threshold to less than a second speed threshold, and the rate of change of acceleration is greater than a first rate of change of acceleration threshold, then an emergency stop event is determined to have occurred.

[0007] Furthermore, the construction of the acquisition time window includes: taking the triggering time of the emergency stop event as the base time, backtracking for a first preset duration, and extending backward for a second preset duration to construct the acquisition time window.

[0008] Further, determining the drift of the extrinsic parameter matrix between the lidar and the camera based on the point cloud data frame sequence and image frame sequence within the acquisition time window includes: The point cloud data frame sequence within the acquisition time window is divided into a sequence before the emergency stop event and a sequence after the emergency stop event. A first point cloud model is generated based on the sequence before the emergency stop event, and a second point cloud model is generated based on the sequence after the emergency stop event. The second point cloud model is registered with the first point cloud model to obtain the point cloud registration residual; The first optical flow field before the emergency stop event and the second optical flow field after the emergency stop event are extracted from the image frame sequence within the acquisition time window. The optical flow field is used to characterize the motion vector of pixels in the image between adjacent frames. The image optical flow residual is obtained based on the first optical flow field and the second optical flow field. The drift amount of the extrinsic parameter matrix is ​​determined based on the point cloud registration residual and the image optical flow residual.

[0009] Furthermore, the registration of the second point cloud model with the first point cloud model includes: using the iterative nearest point algorithm to register the second point cloud model with the first point cloud model, and calculating the sum of squared residuals of the registered point clouds as the point cloud registration residuals.

[0010] Further, obtaining the image optical flow residual based on the first optical flow field and the second optical flow field includes: calculating the mean amplitude of the first optical flow field and the mean amplitude of the second optical flow field, and taking the absolute value of the difference between the two as the image optical flow residual.

[0011] Furthermore, determining the drift of the extrinsic parameter matrix based on the point cloud registration residual and the image optical flow residual includes: using the drift of the extrinsic rotation matrix ΔR and the drift of the extrinsic translation vector Δt as the state variables to be estimated, constructing an objective function constrained by the point cloud registration residual and the image optical flow residual, and solving the objective function to obtain ΔR and Δt.

[0012] Furthermore, the objective function is: Register the point cloud residual function With image optical flow residual function Minimize the weighted sum of squares, the point cloud registration residual function Defined as: ; The image optical flow residual function Defined as: ; in, This indicates the use of the extrinsic parameter drift amount The updated extrinsic matrix obtained after applying the current extrinsic matrix is ​​used to project the point cloud onto the image plane and calculate the mean amplitude of the optical flow field after the sudden stop event. This represents the mean amplitude of the optical flow field before the emergency stop event, calculated after projecting the point cloud onto the image plane using the current extrinsic parameter matrix. The objective function is: ; And find the solution that minimizes the objective function. and ; in, The extrinsic rotation matrix drift is... The displacement of the extrinsic translation vector. and The rotation matrix and translation vector are obtained by iteratively registering the second point cloud model with the first point cloud model to the nearest point. For points in the second point cloud model, For the first point cloud model and The corresponding nearest point, The weighting coefficient, symbol This indicates finding the minimum value, with the symbol... The square norm of a vector. The total number of points in the point cloud, symbol This indicates taking the absolute value.

[0013] Furthermore, the compensation update of the current extrinsic parameter matrix includes: applying the rotation matrix drift ΔR and translation vector drift Δt from the drift of the extrinsic parameter matrix to the current extrinsic parameter matrix. ; ; in and The external parameter rotation matrix and translation vector before the update. and Let be the updated extrinsic rotation matrix and translation vector.

[0014] Secondly, the present invention provides a multimodal reconstruction system for real-time situational awareness at disaster sites, the system being used to execute the method, comprising: The data acquisition module is used to acquire point cloud data collected by the lidar on the UAV, pose data collected by the integrated navigation system, and image data collected by the camera; wherein, the spatial transformation relationship between the lidar coordinate system and the camera coordinate system is represented by an extrinsic parameter matrix, which includes a rotation matrix and a translation vector; The emergency stop detection module is used to monitor the flight status during the flight of the UAV. When an emergency stop event is detected, a data acquisition time window is triggered, and the point cloud data frame sequence of the lidar and the image frame sequence of the camera are acquired within the data acquisition time window. The calibration module is used to determine the drift of the extrinsic parameter matrix between the lidar and the camera based on the point cloud data frame sequence and image frame sequence within the acquisition time window, and to compensate and update the current extrinsic parameter matrix based on the drift of the extrinsic parameter matrix. The fusion module is used to fuse the point cloud data and the image data according to the updated extrinsic parameter matrix to generate a three-dimensional model; The identification output module is used to identify disaster elements in the three-dimensional model and generate three-dimensional situation analysis results of the disaster site.

[0015] The technical effects of this invention are: First, the sudden stop maneuver is transformed from an interference source into a calibration signal source. In existing technologies, sudden stops cause motion blur in point clouds and images, and drastic changes in IMU data. Those skilled in the art typically consider these as interference factors in data acquisition and avoid or eliminate them. This invention overcomes the above-mentioned technical bias by utilizing the abrupt change in motion state from dynamic to static during a sudden stop as a natural calibration trigger signal. External parameter calibration can be initiated without additional hardware or calibration objects, breaking the established technical paradigm of "sudden stops interfering with data acquisition."

[0016] Second, the invention solves the extrinsic parameter drift based on the multimodal consistency constraint during the motion-to-station transition. This invention utilizes the registration residuals of point clouds before and after a sudden stop, and the jump residuals of image optical flow amplitude, as joint observation information. It extracts common constraints on extrinsic parameter drift from data from two types of heterogeneous sensors, solving the problem that single-modal information is insufficient to separate carrier motion from extrinsic parameter drift, and achieving extrinsic parameter drift calculation without requiring scene features.

[0017] Third, the extrinsic parameter matrix is ​​updated online during flight. After obtaining the extrinsic parameter drift, the extrinsic parameter matrix is ​​immediately updated and compensated, and the calibrated extrinsic parameter matrix is ​​used for subsequent point cloud-image fusion. This eliminates the need to interrupt flight operations, ensuring the continuity and timeliness of 3D reconstruction in disaster emergency scenarios. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a multimodal reconstruction method for real-time disaster scene conditions proposed in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This paper addresses the issue mentioned in the background technology that, during UAV operations at disaster sites, extrinsic parameters drift in real time due to the impact of sudden stops and maneuvers, leading to a decrease in the accuracy of point cloud and image fusion. Existing solutions cannot perform online extrinsic parameter calibration without relying on calibration objects or requiring landing. (Reference) Figure 1 As shown in the figure, this embodiment of the invention provides a method for multimodal reconstruction of real-world disaster scene conditions, specifically including the following steps: I. Data Acquisition and Initial Sensor Calibration. The multi-sensor system on the UAV platform includes: a 1550nm long-range lidar, a combined navigation system (including a GNSS receiver and an inertial measurement unit, IMU), and a multispectral camera. The spatial transformation relationship between the lidar coordinate system and the camera coordinate system is characterized by an extrinsic parameter matrix, which includes a rotation matrix R and a translation vector t.

[0021] Before the drone takes off, perform initial sensor calibration: (1) Camera intrinsic parameter calibration: Zhang Zhengyou's chessboard calibration method was adopted. Chessboard images were taken in different poses, corner coordinates were extracted, and the camera intrinsic parameter matrix K (including focal length) was solved by maximum likelihood estimation. , and principal point coordinates , ) and distortion coefficient (radial distortion) , and tangential distortion , Specifically, the mapping relationship between the world coordinates of the checkerboard corner points and their projected coordinates in the image is established, and the intrinsic parameter matrix and distortion coefficients are solved by minimizing the reprojection error.

[0022] (2) Initial calibration of lidar-camera extrinsic parameters: The hand-eye calibration method was adopted. A calibration board was placed in the common field of view of the lidar and camera, and multiple sets of point cloud and image data under the pose of the calibration board were collected. The point cloud of the calibration board plane was extracted from the point cloud, and the plane equation was fitted; the corner pixel coordinates of the calibration board were extracted from the image. The geometric constraint relationship of the calibration board plane in the lidar coordinate system and the camera coordinate system was established, and the rotation matrix was solved. Translation vector This ensures that the projections of the calibration plate plane normal vectors onto the two coordinate systems are consistent, thus completing the initial external parameter calibration.

[0023] (3) Integrated navigation system-LiDAR lever calibration: Measure the three-dimensional spatial offset (lever value) and installation angle between the integrated navigation system and the LiDAR, and establish the transformation relationship between the IMU coordinate system and the LiDAR coordinate system.

[0024] After the drone takes off, the lidar collects point cloud data at 10Hz, the integrated navigation system collects pose data at 100Hz, and the multispectral camera collects image data at 30fps. All three use the same GPS clock source for hardware time synchronization: each sensor records a hardware timestamp of the GPS clock when collecting each frame of data, with a synchronization accuracy better than 1 millisecond.

[0025] The spatiotemporal synchronization processing of multi-source sensors is as follows: using the GPS clock as a reference, the hardware timestamp of each sensor data frame is obtained; linear interpolation is used to interpolate the pose data to the same timestamp as the point cloud data; the image timestamp and the point cloud timestamp are matched for nearest neighbor; based on the pose information, the point cloud is transformed from the lidar coordinate system to the global coordinate system through the current extrinsic parameter matrix to complete the spatial alignment.

[0026] II. Construction of Emergency Stop Event Detection and Data Acquisition Window. The onboard computing platform (NVIDIA Jetson AGXOrin) acquires three-axis acceleration data from the IMU in real time. , , Three-axis angular velocity data , , And the speed control commands from the flight control system, which are used to detect emergency stop events.

[0027] The criteria for determining an emergency stop event are as follows: Condition 1: Speed ​​Condition. Assume the current flight speed of the drone is... (Extracted from pose data provided by the integrated navigation system) That is, before the current moment The speed (take 0.5 seconds) is greater than the first speed threshold. (Take 5 m / s).

[0028] Condition 2: Sudden drop in speed. That is, the current speed is less than the second speed threshold. (Take 0.5 m / s).

[0029] Condition 3: Acceleration rate of change condition. That is, the absolute value of the rate of change of acceleration is greater than the first rate of change threshold of acceleration. (Take 15 m / s) 3 ).

[0030] Condition 4: Braking command condition. The flight control system issued a deceleration braking command at time t.

[0031] If all four conditions are met within 0.3 seconds, the emergency stop event is determined to have occurred at time t.

[0032] One of the core ideas of this invention is that an emergency stop event causes the UAV's flight state to abruptly change from high-speed motion to hovering stillness. This motion-to-stillness transition provides a natural calibration window for the consistency constraints of multiple sensors. Before the emergency stop, the UAV is in motion, and both the point cloud and the image carry motion information; after the emergency stop, the UAV is in a near-still state. By utilizing the data difference before and after this transition, the two coupled factors of "carrier's own motion" and "extrinsic parameter drift" can be effectively separated—the motion state before the emergency stop reflects the normal changes in carrier motion, while the stillness state after the emergency stop serves as a benchmark reference, and the inconsistency between the two can be attributed to extrinsic parameter drift.

[0033] Based on the moment of emergency stop event triggering Based on this, backtrack 1 second to... Continue backward for 1 second Construct a data collection time window [ , The first half of the window covers the high-speed motion phase before the emergency stop, and the second half covers the hovering and stationary phase after the emergency stop.

[0034] Within the acquisition time window, the following LiDAR point cloud data frame sequences are recorded synchronously using the GPS clock as the synchronization reference: (approximately 20 frames, i.e.) IMU inertial data sequences (approximately 400 sets), camera image frame sequences (approximately 60 frames, i.e.) ).

[0035] III. Determination of Extrinsic Parameter Drift. In this step, the drift of the extrinsic parameter matrix between the LiDAR and the camera is determined based on the point cloud data frame sequence and image frame sequence within the acquisition time window. This includes the following sub-steps: (a) Extract the point cloud registration residual and divide the point cloud data frame sequence within the acquisition time window into the pre-stop sequence according to the timestamp. ,correspond to Approximately 10 frames from the moment of the emergency stop) and the sequence after the emergency stop ( ,correspond to (Approximately 10 frames from that moment).

[0036] Accumulate and stitch together the point cloud frames in the sequence before the emergency stop to generate the first point cloud model. The stitching method is as follows: using the pose information corresponding to each point cloud frame (provided by the integrated navigation system), the point cloud of each frame is transformed from the lidar coordinate system to the global coordinate system through the current extrinsic parameter matrix. After superposition and fusion, voxel grid filtering is used to remove duplicate points.

[0037] The same method was used to accumulate and splice the sequence after the abrupt stop to generate the second point cloud model. .

[0038] Will and Perform iterative nearest-point registration. The registration iteration steps are as follows: Step (1): For Each point in (i = 1, 2, ..., N, where N is a constant) (total number of points in the point cloud), in The kd-tree nearest neighbor search is used to find the nearest point. .

[0039] Step (2): Calculate the rotation matrix R and translation vector t that minimizes formula (1): (1) The above minimization problem can be solved using singular value decomposition: calculate separately and The centroid is determined, the point cloud is decentered, and the covariance matrix is ​​calculated. The singular value decomposition of the covariance matrix is ​​performed to obtain R, and then t is calculated.

[0040] Step (3): Apply R and t to Update the point cloud positions.

[0041] Step (4): Repeat steps (1)-(3) until the change in registration error between two adjacent iterations is less than the preset threshold.

[0042] After registration, calculate the point cloud registration residuals according to formula (2). : (2) in, and The rotation matrix and translation vector obtained from ICP registration for The i-th point in for Zhongyu The corresponding nearest point, N is The total number of points in the point cloud.

[0043] The physical meaning of this point cloud registration residual is: if the extrinsic parameters between the lidar and the camera do not drift, then and It should be the point cloud representation of the same scene at different times, with the two completely overlapping, and the registration residual theoretically being zero; any non-zero residual reflects the systematic deviation of the point cloud spatial position caused by the drift of external parameters, and the larger the residual, the more serious the drift of external parameters.

[0044] (ii) Extracting image optical flow residuals. The first optical flow field between adjacent frames before the emergency stop event and the second optical flow field between adjacent frames after the emergency stop event are extracted from the image frame sequence within the acquisition time window. The optical flow field is used to characterize the motion vector of pixels in the image between adjacent frames.

[0045] Specifically, extract The last two frames (times) and The Lucas-Kanade optical flow algorithm was used to calculate the optical flow field between two frames. The basic principle of the Lucas-Kanade algorithm is: based on the assumption of constant image brightness, for two adjacent frames of images... and ,have: ; The equation is expanded using Taylor within a local neighborhood, and the optical flow vector (u,v) is obtained by solving the system of optical flow constraint equations consisting of multiple pixels.

[0046] extract The first two frames after (time) and The same Lucas-Kanade algorithm was used to calculate the optical flow field. .

[0047] Calculate according to formula (3) The average amplitude : (3) Calculate according to formula (4) The average amplitude : (4) Calculate the image optical flow residual using formula (5). : (5) in, The function is the arithmetic mean function. Before the emergency stop, the drone is in high-speed motion, resulting in large inter-frame displacement of pixels and a large optical flow amplitude in the image. After the emergency stop, the drone is in a hovering, stationary state, with inter-frame displacement approaching zero and the optical flow amplitude approaching zero. Therefore, It reflects the change in optical flow amplitude caused by the emergency stop event.

[0048] The physical meaning of the optical flow residual in this image is as follows: if the extrinsic parameters do not drift, the change in the optical flow amplitude should be entirely caused by the change in the motion state of the UAV itself, and the two-dimensional motion field obtained by projecting the point cloud onto the image plane should be consistent with the image optical flow field; if the extrinsic parameters drift, the position of the point cloud projected onto the image plane will deviate, resulting in the two-dimensional motion field inferred from the point cloud pose being inconsistent with the image optical flow field, which manifests as an additional error in the optical flow amplitude.

[0049] (III) Construct the objective function and solve for the extrinsic parameter drift. The extrinsic parameter rotation matrix drift ΔR and the extrinsic parameter translation vector drift Δt are the state variables to be estimated. ΔR is a 3×3 rotation matrix belonging to the special orthogonal group SO(3), and Δt is a 3×1 translation vector.

[0050] An objective function is constructed using point cloud registration residuals and image optical flow residuals as constraints.

[0051] First, the fixed registration result in formula (2) , Replace these with the variables to be estimated, ΔR and Δt, to obtain the point cloud registration residual function with ΔR and Δt as independent variables. Define the point cloud registration residual function. as follows: (6) in, The extrinsic rotation matrix drift is... The displacement of the extrinsic translation vector. and The rotation matrix and translation vector are obtained by iteratively registering the second point cloud model with the first point cloud model to the nearest point. For points in the second point cloud model, For the first point cloud model and The corresponding nearest point, The weighting coefficient, symbol This indicates finding the minimum value, with the symbol... The square norm of a vector. The total number of points in the point cloud, symbol This indicates taking the absolute value. The physical meaning of this formula is: to further apply the estimated extrinsic parameter drift (ΔR, Δt) to the already coarsely registered ICP data. After point cloud, recalculate and The point cloud residuals between them.

[0052] Secondly, define the image optical flow residual function. as follows: (7) in, This represents the optical flow field calculated before the emergency stop by projecting the point cloud before the emergency stop onto the image plane using the current extrinsic parameter matrix. The mean amplitude is calculated as follows: the point cloud before the emergency stop is projected onto the image plane using the current extrinsic parameter matrix, and the optical flow field is calculated by combining the image sequence. Then take The arithmetic mean of the magnitudes of the optical flow vectors of all pixels.

[0053] This represents the optical flow field calculated after the sudden stop event by applying the estimated extrinsic parameter drift (ΔR, Δt) to the current extrinsic parameter matrix and then reprojecting the point cloud after the sudden stop onto the image plane using the updated extrinsic parameter matrix. The mean amplitude is calculated as follows: First, the point cloud after the sudden stop is reprojected onto the image plane using the updated extrinsic parameter matrix, and then the optical flow field is calculated by combining the image sequence. Then take The arithmetic mean of the magnitudes of the optical flow vectors of all pixels. This mean varies with the extrinsic parameter drift (ΔR, Δt).

[0054] Based on the two residual functions mentioned above, an objective function is constructed with the goal of minimizing their weighted sum of squares: (8) in, The weighting coefficient, symbol This indicates finding the minimum value, with the symbol... The square norm of a vector. The total number of points in the point cloud, symbol This indicates taking the absolute value. The weights used to balance the contributions of the point cloud residual and the image optical flow residual to the objective function are determined based on the confidence levels of the LiDAR and camera. In this embodiment, the weights are... =0.8.

[0055] The above objective function is solved iteratively using the Levenberg-Marquardt algorithm. The specific iterative steps are as follows: Step (1): Initialization (3×3 identity matrix). (3×1 zero vector).

[0056] Step (2): In the k-th iteration, the current state variable ( Substitute into formulas (6) and (7) above. and The function calculates the current residual vector: ; And calculate the Jacobian matrix J (i.e., the matrix of first-order partial derivatives of the residual vector with respect to the state variables ΔR and Δt, J is a 2×6 matrix).

[0057] Step (3): Solve the incremental equation to obtain increment ,in This is the damping factor (the damping coefficient of the Levenberg-Marquardt algorithm, used to adjust the weight between the Gauss-Newton step size and the gradient descent step size). It is the identity matrix. for The transpose of .

[0058] Step (4): Update the state variables: ; .

[0059] Step (5): If ||δ|| is less than the preset threshold (rotation angle 0.01°, translation amount 0.001m) or the number of iterations reaches 50, then terminate the iteration; otherwise, update the damping coefficient μ according to the current residual change and return to step (2).

[0060] After the iteration terminates, the optimal solutions ΔR and Δt are output, which are the drift of the extrinsic parameter matrix.

[0061] (iv) Extrinsic parameter matrix compensation and update: Apply the obtained extrinsic parameter matrix drift ΔR and Δt to the current extrinsic parameter matrix. (9) (10) in and The external parameter rotation matrix and translation vector before the update. and Let be the updated extrinsic rotation matrix and translation vector.

[0062] The updated extrinsic parameter matrix ( , It is used for subsequent point cloud-image fusion processing.

[0063] IV. Point cloud and image data are fused to generate a 3D model, and pixel-level fusion is performed based on the updated extrinsic parameter matrix: First, distortion correction is performed on the image, and then the pixel coordinates in the distorted image are normalized relative to the principal point.

[0064] Then, the point cloud is projected onto the image pixel coordinate system based on the updated extrinsic parameter matrix: ; ; in( () represents the coordinates of the point cloud points in the global coordinate system. ) represents the coordinates of the point in the camera coordinate system. ) represents the image pixel coordinates corresponding to this point, and K is the camera intrinsic parameter matrix.

[0065] For each point cloud point, the RGB color value of its corresponding image pixel is assigned to that point cloud point to generate a colored point cloud. For point cloud points that fall outside the image range after projection, their original geometric information is preserved but no texture value is assigned.

[0066] Poisson surface reconstruction is performed on the colored point cloud to generate a textured 3D mesh model.

[0067] V. Dynamic Time-Series Updates Construct Dynamic 3D Situational Scenes, Using the First Timestamp The corresponding first three-dimensional point cloud model As the baseline model. The point cloud is segmented according to its spatial location: for each point, the consistency of the normal vector in its local neighborhood is calculated. If the consistency of the normal vector is higher than a preset threshold (indicating that the point belongs to a planar structure), it is marked as a static background element (such as terrain, permanent buildings, etc.).

[0068] Get the second timestamp ( The corresponding second three-dimensional point cloud model The ICP algorithm is used to... and Perform point cloud registration to obtain the registration transformation matrix. .

[0069] Based on the registration results, calculate and The difference point cloud between them. For Each point in ,exist The kd-tree nearest neighbor search is used to find the nearest point. Calculate distance .like If the distance exceeds a preset threshold (e.g., 0.5 meters), the point is marked as a difference point. All difference points constitute a difference point cloud. Difference point clouds characterize disaster sites. and The dynamic change zone between them (such as the sliding of landslide bodies, the expansion of breaches, etc.).

[0070] The difference point cloud ΔM is superimposed onto the baseline model. Generate an updated dynamic 3D situation scene .

[0071] VI. AI Large-Scale Model for Disaster Element Identification. A semantic segmentation model based on the lightweight backbone network MobileNetV3 is constructed. MobileNetV3 employs depthwise separable convolution and neural architecture search techniques, significantly reducing the number of model parameters and computational load while ensuring segmentation accuracy, making it suitable for real-time operation on airborne computing platforms. This semantic segmentation network takes multi-view projection images of dynamic 3D situational scenes as input and outputs the disaster category probability for each pixel.

[0072] The semantic segmentation model was trained using a pre-annotated dataset of disaster site samples. The dataset was constructed as follows: professional personnel manually annotated landslides, breaches, collapses, and waterlogged areas at the pixel level using annotation tools on collected historical disaster site image data, with at least 2000 annotated images for each disaster category. During training, a cross-entropy loss function and the Adam optimizer were used, with an initial learning rate of 0.001 and a batch size of 16, training continued until the loss function converged.

[0073] Multiple viewpoint projection images of a dynamic 3D situational scene are input into a trained semantic segmentation model, which outputs pixel-level segmentation results of disaster elements. For each pixel, the model outputs a probability vector, and the category with the highest probability is taken as the labeled category of that pixel. The segmentation results are then back-projected into 3D space: for each 3D point, the segmentation results of each viewpoint are obtained based on its pixel coordinates projected onto the images of each viewpoint. The disaster category of the 3D point is determined by majority voting, resulting in a 3D point-level label for the disaster element.

[0074] Based on the segmentation results, the geometric parameters of each disaster element are calculated. The area is calculated as follows: for point clouds labeled with the same disaster category, a triangular mesh for that disaster element is generated using Poisson surface reconstruction, and the mesh surface area is calculated. The volume is calculated as follows: using the local terrain where the disaster element point cloud is located as a reference plane, the volume difference between the disaster element point cloud and the reference plane is calculated. Spatial location is obtained by calculating the center coordinates of the disaster element point cloud.

[0075] VII. Generate Analysis Report and 3D Visualization Situation Map. Based on the identification results of disaster elements, generate a list of disaster elements. Each record in the list includes: disaster type (landslide / breach / collapse / waterlogged area), spatial coordinates (longitude, latitude, elevation), geometric dimensions (area, volume, length, width), and identification confidence level (i.e., the maximum probability output by the model).

[0076] Based on a dynamic 3D situational scenario, a 3D visualization situation map is generated. In this map, different disaster elements are distinguished and displayed using different colors: landslides are shown in red, breaches in orange, collapses in yellow, and waterlogged areas in blue. Label information for these disaster elements (including disaster type and geometric dimensions) is overlaid on the 3D scene.

[0077] The list of disaster elements is displayed in tabular form on the side of the 3D visualization situation map, generating a comprehensive analysis report to enable emergency command personnel to quickly grasp the disaster situation.

[0078] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A multimodal reconstruction method for disaster scene situation based on AI large model, characterized in that, include: The system acquires point cloud data collected by the lidar on the drone, pose data collected by navigation, and image data collected by the camera; wherein the spatial transformation relationship between the lidar coordinate system and the camera coordinate system is represented by an extrinsic parameter matrix, which includes a rotation matrix and a translation vector. During the flight of the UAV, the flight status is monitored. When an emergency stop event is detected, a data acquisition time window is triggered. Within the data acquisition time window, the point cloud data frame sequence of the lidar and the image frame sequence of the camera are acquired. Based on the point cloud data frame sequence and image frame sequence within the acquisition time window, determine the drift of the extrinsic parameter matrix between the LiDAR and the camera; including: The point cloud data frame sequence within the acquisition time window is divided into a sequence before the emergency stop event and a sequence after the emergency stop event. A first point cloud model is generated based on the sequence before the emergency stop event, and a second point cloud model is generated based on the sequence after the emergency stop event. The second point cloud model is registered with the first point cloud model to obtain the point cloud registration residual; The first optical flow field before the emergency stop event and the second optical flow field after the emergency stop event are extracted from the image frame sequence within the acquisition time window. The optical flow field is used to characterize the motion vector of pixels in the image between adjacent frames. The image optical flow residual is obtained based on the first optical flow field and the second optical flow field. Based on the point cloud registration residual and the image optical flow residual, determine the drift of the extrinsic parameter matrix; including: using the drift of the extrinsic rotation matrix ΔR and the drift of the extrinsic translation vector Δt as the state variables to be estimated, constructing an objective function constrained by the point cloud registration residual and the image optical flow residual, and solving the objective function to obtain ΔR and Δt; The objective function is: Register the point cloud residual function With image optical flow residual function Minimize the weighted sum of squares, the point cloud registration residual function Defined as: ； The image optical flow residual function Defined as: ； in, This indicates the use of the extrinsic parameter drift amount The updated extrinsic matrix obtained after applying the current extrinsic matrix is ​​used to project the point cloud onto the image plane and calculate the mean amplitude of the optical flow field after the sudden stop event. This represents the mean amplitude of the optical flow field before the emergency stop event, calculated after projecting the point cloud onto the image plane using the current extrinsic parameter matrix. The objective function is: ； And find the solution that minimizes the objective function. and ; in, The extrinsic rotation matrix drift is... The displacement of the extrinsic translation vector. and The rotation matrix and translation vector are obtained by iteratively registering the second point cloud model with the first point cloud model to the nearest point. For points in the second point cloud model, For the first point cloud model and The corresponding nearest point, The weighting coefficient, symbol This indicates finding the minimum value, with the symbol... The square norm of a vector. The total number of points in the point cloud, symbol Indicates taking the absolute value; The current extrinsic parameter matrix is ​​updated and compensated based on the drift of the extrinsic parameter matrix. Based on the updated extrinsic parameter matrix, the point cloud data and the image data are fused to generate a 3D model; The disaster elements in the three-dimensional model are identified, and three-dimensional situation analysis results of the disaster site are generated.

2. The method according to claim 1, characterized in that, The detection of the emergency stop event includes: when the UAV's flight speed drops from greater than a first speed threshold to less than a second speed threshold, and the rate of change of acceleration is greater than a first rate of change of acceleration threshold, an emergency stop event is determined to have occurred.

3. The method according to claim 1, characterized in that, The construction of the acquisition time window includes: taking the trigger time of the emergency stop event as the base time, backtracking for a first preset duration, and extending forward for a second preset duration to construct the acquisition time window.

4. The method according to claim 1, characterized in that, The process of registering the second point cloud model with the first point cloud model includes: using the iterative nearest point algorithm to register the second point cloud model with the first point cloud model, and calculating the sum of squared residuals of the registered point clouds as the point cloud registration residuals.

5. The method according to claim 1, characterized in that, The step of obtaining the image optical flow residual based on the first optical flow field and the second optical flow field includes: calculating the mean amplitude of the first optical flow field and the mean amplitude of the second optical flow field, and taking the absolute value of the difference between the two as the image optical flow residual.

6. The method according to claim 1, characterized in that, The compensation update of the current extrinsic parameter matrix includes: applying the rotation matrix drift ΔR and translation vector drift Δt from the drift of the extrinsic parameter matrix to the current extrinsic parameter matrix. ； ； in and The external parameter rotation matrix and translation vector before the update. and Let be the updated extrinsic rotation matrix and translation vector.

7. A multimodal disaster scene situation reconstruction system based on AI large model, characterized in that, The system is used to perform the method according to any one of claims 1-6, comprising: The data acquisition module is used to acquire point cloud data collected by the lidar on the UAV, pose data collected by the integrated navigation system, and image data collected by the camera; wherein, the spatial transformation relationship between the lidar coordinate system and the camera coordinate system is represented by an extrinsic parameter matrix, which includes a rotation matrix and a translation vector; The emergency stop detection module is used to monitor the flight status during the flight of the UAV. When an emergency stop event is detected, a data acquisition time window is triggered, and the point cloud data frame sequence of the lidar and the image frame sequence of the camera are acquired within the data acquisition time window. The calibration module is used to determine the drift of the extrinsic parameter matrix between the lidar and the camera based on the point cloud data frame sequence and image frame sequence within the acquisition time window, including: The point cloud data frame sequence within the acquisition time window is divided into a sequence before the emergency stop event and a sequence after the emergency stop event. A first point cloud model is generated based on the sequence before the emergency stop event, and a second point cloud model is generated based on the sequence after the emergency stop event. The second point cloud model is registered with the first point cloud model to obtain the point cloud registration residual; The first optical flow field before the emergency stop event and the second optical flow field after the emergency stop event are extracted from the image frame sequence within the acquisition time window. The optical flow field is used to characterize the motion vector of pixels in the image between adjacent frames. The image optical flow residual is obtained based on the first optical flow field and the second optical flow field. Based on the point cloud registration residual and the image optical flow residual, determine the drift of the extrinsic parameter matrix; including: using the drift of the extrinsic rotation matrix ΔR and the drift of the extrinsic translation vector Δt as the state variables to be estimated, constructing an objective function constrained by the point cloud registration residual and the image optical flow residual, and solving the objective function to obtain ΔR and Δt; The objective function is: Register the point cloud residual function With image optical flow residual function Minimize the weighted sum of squares, the point cloud registration residual function Defined as: ； The image optical flow residual function Defined as: ； in, This indicates the use of the extrinsic parameter drift amount The updated extrinsic matrix obtained after applying the current extrinsic matrix is ​​used to project the point cloud onto the image plane and calculate the mean amplitude of the optical flow field after the sudden stop event. This represents the mean amplitude of the optical flow field before the emergency stop event, calculated after projecting the point cloud onto the image plane using the current extrinsic parameter matrix. The objective function is: ； And find the solution that minimizes the objective function. and ; in, The extrinsic rotation matrix drift is... The displacement of the extrinsic translation vector. and The rotation matrix and translation vector are obtained by iteratively registering the second point cloud model with the first point cloud model to the nearest point. For points in the second point cloud model, For the first point cloud model and The corresponding nearest point, The weighting coefficient, symbol This indicates finding the minimum value, with the symbol... The square norm of a vector. The total number of points in the point cloud, symbol Indicates taking the absolute value; The extrinsic parameter matrix is ​​then compensated and updated based on the drift of the extrinsic parameter matrix. The fusion module is used to fuse the point cloud data and the image data according to the updated extrinsic parameter matrix to generate a three-dimensional model; The identification output module is used to identify disaster elements in the three-dimensional model and generate three-dimensional situation analysis results of the disaster site.

Images