Autonomous detection unmanned aerial vehicle control system for closed space disaster rescue

Abstract

The application provides an autonomous detection unmanned aerial vehicle control system for closed space disaster rescue, relates to the technical field of unmanned aerial vehicle detection and rescue, and comprises: a multi-modal environment perception module, which is used for synchronously collecting laser point cloud data and visual image data of a closed space, and performing space-time alignment and semantic fusion processing on the laser point cloud data and the visual image data to generate semantic enhanced point cloud data; a tightly coupled SLAM positioning and map construction module, which is used for solving the pose of an unmanned aerial vehicle in a closed space in real time based on the semantic enhanced point cloud data by tightly coupling correlation and fusion of visual features and laser features and inertial measurement data, and synchronously incrementally constructing a three-dimensional environment map with semantic information; and an obstacle avoidance and path planning module, which is used for constructing and updating a three-dimensional environment map according to the real-time solved unmanned aerial vehicle pose. The application can realize accurate positioning mapping, efficient obstacle avoidance, accurate personnel identification and real-time rescue decision support in an extreme environment.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) detection and rescue technology, and in particular to an autonomous detection UAV control system for disaster relief in confined spaces. Background Technology

[0002] In disaster relief scenarios involving confined spaces such as tunnels, mines, and underground parking lots, the environment often presents extreme characteristics such as GNSS signal rejection, unstructured structures, and complex lighting. Traditional manual detection methods are inefficient and pose extremely high safety risks. Drones, with their flexibility and maneuverability, have become core equipment for disaster detection and the search for trapped personnel in such scenarios.

[0003] However, existing rescue drones suffer from the following technical shortcomings: in confined spaces denied by GNSS, they lack the ability to achieve high-precision positioning and semantic map construction through tight coupling of visual and laser technologies. For example, due to the sparse feature points, repetitive textures, or abrupt changes in illumination in confined spaces, single visual sensors are prone to feature matching failures, single laser radars struggle to capture environmental semantic information, and traditional multi-sensor fusion is mostly loosely coupled, failing to achieve deep correlation between visual and laser features. This results in drones being unable to continuously output high-precision pose data in the absence of GNSS signals, and the constructed maps only contain geometric structures and lack semantic annotations, making it difficult to meet the refined environmental information requirements of subsequent missions.

[0004] This defect leads to a series of related problems: due to insufficient positioning accuracy and lack of map semantics, drones have difficulty accurately judging their spatial relationship with obstacles, lack reliable basis for autonomous obstacle avoidance, are prone to collisions, and cannot safely and continuously complete the detection mission; moreover, based on unreliable positioning data and semantically empty maps, it is difficult to effectively correlate personnel identification information from multiple sensors in time and space, resulting in insufficient 3D positioning accuracy and status recognition accuracy of trapped personnel, which in turn delays the best rescue opportunity and reduces the success rate of rescue. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide an autonomous detection unmanned aerial vehicle (UAV) control system for disaster relief in confined spaces, which can achieve accurate positioning and mapping, efficient obstacle avoidance, accurate personnel identification and real-time rescue decision support in extreme environments, thereby improving the efficiency and safety of disaster relief in confined spaces.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] An autonomous detection unmanned aerial vehicle (UAV) control system for disaster relief in confined spaces includes:

[0008] The multimodal environment perception module is used to simultaneously collect laser point cloud data and visual image data in a confined space, and to perform spatiotemporal alignment and semantic fusion processing on the laser point cloud data and visual image data to generate semantically enhanced point cloud data.

[0009] The tightly coupled SLAM localization and mapping module is used to calculate the pose of the UAV in a confined space in real time based on semantically enhanced point cloud data, by tightly coupling and fusing visual features and laser features with inertial measurement data, and synchronously incrementally construct a 3D environment map with semantic information.

[0010] The obstacle avoidance and path planning module is used to identify obstacles in the environment and analyze passable areas based on the real-time calculated UAV pose and the constructed and updated 3D environment map, combined with real-time acquired semantically enhanced point cloud data, and then plan and dynamically adjust the collision-free flight path of the UAV online.

[0011] The trapped personnel identification and analysis module is used to identify trapped personnel and determine their precise location in the 3D environment map by performing multi-source detection and spatiotemporal correlation based on semantically enhanced point cloud data and constructed 3D environment map, and further analyze the trapped personnel's posture and vital signs information.

[0012] The data feedback and intelligent decision-making module is used to transmit the constructed 3D environment map, the planned collision-free flight path of the UAV, and the precise location and vital signs of the trapped personnel back to the command platform through an adaptive anti-interference communication network. This allows for the fusion of multi-source data to generate a global rescue situation and, based on this, generate rescue decision support information.

[0013] The above-described solution of the present invention has at least the following beneficial effects:

[0014] By employing multimodal sensor spatiotemporal alignment and semantic fusion technology, vision-laser tightly coupled SLAM, and factor graph model optimization, the core problems of insufficient positioning accuracy and missing semantic maps in GNSS-denied environments are overcome, thereby achieving high-precision autonomous positioning of UAVs in confined spaces and the construction of 3D maps with semantic information. By using multi-source detection spatiotemporal correlation and attitude recognition fusion calculation, the low accuracy of personnel identification using a single sensor is overcome, thereby improving the 3D positioning accuracy of trapped personnel and the reliability of vital sign assessment. By employing dual-band Mesh self-organizing network and global data fusion decision-making technology, the problems of communication obstruction and fragmented rescue information in confined spaces are overcome, thereby ensuring real-time transmission of key data and visualization of the overall rescue situation. Because the modules work together to form a closed loop of detection, transmission, and decision-making, the shortcomings of existing UAV systems in terms of autonomy and intelligence are overcome, thereby improving the efficiency of disaster rescue in confined spaces and the safety of rescue personnel. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of an autonomous detection unmanned aerial vehicle (UAV) control system for disaster relief in confined spaces, provided by an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the process of constructing a three-dimensional environmental map with semantic information in real time by using semantically enhanced point cloud data, tightly coupling and associating visual features and laser features with inertial measurement data, and synchronously incrementally constructing the pose of a UAV in a confined space. Detailed Implementation

[0017] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0018] like Figure 1 As shown, embodiments of the present invention propose an autonomous detection unmanned aerial vehicle (UAV) control system for disaster relief in confined spaces, comprising:

[0019] The multimodal environment perception module is used to simultaneously collect laser point cloud data and visual image data in a confined space, and to perform spatiotemporal alignment and semantic fusion processing on the laser point cloud data and visual image data to generate semantically enhanced point cloud data.

[0020] The tightly coupled SLAM localization and mapping module is used to calculate the pose of the UAV in a confined space in real time based on semantically enhanced point cloud data, by tightly coupling and fusing visual features and laser features with inertial measurement data, and synchronously incrementally construct a 3D environment map with semantic information.

[0021] The obstacle avoidance and path planning module is used to identify obstacles in the environment and analyze passable areas based on the real-time calculated UAV pose and the constructed and updated 3D environment map, combined with real-time acquired semantically enhanced point cloud data, and then plan and dynamically adjust the collision-free flight path of the UAV online.

[0022] The trapped personnel identification and analysis module is used to identify trapped personnel and determine their precise location in the 3D environment map by performing multi-source detection and spatiotemporal correlation based on semantically enhanced point cloud data and constructed 3D environment map, and further analyze the trapped personnel's posture and vital signs information.

[0023] The data feedback and intelligent decision-making module is used to transmit the constructed 3D environment map, the planned collision-free flight path of the UAV, and the precise location and vital signs of the trapped personnel back to the command platform through an adaptive anti-interference communication network. This allows for the fusion of multi-source data to generate a global rescue situation and, based on this, generate rescue decision support information.

[0024] In this embodiment of the invention, a multimodal environment perception module achieves precise data fusion and semantic enhancement, combined with tightly coupled SLAM technology, enabling high-precision autonomous positioning of UAVs and the construction of 3D maps with semantic information in confined spaces denied by GNSS. An obstacle avoidance and path planning module can identify obstacles in real time and dynamically adjust flight paths, ensuring safe and stable exploration by the UAV. A trapped personnel identification and analysis module, through multi-source detection and spatiotemporal correlation, accurately locates trapped personnel and analyzes their attitude and vital signs, improving the personnel detection rate and identification accuracy. A data feedback and intelligent decision-making module achieves real-time feedback of key information through an adaptive anti-interference communication network. The command platform integrates multi-source data to generate a global rescue situation and provides decision support, effectively improving the intelligence level, rescue efficiency, and safety of rescue personnel in confined space disaster rescue.

[0025] In a preferred embodiment of the present invention, laser point cloud data and visual image data of a confined space are simultaneously acquired, and spatiotemporal alignment and semantic fusion processing are performed on the laser point cloud data and visual image data to generate semantically enhanced point cloud data, including:

[0026] The system controls the lidar and vision sensors on the drone to simultaneously collect raw lidar point cloud data and raw visual image data in a confined space. Specifically, this involves activating the drone's 360-degree horizontal scanning lidar and RGBD camera via unified control commands, setting the same acquisition frequency to ensure synchronous operation. The lidar continuously captures the three-dimensional spatial position, outline, and distance information between various objects in the confined space, forming continuous raw lidar point cloud data. The RGBD camera simultaneously captures the scene within its detection range, simultaneously acquiring color image data containing color details of object surfaces and depth image data recording the distance between objects and the camera. This ensures that the two sets of raw data are completely corresponding in the acquisition sequence and can comprehensively cover the environmental information around the drone.

[0027] The acquired raw laser point cloud data and raw visual image data are time-stamped and spatially aligned. Simultaneously, distortion correction is performed on the raw visual image data to obtain time- and space-aligned laser point cloud data and visual image data. Specifically, this includes: using the PPS second pulse signal output from the UAV's onboard high-precision clock as a unified time reference, performing microsecond-level calibration on the timestamps of each frame of raw laser point cloud data and raw visual image data; calculating the time difference between the two types of data and eliminating invalid data frames with a time deviation exceeding 50 microseconds to ensure complete synchronization of each corresponding set of data at the time of acquisition; and constructing a corresponding coordinate transformation matrix based on pre-obtained sensor extrinsic parameters using a 12×9 checkerboard calibration board, including the relative position coordinates and attitude angles of the LiDAR and RGBD camera, for each three-dimensional coordinate system in the laser point cloud. Each punctuation point is subjected to matrix operations to transform it from the coordinate system of the LiDAR itself to the coordinate system of the UAV. At the same time, combined with the depth information of the RGBD camera, the three-dimensional coordinates corresponding to each pixel in the visual image are calculated, and then the spatial transformation is completed through the same coordinate transformation matrix to achieve complete unification of the spatial dimensions of the two types of data. In response to the radial and tangential distortion problems of the RGBD camera lens, based on the set camera intrinsic parameters, including focal length, principal point coordinates, and distortion coefficient, the distortion offset of each pixel caused by the optical characteristics of the lens is first calculated, and then the correct position of the pixel in the distortion-free image is deduced in reverse based on the offset. The coordinates of pixels with severe distortion in the edge area are adjusted to correct problems such as straight line bending and object contour deformation in the image. Finally, time-synchronized, spatially unified, and geometrically distortion-free LiDAR point cloud data and visual image data are obtained.

[0028] The semantic information of pixels in the temporally and spatially aligned visual image data is registered and fused with the temporally and spatially aligned laser point cloud data at the point cloud level. This assigns corresponding color and texture semantic features to the points in the laser point cloud data, obtaining semantically colored intermediate point cloud data. Specifically, this involves: performing pixel-by-pixel fine analysis on the temporally and spatially aligned visual image data; conducting recognition based on a pre-set multi-category object feature template library, which includes features such as the uniformity of planar texture of walls, the color gradient of the ground, the irregular contours and size ranges of obstacles, the typical height and weight ratio of the human body, limb contour shapes, and skin grayscale range features; for each pixel, extracting its RGB value, local grayscale gradient changes, and the correlation between adjacent pixels, and comparing and matching these with various target features in the feature template library one by one to determine the specific target category corresponding to the pixel, generating clearly defined categories such as walls, ground, obstacles, and suspected human bodies. The system identifies the semantic information of each semantic pixel and records its two-dimensional coordinates in the image. Then, combining the spatially and temporally aligned visual image depth information with the set camera shooting distance parameters, it calculates the three-dimensional coordinates of each semantic pixel in the UAV's body coordinate system, ensuring that these three-dimensional coordinates are completely consistent with the spatial coordinate system of the laser point cloud data. Using the three-dimensional coordinates of the semantic pixels as a reference, it searches for laser points in the spatially and temporally aligned laser point cloud data with coordinate errors within 0.03 meters, accurately assigning the color features and category semantic labels of the semantic pixels to the corresponding laser points. For areas with high laser point cloud density and where adjacent laser points do not match semantic pixels, it uses interpolation based on the category information and color gradient of adjacent matched semantic pixels to ensure that each point in the laser point cloud data is assigned corresponding color details and clear semantic attributes, ultimately forming semantically colored intermediate point cloud data that combines three-dimensional geometric shape, true color information, and clear category identification.

[0029] Based on semantically colored intermediate point cloud data, an enhanced point cloud incorporating visual semantic information is output as semantically enhanced point cloud data. Specifically, this includes: noise filtering of the semantically colored intermediate point cloud data; setting a neighborhood radius threshold of 0.1 meters; counting the number of neighboring points within this radius for each point; and identifying and removing isolated, scattered points with fewer than 5 neighboring points as noise points. Simultaneously, based on the effective detection range of the LiDAR (10 to 50 meters), abnormal data points exceeding this range are removed to avoid invalid data interfering with subsequent processing. For redundant point cloud data with spatial spacing less than 0.05 meters and identical semantic labels, their average coordinates are calculated. Semantic labels are retained for individual points to reduce data volume while ensuring the integrity of environmental information. Subsequently, the processed intermediate point cloud data is standardized and converted to the PCD standard format. The 3D coordinates are retained to six decimal places, and the semantic labels are numerically encoded, such as 1=wall, 2=ground, 3=obstacle, 4=suspected human body. The coordinate representation adopts the right-hand coordinate system rule to ensure that the data format can be seamlessly used by subsequent positioning and mapping, obstacle recognition, personnel detection and other processes. Finally, the enhanced point cloud that integrates the 3D geometric information of LiDAR, the color information of visual image and the semantic category information is output as semantically enhanced point cloud data.

[0030] like Figure 2 As shown, in another preferred embodiment of the present invention, based on semantically enhanced point cloud data, the pose of the UAV in a confined space is calculated in real time by tightly coupling and fusing visual features and laser features with inertial measurement data, and a three-dimensional environment map with semantic information is incrementally constructed simultaneously, including:

[0031] Geometric and edge features of laser point clouds are extracted from semantically enhanced point cloud data. Simultaneously, visual feature points and their descriptors are extracted from the corresponding visual image data. Specifically, for the generated semantically enhanced point cloud data, key category regions labeled as walls, obstacles, or suspected human bodies, with a point cloud coverage exceeding 30%, are prioritized. These regions are then divided into several independent sub-regions using a 1m x 1m x 1m cube size to ensure each sub-region accurately reflects local environmental features. For laser points within each sub-region, a neighborhood radius of 0.1 meters is set based on the measurement accuracy of the lidar. The neighborhood space of each laser point is defined using this radius, and the Euclidean distance between each point and all points in its neighborhood is calculated. Points with distances within the 0.1-meter range are selected. The effective neighboring points are determined by: first, calculating the mean of the three-dimensional coordinates of these effective neighboring points; then, subtracting the mean from the coordinates of each neighboring point to obtain decentralized coordinates; calculating the covariance of the decentralized coordinates; solving for the eigenvalues ​​and eigenvectors corresponding to the covariance; and selecting the eigenvector corresponding to the largest eigenvalue as the normal vector of the neighboring point set. Taking the current laser point as the center, selecting adjacent laser points in the four directions (front, back, left, and right) within the sub-region, and repeating the above operation to obtain the normal vector of the neighborhood of each adjacent laser point. Then, calculating the angle between the normal vector of the current laser point and the normal vector of each adjacent point, taking the average of these angles as the angle change, and combining this with the spatial distance between adjacent laser points to calculate the rate of change of the normal vector, thereby determining the curvature of the laser point's location.

[0032] A curvature threshold is set. When the curvature value of a laser point exceeds three times the average curvature of all laser points in its sub-region, it is considered a curvature abrupt change. Simultaneously, the point cloud density in the neighborhood of this laser point is calculated. By statistically analyzing the variance of the neighborhood points in space, if the variance exceeds a preset threshold, it is considered an uneven distribution of neighborhood points. Points selected based on these two conditions are used as laser geometric feature points that can accurately characterize the geometric structure of the environment. Furthermore, the overall contour of the semantically enhanced point cloud is scanned frame by frame. When the point cloud density difference between adjacent sub-regions exceeds 50%, or the turning angle of the spatial contour exceeds 30 degrees, these key turning points are identified. The points are connected sequentially to form continuous laser edge features, fully presenting the outline of the object. At the same time, for the processed visual image data without geometric distortion and with spatiotemporal alignment, the focus is on key areas with gray value changes exceeding 80, such as object corners and light-dark boundaries, and corner points and edge points are extracted as visual feature points. With each visual feature point as the center, a 32×32 pixel neighborhood window is selected, and multi-dimensional information such as gray-level gradient changes in the horizontal and vertical directions and texture distribution frequency within the window is statistically analyzed to generate a descriptor that can uniquely represent the visual feature point, ensuring that the descriptor can accurately reflect the local image attributes of the feature point.

[0033] The extracted visual feature points and descriptors are projected onto the corresponding laser point cloud frames, and associated and matched with geometric feature points and edge features to form a feature set tightly coupled with the laser. Based on this feature set, laser odometry constraints between adjacent frames are calculated. Specifically, this includes: calling calibrated camera intrinsic parameters and sensor extrinsic parameters, where the camera intrinsic parameters include the horizontal and vertical focal lengths and the horizontal and vertical coordinates of the image's principal point; and the sensor extrinsic parameters include the relative translation data and attitude rotation parameters between the UAV body and the RGBD camera. Combining the microsecond-level timestamp alignment reference between the visual image and the laser point cloud, the cross-coordinate system transformation process for the visual feature points is initiated.

[0034] First, the depth value corresponding to the pixel position of each visual feature point is accurately extracted from the processed depth image. The unit of the depth value is uniformly converted to meters and three decimal places are retained to ensure that the accuracy of the depth data meets the requirements of subsequent calculations. Based on the pinhole camera model, a mapping relationship between pixel coordinates and camera coordinate system is constructed. The specific operation logic is as follows: taking the principal point coordinates of the image as the reference, the offset of the pixel coordinates of the visual feature point relative to the principal point coordinates in the horizontal and vertical directions is calculated respectively. Then, these two offsets are multiplied by the corresponding depth value respectively. The results are divided by the focal length of the camera in the corresponding direction to obtain the horizontal and vertical coordinates of the visual feature point in the RGBD camera's own coordinate system. The depth coordinates in the camera coordinate system are directly equal to the extracted depth value. Through this series of operations, the conversion from two-dimensional pixel coordinates to three-dimensional camera coordinates is completed.

[0035] Then, a mapping is performed from the camera coordinate system to the UAV body coordinate system. First, the three-dimensional coordinates in the camera coordinate system are expanded into four-dimensional homogeneous coordinates. Then, the rotation matrix in the sensor extrinsic parameters is used to adjust the spatial orientation of the coordinates by rotation. After that, the translation vector in the sensor extrinsic parameters is added to complete the spatial position migration of the coordinates. Finally, the three-dimensional projected coordinates of each visual feature point in the UAV body coordinate system are obtained. During the conversion process, the single-point position error needs to be checked in real time. By comparing the deviation between the projected coordinates and the actual distribution of the laser point cloud, it is ensured that the single-point position error of each conversion does not exceed 0.01 meters.

[0036] Centered on the 3D projected coordinates and considering the ±2 cm measurement accuracy of the lidar, a spherical search range of 0.05 meters was set. When screening candidate laser feature points, the coordinate differences between each extracted laser geometric feature point, edge feature point, and the 3D projected coordinates in the horizontal, vertical, and depth directions were first calculated. Then, the differences in each direction were squared, and the square root of the sum of the squared results in all three directions was calculated to obtain the straight-line distance between the two points. Only laser feature points with a distance not exceeding 0.05 meters were retained as candidate points. Simultaneously, the number of point clouds within a 0.03-meter radius of each candidate point was counted, and only candidate points with at least 10 point clouds were retained, eliminating isolated noise points. To ensure that candidate feature points accurately reflect the actual characteristics of the environment, a semantic consistency check is then initiated. The numerically encoded semantic label corresponding to each candidate laser feature point is read from the semantically enhanced point cloud. The encoding rule for the semantic label is: 1 represents a wall, 2 represents the ground, 3 represents an obstacle, and 4 represents a suspected human body. Simultaneously, in the pixel semantic map generated in step 1.3, the pixel position corresponding to the visual feature point is accurately located, and the semantic code corresponding to that pixel is extracted. By comparing the semantic code values ​​of the candidate laser feature point and the visual feature point bit by bit, it is ensured that the two are completely consistent, for example, both being obstacle codes or both being suspected human body codes. This prevents cross-matching between dynamic and static categories and ensures the semantic rationality of feature matching.

[0037] After semantic verification, two-dimensional similarity matching is performed. For the visual descriptor, a 32×32 pixel neighborhood window is selected centered on the visual feature point. The gray-level histogram within the window is extracted, and statistical information such as the gray-level mean, gray-level standard deviation, and gray-level gradient direction distribution of the region is calculated. At the same time, the gray-level statistical information of the image region corresponding to the candidate laser feature point is extracted, and the gray-level distribution similarity between the two is calculated by normalizing the cross-correlation coefficient. The calculation process is as follows: first, the gray-level mean of the two regions is calculated separately, then the deviation of the gray-level value of each corresponding pixel from the mean of its region is calculated one by one, then the sum of the products of the deviations of all corresponding pixels and the sum of the squares of the deviations of the two regions are calculated, and finally the sum of the products of the deviations is divided by the product of the square roots of the two sums of squares. The result is the gray-level distribution similarity. The closer this value is to 1, the more similar the gray-level distributions of the two regions are.

[0038] For laser feature points, geometric features such as the curvature of the point cloud within a 0.05-meter neighborhood and the variance of the point cloud distribution in space are calculated. Then, the texture gradient variance of the visual feature point's neighborhood is extracted. The spatial structure matching degree is obtained by analyzing the correlation between the two sets of data. A weighted summation method is used to fuse the two similarities, with grayscale distribution similarity accounting for 60% and spatial structure matching degree accounting for 40%. When the total similarity obtained by the weighted summation exceeds 85%, the visual feature point is considered to have initially matched the candidate laser feature point. To eliminate false matches, the RANSAC algorithm is introduced for fine verification: four pairs of non-collinear feature points are randomly selected from the initially matched feature pairs. Based on the coordinate relationship of these points, a mapping relationship between the visual image and the laser point cloud is fitted. Through this mapping relationship, the three-dimensional coordinates corresponding to the laser feature point are reprojected onto the image pixel coordinate system. The straight-line distance between the reprojected pixel coordinates and the original visual feature point pixel coordinates is calculated. An allowable error threshold of 1 pixel is set, and feature pairs with reprojection errors less than this threshold are retained as inliers. Repeat the above operations of random point selection, fitting mapping, error calculation, and filtering of interior points. After a total of 200 iterations, select the matching pair corresponding to the mapping model with the most interior points to form a tightly coupled feature set with one-to-one correspondence between vision and laser, ensuring that the accuracy of the matching pairs in the set is not less than 99%.

[0039] Based on this tightly coupled feature set, key feature point pairs with a matching confidence level exceeding 90% were selected. Corresponding feature points were extracted from adjacent laser point clouds. The coordinate differences of each pair of correlated feature points in the horizontal, vertical, and depth directions were calculated to obtain the three-dimensional displacement data of each pair. Weights were assigned to each displacement data point according to the matching confidence level. The displacement data of all feature point pairs in each direction were multiplied by their corresponding weights, summed, and then divided by the sum of all weights to obtain the overall relative displacement of adjacent frames in that direction. The coordinate data of the feature point pairs were processed using the SVD decomposition method, which decomposes the transformation process of the coordinate data. For the three basic operations of rotation, scaling, and rotation, the optimal rotation direction and translation distance parameters that minimize the matching error of all feature points are obtained by analyzing the inherent characteristics of the data. The obtained rotation parameters are then converted into three attitude angles of the UAV: ​​roll angle, pitch angle, and yaw angle, to obtain the attitude change angle between adjacent frames. Combining the comprehensive measurement accuracy of lidar and vision sensors, the laser odometry constraint threshold is set: the relative position deviation of the UAV between adjacent frames does not exceed 3 cm and the attitude deviation does not exceed 0.5 degrees. At the same time, the 95% confidence interval of the constraint is marked to provide a constraint basis with both accuracy and reliability for subsequent pose calculation.

[0040] By fusing IMU pre-integrated data provided by the inertial measurement unit (IMU), a factor graph model for pose estimation is incorporated, integrating the tightly coupled feature set of vision and laser, the IMU pre-integrated data, and laser odometry constraints. Specifically, this involves: acquiring raw angular velocity and acceleration data of the UAV in real-time at a frequency of 1000Hz from the IMU; setting a short-term integration time window of 50 milliseconds, with each window corresponding to 50 sets of continuous sampling data to ensure complete capture of the UAV's motion state over a short period; and performing hierarchical preprocessing on the raw data within each window: firstly, a second-order low-pass filter is used with a cutoff frequency of 20Hz. This filtering method effectively filters high-frequency vibration noise generated by UAV propeller rotation and fuselage shaking, while preserving the original characteristics of the motion signal to the greatest extent possible, avoiding signal distortion. Next, zero bias correction is performed. First, the initial zero bias value calibrated by static sampling for 30 seconds before the sensor leaves the factory is subtracted. Then, combined with the real-time motion status of the UAV, the zero bias drift is compensated by online dynamic estimation. This method is particularly suitable for scenarios with fluctuating temperatures, such as tunnel construction, and can promptly correct the zero bias shift caused by temperature changes, ensuring the baseline stability of angular velocity and acceleration data.

[0041] In the gravity compensation stage, the projection directions of gravitational acceleration along the three axes of the aircraft coordinate system are determined based on the current roll, pitch, and yaw angles of the UAV. The corresponding gravity projection components along each axis are calculated, and these components are then removed from the acceleration measurements using vector subtraction. This completely eliminates the interference of gravity on the extraction of motion acceleration, ensuring that the acquired acceleration data only reflects the UAV's own motion state. The integration operation uses a trapezoidal integration method. The 50-millisecond integration window is evenly divided into 49 equal sub-intervals according to the sampling interval. For the preprocessed angular velocity and acceleration data within each sub-interval, the average value of the measurements from two adjacent sampling points is taken, multiplied by the 0.1-millisecond time interval between the sub-intervals to obtain the integration result for that sub-interval. The integration values ​​of all sub-intervals are accumulated to complete the calculation for the entire window. Specifically, angular velocity integration obtains the attitude increment through continuous accumulation, while acceleration integration first obtains the velocity increment, and then obtains the relative displacement through a second integration. During the integration process, the propagation of data errors is monitored simultaneously, and the integration accuracy is dynamically adjusted based on the motion characteristics of the confined space rescue scenario. The resulting IMU pre-integration data can accurately capture instantaneous motion trends such as rapid turns and sudden acceleration and deceleration of the drone. In continuous operation scenarios such as indoor warehouse inspection and tunnel construction navigation, the cumulative error can be stably controlled within 0.1cm / ms.

[0042] The pose estimation factor graphical model is an improvement on the ISAM2 incremental optimization architecture of GTSAM. While the original ISAM2 architecture excels at handling large-scale sparse optimization problems and supports incremental fusion of multi-sensor data, it has limitations in multimodal data complementarity constraints and robustness to dynamic scenes. To address the pose estimation requirements of UAV vision, laser, and inertial multi-sensor fusion localization, this invention optimizes three core modules: First, a semantic consistency constraint association mechanism is added, confirming dynamic feature points by setting a semantic confidence threshold ≥0.8, adapting to the semantic verification results in step 2.2. Second, the noise model of the IMU pre-integration factor is reconstructed, dynamically updating the noise covariance matrix based on the error propagation characteristics of each integration window, adapting to the error patterns of short-term integration windows. Third, the constraint factor weight allocation strategy is optimized, introducing a sensor data quality assessment index. When the LiDAR data confidence is below 0.6, the IMU constraint weight ratio is automatically increased, balancing the complementary advantages of vision, laser, and inertial data, making it more suitable for pose estimation in dynamic indoor environments, such as warehouses with moving personnel and complex structural scenes like tunnels and factories.

[0043] The core variable nodes of the factor graph model include the 6-DOF pose parameters of the UAV and the 3D coordinates of map feature points. Each time window corresponds to a UAV pose node, which includes the 3D position coordinates (X, Y, Z) and 3D attitude angles, roll angle φ, pitch angle θ, and yaw angle ψ. The Lie algebra se(3) is used to simplify the optimization calculation. The node ID is incremented in order of timestamp. The map feature point node stores the 3D coordinates (X, Y, Z) of the successfully matched laser and vision joint feature points. m Y m Z m The system includes semantic label encoding (1=wall, 2=ground, 3=obstacle, 4=suspected human body), and introduces an ORB feature response value screening mechanism (response value ≥200) to retain high-confidence feature points. After dynamic feature point identification, its weight in optimization will be temporarily reduced.

[0044] The constraint factors are designed in three categories, all of which use robust kernel functions (Huber kernels) to suppress the influence of outliers. The reprojection error constraint factor is constructed based on the reprojection error in step 2.2. During calculation, distortion correction is first performed using camera intrinsic parameters, and then the map feature points are projected onto the image pixel coordinate system. The Euclidean distance between the map feature points and the original visual feature point coordinates is calculated, with an allowable error range of ±1 pixel. The factor weight corresponding to dynamic targets is temporarily reduced by 50%. The IMU motion constraint factor is associated with adjacent pose nodes and incorporates the noise covariance matrix estimated in real time during the pre-integration process to accurately model the uncertainty of IMU measurement. The laser odometry constraint factor connects adjacent frame pose nodes. The noise model is dynamically adjusted according to the ±2cm measurement accuracy of the lidar and the complexity of the scene, reducing the noise figure by 30% in scenes with strong structural features such as tunnels. Based on the basic allocation strategy of 0.4 for the reprojection error constraint weight, 0.3 for the IMU motion constraint weight, and 0.3 for the laser odometry constraint weight, the association between factors and corresponding nodes is established. Each visual-laser feature pair corresponds to a reprojection error constraint factor, and the IMU and laser odometry constraint factors form a continuous motion trajectory constraint chain.

[0045] The model initialization adopts a laser odometry-led + IMU pre-integration correction strategy. The first frame pose output by the laser odometry in step 2.2 is used as the estimated value of the initial pose node. The stability is verified by combining the initial pose calculation result of IMU. If the pose deviation is ≤0.1 degrees, it is confirmed to be valid and the node is fixed as the global reference frame. The initial values ​​of subsequent pose nodes are predicted by the previous pose node and the IMU pre-integration data. The initial values ​​of map feature point nodes are directly adopted from the three-dimensional coordinates after the fusion of vision and laser in step 2.2. In the incremental optimization training phase, when new data arrives, optimization is performed only on the local window consisting of the newly added node and the three adjacent historical nodes. The Levenberg-Marquardt algorithm is used to minimize the local weighted residual sum of squares, and the sparse matrix COLAMD reordering technique is used to improve computational efficiency. The time taken for a single local optimization is controlled within 10 milliseconds to meet real-time requirements. When the number of newly added nodes reaches 20, or the change in the residual of the local optimization exceeds 0.01 for three consecutive times, global optimization is triggered. By relinearizing all constraint factors, updating the sparse matrix structure of the factor graph, and using the Cholesky decomposition algorithm to solve quickly, cumulative errors are eliminated, ensuring the stability of positioning accuracy in long-term operations.

[0046] The optimization process employs dual convergence criteria: first, the change in pose node estimates after two consecutive optimizations must be controlled within a position deviation of ≤0.01cm and an attitude deviation of ≤0.001 degrees; second, the sum of squared global weighted residuals must be less than 1e-6. Meeting either condition stops the current optimization iteration and outputs the optimal pose parameters. If convergence is not achieved after 30 iterations, the robust kernel function parameters are automatically adjusted, with the Huber kernel threshold adjusted from 1.345 to 2.0. Abnormal constraint factors exceeding three standard deviations are removed before re-optimization. The optimized pose data can be directly used for UAV obstacle avoidance path planning. In warehouse scenarios, it can accurately avoid shelves and moving personnel; in tunnel scenarios, it can maintain flight along the centerline. It also adapts to the positioning needs of different UAV operation modes, such as low-speed inspection and medium-speed navigation, providing reliable positioning support for various UAV operations.

[0047] Within a sliding window, joint nonlinear optimization is performed on the factor graph model to calculate the UAV pose in real time. Specifically, this includes setting a sliding window containing the latest 12 frames of data. During dynamic window maintenance, keyframe determination is based not only on the inter-frame feature point matching ratio but also on the semantic consistency and response quality of the feature points: frames with a high proportion of static semantic feature points and strong matching stability are prioritized; when the number of effectively matched feature points is less than 60% of the total feature points, the frame is automatically marked as a keyframe; for non-keyframes, only their core constraint relationships with adjacent keyframes are retained, while redundant features and constraint factors are eliminated. Through this selective retention strategy, the model effectively retains keyframes while maintaining the core semantic feature points of the UAV. While ensuring optimization accuracy, the computational complexity is effectively controlled to meet the computing power requirements of real-time positioning in confined spaces. Joint nonlinear optimization is performed on the constructed multi-sensor fusion factor graph model, with the optimization objective focusing on minimizing the weighted total error of all constraint factors. In the initialization stage, the IMU pre-integration data is no longer used alone. Instead, the pose results of the laser odometry in step 2.2 are fused with the IMU pre-integration information to obtain the initial estimates of the UAV pose parameters and map feature point coordinates within the window. The pose results of the laser odometry provide a global reference, and the IMU pre-integration data corrects short-term motion details, ensuring that the initial values ​​are closer to the real state, laying the foundation for subsequent optimization.

[0048] During iterative optimization, the current error term is first calculated for each constraint factor—reprojection error constraint factor, IMU motion constraint factor, and laser odometry constraint factor—and abnormal errors are suppressed to avoid large errors caused by dynamic target mismatch or sensor noise interfering with the overall optimization results. Based on the weight allocation of each constraint factor, the parameter adjustment step size is dynamically adjusted: constraints with higher weights are adjusted more cautiously, while constraints with lower weights are given greater adjustment flexibility. After each fine-tuning, the weighted total error is recalculated, and the changes in position and attitude parameters are monitored to ensure stable convergence of the optimization process. Convergence is judged using a dual standard: on the one hand, the weighted total error is monitored to see if it drops to a preset threshold; on the other hand, the error change over multiple consecutive iterations is tracked. When both requirements are met, the optimization is considered converged and the iteration stops. If convergence is not achieved after reaching the upper limit of iterations, the current optimal parameters are output and marked as needing optimization. This mark is also fed back to the sliding window management module, triggering the encrypted selection of keyframes and re-verification of constraint factors in the next round of windowing, improving the success rate of subsequent optimizations.

[0049] Through the above optimization process, high-precision pose data of the UAV in the global coordinate system of a confined space is calculated in real time, including three-dimensional position coordinates and attitude angles. The result fully integrates the complementary advantages of vision, laser and IMU, and can resist interference such as feature occlusion and lighting changes in a confined space. The position accuracy reaches the centimeter level and the attitude accuracy reaches the 0.1 degree level, providing a continuous and stable pose reference for subsequent incremental map construction and ensuring the consistency and accuracy of map construction.

[0050] Based on the real-time calculated UAV pose, the semantically enhanced point cloud data is aligned frame by frame and fused into the global map, and the global 3D environment map with semantic information is updated and output in an incremental manner.

[0051] In a preferred embodiment of the present invention, based on the real-time calculated UAV pose, semantically enhanced point cloud data is aligned frame by frame and fused into the global map, and a 3D environment map with semantic information is updated and output incrementally, including:

[0052] Based on the real-time calculated UAV pose, the semantically enhanced point cloud data of the current frame is transformed from the sensor coordinate system to the global coordinate system to obtain the current frame point cloud in the global coordinate system. Specifically, this includes: based on the calculated UAV 3D position coordinates and attitude angles, firstly, calling the spatiotemporally aligned and calibrated sensor extrinsic parameter data, including the rotation matrix and translation vector of the LiDAR and RGBD camera, and combining the online closed-loop verification results in a confined space GNSS denied environment, correcting the extrinsic parameter drift, and accurately transforming the current frame semantically enhanced point cloud from the sensor coordinate system of the 360-degree LiDAR or RGBD camera to the UAV body coordinate system, completing the rigid alignment of multi-sensor data, ensuring that the geometric information of the LiDAR and the color and texture information of the camera are unbiased; then, according to the UAV's roll angle, pitch angle, and yaw angle, constructing the rotation relationship from the body coordinate system to the global coordinate system in the confined space in the ZYX rotation order: first calibrating the heading around the global Z-axis through the yaw angle, then... The vertical tilt is corrected by pitch angle, and the horizontal tilt attitude is calibrated by roll angle. These three steps are connected to ensure that the attitude mapping fits the motion characteristics of the UAV in narrow spaces such as tunnels and mines. The translation transformation is based on the UAV's three-dimensional position coordinates, which are derived from the global reference results of SLAM autonomous localization. Vector superposition maps the origin of the body coordinate system to the corresponding position in the global coordinate system. For each semantically enhanced point cloud in the body coordinate system, rotation transformation is first performed to correct the spatial direction, and then translation transformation is superimposed to adjust the spatial position. The entire process assigns a unique semantic index to each point cloud and binds semantic tags such as walls, obstacles, and suspected trapped persons, as well as color and texture features, to the index, while simultaneously preserving multimodal perception information. After the transformation, the accuracy is checked by calculating the spatial deviation between the point cloud and the mapped points in adjacent frames to ensure that the single-point position error is adapted to the positioning requirements of rescue in confined spaces. Finally, the current frame point cloud with accurate geometric position and complete semantic information in the global coordinate system is obtained.

[0053] The process involves registering and fusing the current frame point cloud in the global coordinate system with the existing global 3D environment map. This updates the geometric structure and semantic information in the global 3D environment map, resulting in a fused and updated global map. Specifically, this includes: first, extracting highly recognizable common feature points such as wall corners and building edges from the current frame point cloud and the existing global map; then, comparing the spatial distribution and morphology of these feature points to find a large number of reliable corresponding point pairs, quickly reducing the spatial discrepancy between the two and completing the initial alignment; finally, repeatedly adjusting the position of the current frame point cloud to continuously reduce the distance between corresponding feature points until the deviation between two adjacent adjustments is no longer significant. To achieve precise alignment, during geometric fusion, point cloud data in overlapping areas are weighted according to the measurement reliability of the sensors, and the weighted average is used for integration. This preserves detailed features such as wall textures and device edges while eliminating random measurement noise. When updating semantic information, the credibility of semantic labels at the same spatial location is evaluated. Combining the actual effects of laser and visual perception, labels that better fit the real scene are retained. If a change in the position of a dynamic target is detected, the range of the changed area is determined by comparing the point cloud distribution of the previous and next frames. The semantic information and geometric coordinates of the area are updated in real time, ultimately resulting in a fused and updated global map with coherent geometric structure and consistent semantic information.

[0054] The merged and updated global map undergoes consistency maintenance and redundancy removal. Keyframe point cloud data is extracted and retained to optimize the map's storage structure and query efficiency, resulting in an optimized global map. Specifically, during consistency maintenance, other point clouds within a certain range around each point cloud are examined, their overall geometric shape is calculated, and isolated noise points that significantly deviate from the overall structure are removed, making the map's geometry more regular. Simultaneously, the consistency of each point cloud's semantic label with the labels of most surrounding points is verified, and conflicting abnormal semantic points are corrected to ensure consistent semantic expression within the same area. During redundancy removal, keyframes are divided into multiple groups based on their temporal order and spatial distribution. From each group, the main keyframes with rich features and clear environmental details are selected, while other point cloud data with redundant information and limited supplementary value to the map are removed. The retained main keyframe point clouds are then layered and organized according to spatial regions and timestamps to simplify the data storage structure and reduce redundancy. Furthermore, the consistency of distant regions in the global map is periodically verified. If spatial deviations are found between different regions, they are corrected by realigning the relevant point clouds to ensure the geometric accuracy and semantic consistency of the entire map, ultimately resulting in a structurally optimized and query-efficient global map.

[0055] Based on the optimized global map, a semantically enhanced 3D environment map is generated and output in real time for path planning and personnel identification. Specifically, this includes: structuring the optimized global map, classifying and integrating point clouds with the same semantic labels, identifying the 3D spatial range, distribution, and boundary positions of walls, ground, obstacles, and suspected human bodies, and establishing a precise correspondence between semantic labels and spatial regions; for path planning needs, defining passable areas based on ground semantic labels, and combining the UAV's motion characteristics to delineate suitable flight navigation areas, clearly identifying impassable areas occupied by obstacles and areas of dynamic personnel activity, providing clear environmental constraints for path planning; for personnel identification needs, extracting point cloud clusters corresponding to suspected human body semantic labels, further filtering them based on spatial features such as cluster height and volume to eliminate falsely identified static objects, and marking the target's real-time 3D position and movement trend; and outputting this structured, semantically enhanced 3D environment map in real time according to the UAV's pose update frequency, providing accurate environmental references for subsequent path planning, indicating a clear target search range for the personnel identification module, and supporting the UAV in efficiently completing its tasks.

[0056] In a preferred embodiment of the present invention, based on the real-time calculated UAV pose and the constructed and updated 3D environment map, combined with real-time acquired semantically enhanced point cloud data, obstacles in the environment are identified and passable areas are analyzed, thereby online planning and dynamic adjustment of the UAV's collision-free flight path, including:

[0057] Based on real-time acquired semantically enhanced point cloud data of the current frame, obstacle point cloud clusters are identified. By comparing and incrementally updating the identification results with the global 3D environment map, a navigation map containing real-time obstacle information is obtained. Specifically, the navigation map is a local drivability map updated in real-time based on the global 3D environment map. The global 3D environment map contains the complete geometric structure and semantic information of the environment. The navigation map further integrates real-time detected obstacle information and drivability analysis results on the basis of the global 3D environment map to form an environmental representation specifically for UAV path planning. Based on the real-time acquired semantically enhanced point cloud data of the current frame, point cloud data with semantic labels of walls and obstacles are first accurately filtered out, and dynamic target point clouds with semantic labels of suspected human bodies are excluded to avoid misjudging trapped personnel as obstacles hindering rescue. By observing the spatial aggregation state of the point clouds... The study focuses on analyzing the extent, density, and overall volume of continuously distributed point clouds in 3D space. Isolated, low-density noise points are eliminated, and point clouds meeting the defined volume criteria are categorized into complete obstacle point cloud clusters. The 3D boundaries and spatial occupancy of each obstacle are clearly defined. Subsequently, this identified obstacle information is meticulously compared region-by-region and feature-by-feature with the constructed global 3D environment map. Newly added obstacles not recorded in the map, as well as dynamic obstacles whose location or shape has changed, are accurately marked. Simultaneously, static obstacle information that persists and remains unchanged in the map is retained. Newly identified obstacle data is systematically integrated into the global map through incremental updates, while records of obstacles that have disappeared from the map are removed. The final result is a navigation map containing real-time, complete, and accurate obstacle information, providing reliable environmental data support for subsequent path planning.

[0058] Based on the geometric structure, semantic information, and real-time calculated UAV pose in the navigation map, a 3D space accessibility analysis is performed to generate a local cost map centered on the UAV's current location, quantifying the access costs of different areas. Specifically, this includes: first, retrieving complete geometric structure information from the navigation map to determine the 3D spatial location, distribution spacing, and morphological characteristics of walls and obstacles; simultaneously, combining semantic tags to accurately distinguish static obstacles, dynamic targets such as suspected human figures, and potentially dangerous areas such as loosely structured collapsed areas and narrow passages; and simultaneously acquiring real-time calculated physical characteristic data such as the UAV's 3D position, attitude angles, fuselage dimensions, and minimum turning radius to comprehensively grasp the core information of the environment and the UAV; based on the characteristics of rescue scenarios with narrow and complex enclosed spaces, and combined with the UAV's sensor detection range and short-distance flight planning requirements, an appropriate local analysis range is set centered on the UAV's current location to ensure coverage of key areas of the subsequent flight path without increasing unnecessary computation.

[0059] A detailed accessibility analysis was conducted on the three-dimensional space within this local area: areas far from various obstacles, with open and unobstructed spaces and no potential risks were classified as low access cost, suitable for rapid drone reconnaissance; areas close to static obstacles but with a safety clearance of no less than 1.2 times the drone's fuselage size (to allow for drone sway redundancy) and no collision risk were classified as medium access cost, requiring careful passage with controlled flight speed; narrow passages within a certain range around dynamic targets (to avoid disturbing trapped personnel), narrow passages with a width less than the drone's minimum turning radius, and potentially dangerous areas with loose structures were classified as high access cost, and should be avoided in principle. Subsequently, the low, medium, and high access costs were quantified into reasonable numerical ranges, and values ​​were assigned to each subdivided location in the three-dimensional space to ensure that the cost values ​​accurately reflect the access difficulty and safety risks. Finally, a local cost map centered on the drone's current location, with clear cost divisions and tailored to the needs of the rescue scenario, was generated, providing a clear and reliable environmental constraint basis for subsequent path planning.

[0060] Based on the local cost map, the UAV's pose, and the next-stage exploration target point issued by the command platform, a collision-free flight path from the current location to the local target point is planned online. This includes: determining the passage cost values ​​for each sub-region in the local cost map, the UAV's current 3D pose, and the precise coordinates of the next-stage exploration target point issued by the command platform; integrating this key information as the core input for path planning; and considering the operational requirements of disaster relief in confined spaces, fully taking into account the UAV's kinematic characteristics, specifically including actual maneuverability limitations such as maximum flight speed, minimum turning radius, and climb and descent rates. This ensures that the planned path is within the drone's operational range. When searching for a path online, the drone starts from a low-cost area corresponding to its current location and extends towards the target point along a reasonable trend of gradually increasing costs, avoiding high-cost areas throughout. At the same time, it takes into account the smoothness of the path and the shortest distance. By setting the spacing between path nodes to match the drone's flight control precision, continuous path nodes are connected in an orderly manner to form a complete trajectory. Each node corresponds to a specific coordinate in three-dimensional space. Finally, a collision-free flight path from the current location to a local target point that meets the drone's maneuverability is planned, adapting to the complex environment of a confined space.

[0061] The system controls the drone to fly along the collision-free flight path, while continuously monitoring environmental changes based on real-time updates of the navigation map. If a new obstacle is detected that renders the original path impassable, path replanning is immediately triggered. A local cost map is regenerated using the latest environmental information, and a new safe path is searched. Specifically, this includes: using the drone's flight control terminal to output precise attitude control and power output commands in real-time based on the planned collision-free flight path, driving the drone to maintain a stable flight attitude and advance along the preset path, while continuously receiving real-time updates of the navigation map, focusing on monitoring environmental changes within a certain range ahead and around the path; by comparing the newly updated obstacle information with the three-dimensional spatial relationship of the original path in real-time, it is determined whether a new obstacle has encroached on the safe zone of the original path. The safe zone is set at 1.5 times the drone's fuselage size, reserving sufficient obstacle avoidance redundancy; if a new obstacle is detected encroaching on the safe zone and rendering the original path impassable, the path replanning mechanism is immediately triggered. The drone pauses its original flight path and maintains its current stable attitude to avoid loss of control due to unforeseen circumstances. Based on the latest navigation map, a new local accessibility analysis is conducted, and the quantification values ​​of access costs in each area are precisely adjusted to generate an updated local cost map. When planning a new path, the drone's kinematic boundaries are first defined, including actual maneuverability limitations such as maximum flight speed, minimum turning radius, and climb or descent rate, to ensure that the path matches the drone's execution capabilities. During the search process, the spacing, turning angle, and altitude changes of each path node are all controlled within the above constraints: the turning angle does not exceed the drone's maximum turning threshold, the altitude change does not exceed the upper limit of the climb or descent rate, and the node spacing is adapted to the flight control precision of a confined space environment. Then, following the path search logic in step 3.3, a safe path is finally formed that avoids new obstacles and allows the drone to execute stably and smoothly, ensuring that the drone can continuously advance towards the target point in a dynamically changing environment without any collisions.

[0062] In a preferred embodiment of the present invention, based on semantically enhanced point cloud data and a constructed 3D environment map, multi-source detection and spatiotemporal correlation are performed to identify trapped personnel and determine their precise location in the 3D environment map. Further analysis of the trapped personnel's posture and vital signs information includes:

[0063] Based on semantically enhanced point cloud data, candidate point cloud clusters with human morphological features are identified through scanning using preset point cloud geometric and contour feature templates. Simultaneously, based on the corresponding visual image data, a lightweight convolutional neural network is used for human target detection, generating human candidate bounding boxes in the 2D image. Specifically, this involves: first, accurately filtering point cloud clusters with semantically labeled as potentially human from the generated semantically enhanced point cloud data, while removing ground points below the minimum human height and structural points above the maximum human height; then, further filtering based on a threshold for the number of neighboring point clouds, where the threshold is set to a certain radius around each point. The system minimizes the number of adjacent points to eliminate scattered, isolated noise points and initially focuses on point cloud data that conforms to the spatial distribution characteristics of the human body. Combining common human postures in disaster relief in confined spaces, such as lying flat, curled up, lying on one's side, and leaning against something, a multi-scale, part-specific point cloud geometric and contour feature template library is constructed. This template library not only includes the standard range of overall dimensions, such as height, shoulder width, and body thickness, but also subdivides into local feature templates, such as the head (approximately spherical contour with concentrated point cloud density), the torso (approximately elliptical cylindrical structure with smooth contour), and the limbs (long strips extending with gradually tapering ends). Each posture corresponds to a complete set of overall and local combined templates.

[0064] The initial screening of point cloud clusters is subjected to refined scanning and matching on a region-by-region and part-by-part basis: First, the size of the point cloud cluster is quickly determined to be within a reasonable range for the human body using the overall template. Then, the point cloud cluster is divided into approximate sections for the head, torso, and limbs, and compared with the local features of the corresponding parts in the template library. Shape similarity (i.e., contour overlap), point cloud density similarity (i.e., local point cloud distribution compactness), and structural continuity similarity (i.e., point cloud transition smoothness at limb connection points) are calculated. A weighted comprehensive similarity score is obtained. False detection point cloud clusters with comprehensive scores below the threshold, those that fail to match local features (e.g., no concentrated head point cloud clusters, broken torso contours), or those that conform to the rigid structural characteristics of the collapsed object are removed. Candidate point cloud clusters whose shapes fit the overall local features of the human body are retained.

[0065] Meanwhile, to address the issues of insufficient lighting in enclosed spaces, potential smoke or dust obstruction, and significant local shadows, the preprocessed color image undergoes multi-stage refinement enhancement: First, Gaussian filtering is used to remove image noise under low-light conditions. Then, regional histogram equalization is applied, dividing the image into independent 16×16 pixel windows. Grayscale adjustment is performed only on non-edge windows to avoid artifacts at window boundaries and resolve local overexposure or underexposure issues caused by global equalization. Brightness compensation is performed separately for shadow areas, dynamically adjusting the brightness of shadow areas based on the grayscale mean of neighboring non-shadow areas. For areas obscured by smoke or dust, color channel separation enhancement is used to strengthen the grayscale difference between human skin color and the environment in the RGB channels, while weakening the obstruction. The blurring effect is ultimately improved to enhance the distinction between human body contours and environmental background. Subsequently, a lightweight convolutional neural network optimized by channel pruning, INT8 quantization, and layer fusion is used. This network is adapted to airborne low-computing-power embedded ARM chips or FPGA platforms to ensure real-time performance and perform frame-by-frame detection. The network is pre-trained on a dedicated dataset covering enclosed rescue scenarios such as tunnels, mines, and warehouses. The dataset not only includes human body samples in different postures (lying flat, curled up, lying on their side) and different lighting conditions (dim, backlight, and local strong light), but also adds interference samples such as smoke, dust, collapsed debris, and rescue tools. Data augmentation techniques such as rotation, scaling, partial occlusion, and brightness distortion are used to expand the sample diversity and improve the model's anti-interference ability.

[0066] During the detection process, the network first extracts multi-scale features from the image, adapts to human body sizes at different distances, and generates initial human candidate boxes and confidence scores. Then, a non-maximum suppression algorithm with a dynamic IOU threshold is used. The IOU threshold is lowered to 0.3 in dense scenes and raised to 0.5 in sparse scenes to balance missed detections and duplicate detections, eliminating duplicate boxes with high overlap. At the same time, the network dynamically calculates the reasonable pixel range of the human body in the image by combining the real-time flight altitude of the drone, the focal length of the camera, and the pixel size of the sensor. Following the rule that the human body pixel size decreases with increasing height, the network replaces the fixed threshold and eliminates abnormal boxes with pixel areas exceeding this range. The network also uses aspect ratio verification to eliminate distorted boxes with aspect ratios exceeding the normal range of the human body, such as excessively narrow or long projection boxes of collapsed objects. This further filters the data and ensures that each candidate box can accurately select potential human targets. The network also maintains timestamp synchronization with the candidate point cloud clusters on the point cloud side, with the timestamp synchronization error controlled within milliseconds, laying a reliable foundation for subsequent spatiotemporal correlation matching.

[0067] The human candidate bounding box information in the 2D image is combined with the corresponding depth information and projected into 3D space to generate a corresponding 3D visual detection box. The corresponding 3D visual detection box is spatiotemporally correlated and matched with the candidate point cloud cluster. For the successfully matched candidate targets, weighted fusion calculation is used to accurately calculate the 3D coordinates of the trapped person in the global 3D environment map. Specifically, this includes: extracting pixel-by-pixel depth information of the corresponding region of each 2D human candidate bounding box from the depth image acquired by the RGBD camera; firstly, removing abnormal depth values ​​caused by smoke, dust, or sensor noise by using neighborhood mean filtering, retaining continuous and stable effective depth data to ensure the reliability of the depth information; combining the completed camera intrinsic parameter calibration data, including focal length, principal point coordinates, distortion coefficients, etc., and the latest pose of the UAV after real-time calculation and sliding window optimization, binding the coordinates of the four vertices of the 2D candidate box with the effective depth values ​​of the corresponding positions, and completing the mapping from the image coordinate system to the camera coordinate system and then to the global 3D environment coordinate system through rigid coordinate transformation to generate a 3D visual detection box containing the spatial position range, height, width, and thickness information of the human target, ensuring that the spatial position of the detection box accurately corresponds to the real target.

[0068] Then, a rigorous spatiotemporal correlation verification is performed. First, the sensor synchronous trigger signal confirms that the 3D vision detection box and the candidate point cloud cluster come from the same frame of data acquisition. The timestamp alignment error is controlled within the millisecond level to avoid misalignment caused by cross-frame matching. Then, the spatial overlap volume ratio, the Euclidean distance between the center points and the coverage ratio of the point cloud cluster in the detection box are calculated. When the overlap volume ratio is not lower than the preset standard, the center point distance is within the reasonable size range of the human body, and the coverage ratio of the point cloud in the detection box reaches the threshold, it is judged as a successful match.

[0069] For successfully matched candidate targets, the geometric centroid coordinates of the candidate point cloud clusters are first calculated. The mean of all valid point cloud coordinates within the cluster is then used to solve the problem, and the center coordinates of the 3D visual detection box are determined simultaneously. The weight allocation adopts a dynamic adjustment mechanism. The point cloud density is quantified according to the number of points per unit volume. The closer the target is to the drone and the higher the point cloud density, the greater the weight. The clarity of the visual image texture is evaluated by the mean of the edge gradient of the candidate box region. The clearer the texture, the higher the weight of the center coordinates of the visual detection box. Finally, the accurate 3D coordinates of the trapped personnel in the global 3D environment map are obtained by weighted summation. Cases with low overlap, excessive distance between center points, insufficient point cloud coverage, or multiple candidate boxes repeatedly matching the same target are directly judged as false detections and eliminated to ensure that the final output of the trapped personnel's 3D coordinates has both accuracy and uniqueness.

[0070] For successfully matched candidate targets, their corresponding color image regions and local point cloud data are extracted and input into a pre-trained pose recognition neural network model to analyze the key skeletal points and pose of the human body. At the same time, combined with infrared thermal imaging data, the static or dynamic attributes and vital sign activity of the target are comprehensively evaluated to generate structured personnel information containing position, pose, and state confidence. Specifically, the spatial range of the three-dimensional visual detection box corresponding to the successfully matched candidate target is appropriately expanded by 10% to 15% to extract the corresponding color image region and local point cloud data. The expanded range ensures that the key parts of the human target, such as the head, torso, and limbs, are fully included, while retaining a small amount of environmental reference information around the limbs to avoid feature omissions caused by the limb ends exceeding the detection box or being partially obscured by collapsed objects. After cropping, the color image is subjected to brightness normalization and noise filtering in sequence. The kernel size of the noise filter is dynamically adjusted according to the detail density of the rescue scene image. Local point cloud data is processed by radius neighborhood search to remove discrete noise points at the edges, and then voxel downsampling is performed. The voxel size is adapted to the onboard computing power to balance detail preservation and computational efficiency, and finally the core geometric structure and surface texture information are preserved.

[0071] The model architecture is based on a lightweight MobileNetV3 backbone network, removing 30% of the top-level redundant convolutional layers and replacing traditional convolutions with depthwise separable convolutions. Combined with channel shuffle and attention mechanisms (SE modules), the number of parameters is reduced to less than 5 million, making it compatible with low-computing-power embedded ARM chips or FPGA platforms, ensuring single-frame inference time is controlled within 20 milliseconds. A visual-point cloud dual-branch feature extraction structure is designed: the image branch extracts multi-scale visual features through a feature pyramid network (FPN). The bottom-level features focus on details such as limb edges and skin texture, while the top-level features capture the overall human contour, emphasizing the feature responses of key areas in rescue scenarios such as the head, shoulders, and hips. The point cloud branch first converts local point clouds into fixed-size voxel meshes and extracts 3D geometric features through a simplified PointNet structure, focusing on capturing the elliptical cylindrical shape of the human torso, the elongated extension trend of the limbs, and the angular changes at bone joints. The features from both branches are then stitched and fused before being input into a lightweight self-attention module to further suppress the feature weights of background interference such as collapsed objects, rescue ropes, and safety helmets.

[0072] The training dataset was specifically designed for confined space rescue, covering four typical rescue scenarios: tunnel collapse, mine flooding, warehouse fire, and underground parking lot collapse. It includes samples of trapped individuals aged 20 to 60, primarily adults with a small number of teenagers, dressed in various clothing, including everyday clothes, work clothes, and rescue suits. The poses cover eight core poses frequently seen at rescue sites, such as lying flat, curled up, lying on one's side, leaning against something, and limbs partially buried and obscured by rubble or wooden planks (10% to 40%). It also includes twelve common interfering objects such as smoke, dust, rubble piles, rescue tools, and discarded furniture. The total number of samples is 80,000 images and corresponding point cloud data. Each frame of data is manually labeled with 18 key skeletal points: 2 for the head, 4 for the torso, and 12 for the limbs, along with the pose category, occlusion level, and coordinates of the core human body region.

[0073] The data augmentation phase fully simulates the harsh conditions of the rescue environment: brightness distortion is performed on the images to simulate a dim environment with a 30% to 50% reduction in grayscale; local overexposure in backlit scenes; local occlusion is randomly simulated with 10% to 30% of limbs being obscured by debris or smoke; smoke overlay is added with a three-level fogging effect according to the smoke concentration at the rescue site; color shift is performed to simulate red light at a fire scene; and blue light interference is processed in the dim environment of a mine. The point cloud data is rotated by ±15 degrees in three dimensions to simulate changes in the drone's shooting angle; scaling is adjusted by ±20% to adapt to different flight distances; a small number of missing point clouds are randomly removed with 5% to 15% of the edge point clouds; sensor distance limitations or occlusion are simulated; Gaussian noise is added to the noise overlay; and sensor interference is simulated in confined spaces, further expanding the sample diversity and scene adaptability.

[0074] The training process is executed in three progressive stages: The first stage is based on the publicly available human pose datasets COCO and MPII for basic pre-training, building general human feature extraction capabilities, and learning the association rules of skeletal points and basic pose classification logic; The second stage imports a rescue-specific dataset for scene adaptation and fine-tuning, using the heatmap loss function (MSE loss) to optimize the accuracy of key point localization, and combining it with the limb connection loss function (L1 loss) to ensure the rationality of skeletal point connections. To address the problem of imbalanced pose samples, such as the low proportion of curled-up pose samples, a class balancing sampling strategy is adopted to improve the recognition ability of niche poses; The third stage is robust reinforcement training, mixing in noisy data collected from real rescue drills. The learning rate is dynamically decayed, with the initial learning rate set to 0.001 and decreasing to 0.9 every ten training cycles. The gradient pruning gradient threshold is set to 1.0 to avoid gradient explosion. An early stopping strategy is used, stopping the model if the loss on the validation set does not decrease after five consecutive training cycles to avoid model overfitting. The final trained model achieves a pose classification accuracy of over 92% on the rescue scene test set, with a key skeletal point localization error of less than 8 pixels and a false detection rate of less than 5% in occluded scenes.

[0075] The processed color image region and local point cloud data are input into the model. Visual texture and three-dimensional geometric features are captured separately through bi-branch feature extraction. After fusion and attention enhancement, the three-dimensional coordinates and connection relationships of 18 key skeletal points such as head, torso, and limbs are accurately identified. Then, the specific posture type of the trapped person is analyzed, such as curled up in the rubble after the tunnel collapse, lying on their side in the mine under the cover of the support, or lying flat in the smoke of the warehouse fire. The occlusion level and the confidence of the key skeletal points are marked.

[0076] Simultaneously, data from the corresponding area collected by the airborne infrared thermal imager is retrieved. The normal human body temperature range (36℃ to 37.5℃) is retained through temperature threshold filtering. Combining five to ten frames of data, a temperature fluctuation of less than 0.5℃ is considered stable, thus determining whether the target possesses human temperature characteristics. Furthermore, by analyzing minute positional changes in local point clouds across consecutive frames—such as fluctuations in chest point clouds due to breathing or displacements caused by slight limb movements—and combining this with posture recognition results (e.g., whether there are active movements like raising a hand or turning a head), the static or dynamic attributes of the target are determined. For suspected injury scenarios, skeletal point coordinate deviation analysis—such as abnormal angles connecting limb skeletal points or deviations of trunk skeletal points exceeding the normal range—preliminary assessments are made regarding the presence of injury characteristics such as limb twisting.

[0077] Finally, the system integrates the precise 3D coordinates of the trapped personnel, their specific posture types including occlusion level, static or dynamic attributes, and vital signs activity levels. High activity is defined as stable temperature with dynamic characteristics, medium activity as stable temperature alone, and low activity as unstable temperature with no dynamic characteristics. Confidence is calculated across multiple dimensions: posture recognition confidence is based on a weighted average of skeletal point matching and occlusion level; vital sign confidence is calculated by combining infrared temperature stability and inter-frame dynamic change amplitude; and point cloud-visual data consistency confidence is based on similarity determination through bi-branch feature fusion. This process ultimately generates structured personnel information including location, posture, state, and multi-dimensional confidence. Low-confidence results are marked with suspicious causes such as severe smoke obstruction, insufficient point cloud density, and weak infrared signals. This provides commanders with detailed and reliable reference data for prioritizing rescue efforts and developing targeted rescue plans.

[0078] In a preferred embodiment of the present invention, the constructed 3D environment map, the planned collision-free flight path of the UAV, and the precise location and vital signs information of the trapped personnel are transmitted back to the command platform via an adaptive anti-interference communication network. Multi-source data fusion is then performed to generate a global rescue situation, based on which rescue decision support information is generated, including:

[0079] The global 3D environment map, the collision-free flight path, and structured personnel information are encapsulated according to preset data priorities. The data compression rate and transmission frequency are adaptively selected based on the real-time communication link quality. The data is then transmitted to the command platform via a self-organizing network communication link based on a dual-band Mesh protocol. This adaptive anti-interference communication network includes a dual-band Mesh self-organizing network link. Through dynamic channel selection, link quality assessment, and adaptive coding and modulation technology, it automatically switches between the 2.4GHz and 5.8GHz dual-band frequencies according to interference levels and supports multi-hop relay transmission to address multipath fading and signal obstruction issues in confined spaces. Specifically, this involves: first determining preset data priority rules; prioritizing structured personnel information due to its direct impact on rescue decision-making; allocating collision-free flight path information as the core basis for continuous UAV operation as a medium priority; and prioritizing the large volume of global 3D environment map data. The three types of data are then structured and encapsulated according to these priorities, with each data packet containing a timestamp, UAV number, and data verification information to ensure traceability and integrity during data transmission.

[0080] The system monitors the status of the self-organizing network communication links based on the dual-band Mesh protocol in real time. Link quality is assessed using three core indicators: signal strength, packet loss rate, and transmission delay, and categorized into three levels: good, average, and poor. Based on link quality, the system adaptively adjusts data compression strategies and transmission frequencies: When the link is good, a low compression ratio is used to preserve map texture details and personnel information accuracy, with the highest transmission frequency for structured personnel information and environmental maps transmitted at a normal frequency; when the link is average, the compression ratio is moderately increased, and the transmission frequency of environmental maps is reduced, while maintaining high-frequency transmission of personnel information and flight paths; when the link is poor, a high compression ratio is used to compress environmental map and flight path data, ensuring only the minimum transmission requirements for structured personnel information to reduce bandwidth consumption. During communication, the dual-band Mesh network automatically switches to the optimal link. The high-frequency band is used to transmit low-latency personnel information and flight path data, while the low-frequency band is used to transmit environmental map data with stronger anti-interference capabilities. Multi-hop transmission mode is supported; when a single link is blocked, it automatically relays data through other UAV nodes, ensuring stable data transmission to the command platform in complex environments such as confined spaces.

[0081] The command platform receives and decodes data packets transmitted from at least one drone, and simultaneously registers and merges the received local environment map information and personnel information into a unified global 3D situation map of the rescue scenario. Specifically, this includes: activating a multi-channel monitoring mechanism through the command platform to continuously receive data packets transmitted from one or more drones via a dual-band mesh self-organizing network, while recording the device number and data transmission timestamp of each drone; decoding the received data packets and verifying data integrity using CRC checks; if packet loss, verification failure, or data incompleteness is detected, immediately sending a retransmission request to the corresponding drone via the communication link, explicitly specifying the missing data packet number to ensure no critical data is missed; extracting the drone's real-time pose information, spatial coordinate parameters of the local environment map, and data acquisition time from each valid data packet, based on a unified global coordinate system. The system uses the drone's pose as a reference to spatially register local environmental maps, accurately stitching together local maps collected by different drones according to their actual spatial locations. For structured personnel information, it compares data on the same target collected by different drones at different timestamps, and performs deduplication through location coordinate similarity and confidence scores, retaining the personnel status information with the highest confidence and most complete data. For the geometric structure and semantic information of overlapping map areas, a weighted fusion method is used to correct data deviations, with weights dynamically allocated based on the drone's positioning accuracy and sensor detection distance. Finally, all the registered and fused global 3D environmental maps, the deduplicated and optimized structured personnel information, and the flight path information of each drone are integrated into the same visualization framework to generate a global 3D situation map of the rescue scenario with unified spatial coordinates and complete information, providing a reliable data foundation for subsequent intelligent analysis and decision-making.

[0082] Based on a global 3D situational map of the rescue scenario, the system analyzes both detected and undetected areas to assess environmental structural risks and mark the location and status of trapped personnel. Based on this, it generates decision support information including rescue channel suggestions, reconnaissance route planning suggestions, and personnel rescue priority plans.

[0083] In a preferred embodiment of the present invention, based on a global three-dimensional situational map of the rescue scenario, the environmental structural risks are assessed and the location and status information of trapped personnel are marked by analyzing the detected and undetected areas. Based on this, decision support information is generated, including rescue channel suggestions, reconnaissance route planning suggestions, and personnel rescue priority plans, comprising:

[0084] The overall 3D situational awareness map of the rescue scenario is analyzed to identify areas that have been explored by drones and areas that have not yet been explored. Specifically, this involves: retrieving flight trajectory data, sensor detection range parameters, and point cloud data density information of all drones in the overall 3D situational awareness map; combining the real-time pose of each drone and the sensor field of view to back-calculate the detection coverage of each area frame by frame; areas with point cloud data density reaching a preset standard and with complete visual image matching are identified as explored areas. These areas must clearly present the environmental geometry and semantic information, without obvious data loss or breaks; areas with insufficient point cloud density, missing visual images, or only partial coverage and incomplete key information are marked as unexplored areas; at the same time, data consistency verification is performed on overlapping areas explored by multiple drones, retaining the most complete detection results; finally, the boundaries between explored and unexplored areas are clearly delineated in the overall situational awareness map using different colors or textures to confirm the spatial distribution and size of unexplored areas.

[0085] Based on the geometric and semantic information in the global 3D situation map of the rescue scenario, the environmental structural stability and potential risk level of the detected areas are assessed, generating environmental structural risk assessment results. Simultaneously, the location information and corresponding status information of identified trapped personnel are marked in the global 3D situation map. Specifically, based on the geometric structural information in the global 3D situation map, the tilt angle, integrity, and connection stability of structures such as walls, columns, and the ground are analyzed. Semantic tags are used to identify risk sources such as collapsed debris, loose rubble, and fractured structures. Based on characteristics such as tilt angles exceeding safety thresholds, large structural fracture gaps, and loose point cloud distribution, the environmental structural risk is divided into three levels: high, medium, and low. High-risk areas refer to areas where secondary risks are possible at any time. Areas that are collapsed and seriously threaten the safety of rescuers, such as walls with an angle of inclination exceeding 30 degrees or unsupported loose rubble; medium-risk areas refer to areas with structural hazards but short-term stability, such as slightly tilted components or a small number of scattered fragments; low-risk areas refer to areas with intact structures and strong stability, such as intact walls and flat ground. Detailed environmental structural risk assessment results are generated and associated with specific area coordinates. Simultaneously, the three-dimensional coordinates in the structured personnel information are accurately mapped to the corresponding positions on the global situation map. Different icons are used to distinguish the posture types of trapped personnel, such as curled up, lying flat, and lying on their side. Color depth is used to indicate the activity level of vital signs, with darker colors indicating high activity and lighter colors indicating low activity, along with confidence level indicators, ensuring that personnel location and status information are intuitively identifiable on the situation map.

[0086] Based on the environmental structural risk assessment results, the location and status information of identified trapped personnel, and the spatial distribution of unexplored areas, a set of feasible rescue route suggestions, a set of supplementary reconnaissance route suggestions for unexplored areas, and a personnel rescue priority sequence based on status information are generated. Specifically, this includes: combining the environmental structural risk assessment results, prioritizing low-risk areas as the core routes for rescue routes, avoiding high-risk areas, while ensuring that the route width is not less than the minimum size required for the passage of rescue equipment and personnel, and that there are no sharp obstacles or narrow bottlenecks in the route; generating multiple feasible rescue route suggestions, each of which must clearly define the starting point, ending point, risk level of the areas traversed, and the advantages of passage; For areas yet to be explored, supplementary reconnaissance routes are planned based on the principles of prioritizing proximity to already identified trapped personnel, proximity to high-risk areas, and areas with concentrated unexplored areas. These routes must connect to the safety boundaries of already explored areas, avoid high-risk areas, maximize coverage of unexplored areas, and reduce redundant drone flights, forming a set of suggested supplementary reconnaissance routes. For trapped personnel already marked, they are comprehensively prioritized in the following order: activity level of vital signs (high activity prioritized), risk level of the area (personnel in high-risk areas prioritized), suspected degree of injury in posture (curled-up, limb contortion, etc. prioritized), and confidence level (high confidence prioritized), forming a personnel rescue priority sequence. Each sequence node includes the personnel's three-dimensional coordinates, a brief description of their status, and the basis for the ranking.

[0087] This system integrates a set of feasible rescue route suggestions, a set of supplementary reconnaissance route suggestions for unexplored areas, and a personnel rescue priority sequence based on status information. It then generates decision support information to output to command personnel. Specifically, this includes: integrating the sets of feasible rescue route suggestions, supplementary reconnaissance route suggestions, and personnel rescue priority sequences; marking each rescue route on a global 3D situation map with lines of different colors; marking supplementary reconnaissance routes with arrowed trajectory lines; and using numbers or color shades to correspond to personnel rescue priorities. Simultaneously, it generates accompanying textual descriptions detailing the access conditions, potential risks, and response suggestions for each rescue route; the detection focus, estimated time, and equipment compatibility requirements for each supplementary reconnaissance route; and the status details, rescue difficulty, and reasons for prioritizing each priority personnel. The annotated 3D situation map and textual descriptions are then simultaneously output to command personnel, providing decision support information that combines visualization and text. This allows command personnel to intuitively view the overall situation and quickly obtain key details, assisting them in developing efficient and safe rescue implementation plans.

[0088] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An autonomous detection unmanned aerial vehicle (UAV) control system for disaster relief in confined spaces, characterized in that: include: The multimodal environment perception module is used to simultaneously collect laser point cloud data and visual image data in a confined space, and to perform spatiotemporal alignment and semantic fusion processing on the laser point cloud data and visual image data to generate semantically enhanced point cloud data. The tightly coupled SLAM localization and mapping module is used to calculate the pose of the UAV in a confined space in real time based on semantically enhanced point cloud data, by tightly coupling and fusing visual features and laser features with inertial measurement data, and synchronously incrementally construct a 3D environment map with semantic information. The obstacle avoidance and path planning module is used to identify obstacles in the environment and analyze passable areas based on the real-time calculated UAV pose and the constructed and updated 3D environment map, combined with real-time acquired semantically enhanced point cloud data, and then plan and dynamically adjust the collision-free flight path of the UAV online. The trapped personnel identification and analysis module is used to identify trapped personnel and determine their precise location in the 3D environment map by performing multi-source detection and spatiotemporal correlation based on semantically enhanced point cloud data and constructed 3D environment map, and further analyze the trapped personnel's posture and vital signs information. The data feedback and intelligent decision-making module is used to transmit the constructed 3D environment map, the planned collision-free flight path of the UAV, and the precise location and vital signs of the trapped personnel back to the command platform through an adaptive anti-interference communication network. This allows for the fusion of multi-source data to generate a global rescue situation and, based on this, generate rescue decision support information.

2. The autonomous detection unmanned aerial vehicle control system for disaster relief in confined spaces according to claim 1, characterized in that, Simultaneously acquire laser point cloud data and visual image data from a confined space, and perform spatiotemporal alignment and semantic fusion processing on the laser point cloud data and visual image data to generate semantically enhanced point cloud data, including: Control the lidar and vision sensors carried by the drone to simultaneously collect raw lidar point cloud data and raw visual image data in a confined space; The acquired raw laser point cloud data and raw visual image data are time-stamped and spatial coordinate system corrected. At the same time, the raw visual image data is distortion-corrected to obtain time- and space-aligned laser point cloud data and visual image data. The pixel semantic information in the obtained temporally and spatially aligned visual image data is registered and fused with the temporally and spatially aligned laser point cloud data at the point cloud level, thereby assigning corresponding color and texture semantic features to the points in the laser point cloud data and obtaining semantically colored intermediate point cloud data. Based on the intermediate point cloud data with semantic coloring, an enhanced point cloud that integrates visual semantic information is output as semantically enhanced point cloud data.

3. The autonomous detection unmanned aerial vehicle (UAV) control system for disaster relief in confined spaces according to claim 2, characterized in that, Based on semantically enhanced point cloud data, and through the tight coupling and fusion of visual and laser features with inertial measurement data, the pose of a UAV in a confined space is calculated in real time, and a 3D environment map with semantic information is incrementally constructed simultaneously, including: Geometric feature points and edge features of laser point clouds are extracted from semantically enhanced point cloud data; simultaneously, visual feature points and their descriptors are extracted from the corresponding visual image data. The extracted visual feature points and descriptors are projected onto the corresponding laser point cloud frames and associated and matched with geometric feature points and edge features to form a feature set that is tightly coupled between vision and laser. Based on the feature set that is tightly coupled between vision and laser, the laser odometry constraint between adjacent frames is calculated. By fusing IMU pre-integrated data provided by the inertial measurement unit, the feature set tightly coupled with vision and laser, the IMU pre-integrated data, and the laser odometry constraints are incorporated into a factor graph model for pose estimation. Within a sliding window, the factor graph model is subjected to joint nonlinear optimization to calculate the pose of the UAV in real time. Based on the real-time calculated UAV pose, the semantically enhanced point cloud data is aligned frame by frame and fused into the global map, and the global 3D environment map with semantic information is updated and output in an incremental manner.

4. The autonomous detection unmanned aerial vehicle control system for disaster relief in confined spaces according to claim 3, characterized in that, Based on the real-time calculated UAV pose, the semantically enhanced point cloud data is aligned frame by frame and fused into the global map. The resulting 3D environment map with semantic information is updated incrementally and output, including: Based on the real-time calculated UAV pose, the semantically enhanced point cloud data of the current frame is transformed from the sensor coordinate system to the global coordinate system to obtain the point cloud of the current frame in the global coordinate system. The current frame point cloud in the global coordinate system is registered and fused with the constructed global 3D environment map. Based on this, the geometric structure and semantic information in the global 3D environment map are updated to obtain the fused and updated global map. The global map that has been merged and updated is subjected to consistency maintenance and redundancy removal. Keyframe point cloud data is extracted and retained to optimize the map's storage structure and query efficiency, thereby obtaining an optimized global map. Based on the optimized global map, a global 3D environment map with semantic information is generated and output in real time for use in path planning and personnel identification.

5. The autonomous detection unmanned aerial vehicle control system for disaster relief in confined spaces according to claim 4, characterized in that, Based on the real-time calculated UAV pose and the constructed and updated 3D environment map, combined with real-time acquired semantically enhanced point cloud data, obstacles in the environment are identified and passable areas are analyzed. This allows for online planning and dynamic adjustment of the UAV's collision-free flight path, including: Based on the real-time acquired semantically enhanced point cloud data of the current frame, obstacle point cloud clusters are identified. By comparing the identification results with the global 3D environment map and updating them incrementally, a navigation map containing real-time obstacle information is obtained. Based on the geometric structure, semantic information and real-time calculated UAV pose in the navigation map, a 3D space accessibility analysis is performed to generate a local cost map centered on the current UAV position, which quantifies the access costs of different areas. Based on the local cost map, the UAV pose, and the next phase of exploration target points issued by the command platform, a collision-free flight path from the current position to the local target point is planned online. The drone is controlled to fly along the collision-free flight path, while the environment is continuously monitored based on the real-time updates of the navigation map. If a new obstacle is detected that makes the original path impassable, path replanning is immediately triggered, and a local cost map is regenerated using the latest environmental information to search for a new safe path.

6. The autonomous detection unmanned aerial vehicle control system for disaster relief in confined spaces according to claim 5, characterized in that, Based on semantically enhanced point cloud data and a constructed 3D environment map, multi-source detection and spatiotemporal correlation were performed to identify trapped personnel and determine their precise locations within the 3D environment map. Further analysis of the trapped personnel's posture and vital signs was conducted, including: Based on semantically enhanced point cloud data, candidate point cloud clusters with human morphological features are identified by scanning using preset point cloud geometric and contour feature templates. At the same time, based on the corresponding visual image data, a lightweight convolutional neural network is used to detect human targets and generate human candidate bounding box information in two-dimensional images. The human candidate bounding box information in the two-dimensional image is combined with the corresponding depth information and projected into three-dimensional space to generate the corresponding three-dimensional visual detection box. The corresponding three-dimensional visual detection box is spatiotemporally associated and matched with the candidate point cloud cluster. For the successfully matched candidate targets, the three-dimensional coordinates of the trapped person in the global three-dimensional environment map are accurately calculated through weighted fusion calculation. For successfully matched candidate targets, their corresponding color image regions and local point cloud data are extracted and input into a pre-trained pose recognition neural network model to analyze the key skeletal points and pose of the human body. At the same time, combined with infrared thermal imaging data, the static or dynamic attributes and vital signs of the target are comprehensively evaluated to generate structured personnel information containing position, pose and state confidence.

7. The autonomous detection unmanned aerial vehicle control system for disaster relief in confined spaces according to claim 6, characterized in that, The constructed 3D environment map, the planned collision-free flight path of the drones, and the precise location and vital signs information of the trapped personnel are transmitted back to the command platform via an adaptive anti-interference communication network. Multi-source data is then fused to generate a global rescue situation, based on which rescue decision support information is generated, including: The global 3D environment map, the collision-free flight path, and the structured personnel information are encapsulated according to a preset data priority, and the data compression rate and transmission frequency are adaptively selected based on the real-time communication link quality. The data is then sent to the command platform through a self-organizing network communication link based on the dual-band Mesh protocol. The command platform receives and decodes data packets transmitted from at least one drone, and simultaneously registers and merges the received local environmental map information and personnel information into a global 3D situation map of the rescue scenario. Based on a global 3D situational map of the rescue scenario, the system analyzes both detected and undetected areas to assess environmental structural risks and mark the location and status of trapped personnel. Based on this, it generates decision support information including rescue channel suggestions, reconnaissance route planning suggestions, and personnel rescue priority plans.

8. The autonomous detection unmanned aerial vehicle control system for disaster relief in confined spaces according to claim 7, characterized in that, Based on a global 3D situational map of the rescue scenario, the system analyzes both detected and undetected areas to assess environmental structural risks and mark the location and status of trapped personnel. This data generates decision support information including rescue route suggestions, reconnaissance path planning suggestions, and personnel rescue priority plans. Analyze the overall 3D situation map of the rescue scenario to identify areas that have been explored by drones and areas that have not yet been explored; Based on the geometric and semantic information in the global 3D situation map of the rescue scenario, the stability of the environmental structure and the potential risk level of the detected area are assessed, and the environmental structure risk assessment results are generated. At the same time, the location information of the identified trapped personnel and their corresponding status information are marked in the global 3D situation map of the rescue scenario. Based on the results of the environmental structural risk assessment, the location and status information of the identified trapped personnel, and the spatial distribution of unexplored areas, a set of feasible rescue channel suggestions, a set of supplementary reconnaissance route suggestions for unexplored areas, and a personnel rescue priority sequence based on status information are generated. The system integrates a set of feasible rescue route suggestions, a set of supplementary reconnaissance route suggestions for unexplored areas, and a personnel rescue priority sequence based on status information to form decision support information that is output to the command personnel.

Images