An unmanned aerial vehicle obstacle avoidance method and system based on multi-sensor fusion

Abstract

The present application relates to the technical field of unmanned aerial vehicle obstacle avoidance, in particular to an unmanned aerial vehicle obstacle avoidance method and system based on multi-sensor fusion, the method comprising the following steps: generating a sparse depth map and a feature map based on pose data alignment and conversion of multi-source sensor data, establishing a penetration deception conflict grid by combining geometric curvature to calculate conflict factors, constructing a three-dimensional layered octree map according to extracted voxels, establishing an environment mapping map by introducing credibility scores of each layer and forbidden boundaries, and generating a fusion grid map by combining time sequence features to search for an obstacle avoidance path. In the present application, a conflict identification grid is established by identifying a special medium area to overcome the identification of a blind area caused by sensing failure, a layered map is generated on a local model to accurately isolate real obstacles and mirror reflection deception sensing, a gradient environment mapping map is established by introducing credibility scores and risk attributes, fine quantitative evaluation of complex space risks is realized, and a safe and reliable obstacle avoidance flight trajectory is output.

Description

Technical Field

[0001] This invention relates to the field of drone obstacle avoidance technology, and in particular to a drone obstacle avoidance method and system based on multi-sensor fusion. Background Technology

[0002] The field of drone obstacle avoidance technology involves the core aspects of unmanned aerial vehicles (UAVs) perceiving surrounding physical obstacles and planning safe flight routes in complex environments. This field encompasses environmental geometric feature acquisition, local map construction, obstacle avoidance trajectory generation, and low-level attitude control. It primarily involves using external detection equipment to collect spatial depth information and dynamically updating the passable area in the current three-dimensional space based on the aircraft's kinematic constraints. Traditional UAV obstacle avoidance methods refer to the execution scheme where, during autonomous flight, the UAV detects surrounding obstacles and changes its preset flight path to prevent physical collisions. The technical challenge it addresses is how UAVs respond to suddenly appearing static or dynamic physical obstacles in unknown airspace.

[0003] Traditional multi-sensor fusion mechanisms rely primarily on the internal consistency of data sources for judgment in actual operation. When faced with complex environments, they lack a deep understanding of the physical failure modes of sensors. When a lidar beam penetrates a glass curtain wall to obtain a point cloud behind it, the visual camera simultaneously extracts surface reflection features. Due to the high internal consistency of the lidar point cloud, it is easy to mistakenly assign it a high degree of trust and filter out the visual features as noise. This physical perception conflict directly triggers the failure of trust allocation, outputting an incorrect environmental access map, which leads to the failure of obstacle avoidance path planning. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method and system for obstacle avoidance of unmanned aerial vehicles (UAVs) based on multi-sensor fusion.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a drone obstacle avoidance method based on multi-sensor fusion, comprising the following steps:

[0006] S1: Collect LiDAR point cloud data, visual camera video images, and UAV pose data. Use the UAV pose data to align the LiDAR point cloud data to obtain a virtual point cloud frame. Use the virtual point cloud frame and the visual camera video image to perform coordinate transformation and projection association to obtain a sparse depth map. Calculate the point cloud geometric curvature based on the virtual point cloud frame. Extract the surface feature confidence map based on the visual camera video image.

[0007] S2: Generate visual surface candidate regions based on the surface feature confidence map and sparse depth map. Combine point cloud geometric curvature, UAV flight speed, and depth anomaly gradient and echo missing rate calculated based on sparse depth map and virtual point cloud frame to calculate comprehensive conflict factor. Identify medium conflict regions based on the comprehensive conflict factor and establish a penetration misleading conflict identification grid.

[0008] S3: Generate a local spatial planar model based on the medium conflict area and the UAV pose data, generate surface blocking layer voxels from the surface feature confidence map, and extract background detection layer voxels from the virtual point cloud frame by combining the penetration misleading conflict identification grid, and merge the surface blocking layer voxels and the background detection layer voxels to establish a three-dimensional layered octree map.

[0009] S4: Calculate the surface layer confidence score based on the local spatial plane model and the surface feature confidence map, calculate the background layer confidence score based on the background detection layer voxels, generate a high-risk no-entry boundary from the surface barrier layer voxels, and assign the high-risk no-entry boundary, the surface layer confidence score and the background layer confidence score to the three-dimensional layered octree map to establish an environment mapping map with risk gradient attributes;

[0010] S5: Extract the temporal features of the visual camera video image to establish the temporal confidence of the surface barrier layer, and generate the trajectory of the isolated dynamic target based on the voxels of the background detection layer. Establish an elevation risk fusion grid map by replacing the environment mapping map with risk gradient attributes according to the high-risk no-entry boundary, the temporal confidence of the surface barrier layer and the trajectory of the isolated dynamic target. Retrieve the elevation risk fusion grid map to generate a collision-free obstacle avoidance flight path.

[0011] The present invention is improved in that the specific steps of S1 are as follows:

[0012] S111: Collect LiDAR point cloud data, visual camera video images, and UAV pose data; extract the UAV pose data as the pose smoothing node of the quaternion spherical linear interpolation algorithm; calculate the virtual pose deviation corresponding to the exposure time of the LiDAR point cloud data and the visual camera video image based on the quaternion spherical linear interpolation algorithm; adjust the spatial position of the LiDAR point cloud data according to the virtual pose deviation; and perform pose alignment to obtain a virtual point cloud frame.

[0013] S112: Obtain the rotation matrix, translation vector, and camera intrinsic matrix from the joint extrinsic calibration file of the LiDAR and vision camera. Use the rotation matrix and translation vector to calculate the spatial coordinate transformation result of the virtual point cloud frame in the camera coordinate system. Use the spatial coordinate transformation result to unify the spatial reference benchmark of the LiDAR and the camera. Use principal component analysis to calculate the geometric curvature of the point cloud in the local spherical neighborhood of the point cloud. Calculate the projection coordinate relationship between the three-dimensional depth points contained in the virtual point cloud frame and the two-dimensional color pixels contained in the video image of the vision camera according to the camera intrinsic matrix, thereby establishing a sparse depth map.

[0014] S113: Call the video image from the visual camera to perform image color space conversion operation, extract the highlight area of ​​the brightness channel and the high-frequency linear edge of the image, calculate the continuous surface texture contained in the video image from the visual camera based on the gray-level co-occurrence matrix, and generate a surface feature confidence map by calculating the fusion feature map of the highlight area of ​​the brightness channel, the high-frequency linear edge of the image and the continuous surface texture through a weighted algorithm.

[0015] The present invention is improved in that the specific steps of S2 are as follows:

[0016] S211: Divide the spatial grid of the surface feature confidence map according to the multi-scale grid partitioning strategy corresponding to the image pyramid to generate local grid regions, obtain the global illumination intensity measured by the illumination sensing hardware, calculate the arithmetic mean of the confidence of the pixels covered in the local grid region to obtain the surface feature confidence mean, adjust the visual surface presence threshold according to the global illumination intensity, calculate the difference between the surface feature confidence mean and the visual surface presence threshold, and filter the local grid regions whose difference is greater than the preset difference filtering threshold to generate visual surface candidate regions;

[0017] S212: Calculate the effective echo rate by the ratio between the number of effective depth points and the total number of pixels in the grid within the coverage area of ​​the candidate region of the sparse depth map. When the effective echo rate is higher than a preset echo judgment threshold, extract the average depth value corresponding to the projected point cloud of the sparse depth map in the candidate region of the visual surface. Extract the internal depth data corresponding to the adjacent grids outside the local grid region of the sparse depth map. Calculate the reference depth mean and depth standard deviation based on the internal depth data. Calculate the depth difference between the average depth value and the reference depth mean. Calculate the depth anomaly gradient by the ratio between the depth difference and the depth standard deviation. Call the point cloud geometric curvature. When the point cloud geometric curvature is lower than a preset curvature judgment threshold, calculate the echo difference between the value 1 and the effective echo rate to obtain the echo missing rate.

[0018] S213: Collect the UAV flight speed, assign depth anomaly weight coefficient and echo missing weight coefficient based on the UAV flight speed, calculate the anomaly product value based on the depth anomaly weight coefficient and the depth anomaly gradient, calculate the missing product value based on the echo missing weight coefficient and the echo missing rate, calculate the sum of the anomaly product value and the missing product value to obtain the comprehensive conflict factor, filter the regions where the comprehensive conflict factor is greater than the preset conflict alarm threshold to generate media conflict regions, and for the media conflict regions, when the depth anomaly gradient is greater than the preset anomaly gradient threshold, mark the media conflict regions and establish a penetration misleading conflict identification grid.

[0019] The present invention is improved in that the process of adjusting the visual surface presence threshold according to the global illumination intensity specifically includes:

[0020] Obtain the baseline lighting environment parameters recorded in the factory calibration file of the lighting sensing hardware;

[0021] Extract the standard illuminance value and initial response scaling factor from the reference lighting environment parameters;

[0022] Calculate the environmental illuminance deviation between the global illumination intensity and the standard illuminance value;

[0023] A smooth attenuation correspondence dataset generated by fitting multiple sets of surface feature confidence fluctuation data under known test lighting conditions is used to set up a nonlinear lighting compensation function mapping table.

[0024] The dynamic compensation coefficient corresponding to the environmental illuminance deviation value is retrieved based on the nonlinear illumination compensation function mapping table.

[0025] The illumination adjustment control weight is obtained by calculating the numerical product between the dynamic compensation coefficient and the initial response scaling factor.

[0026] Under ideal diffuse reflection conditions without ambient light interference, the mean confidence score of surface features of statistical surface feature samples is used as the static surface confidence benchmark value.

[0027] The visual surface presence threshold is obtained by calculating the product between the illumination adjustment control weight and the static surface confidence benchmark value.

[0028] The present invention is improved in that the specific steps of S3 are as follows:

[0029] S311: Extract the coordinate boundary of the medium conflict region in the three-dimensional global coordinate system, divide the surface blocking layer and the background detection layer according to the coordinate boundary, call the penetration misleading conflict marker grid, the virtual point cloud frame and the reflection intensity value, extract the reflection intensity value of the outer boundary of the medium conflict region that does not have the penetration misleading conflict marker grid in the virtual point cloud frame, extract the point cloud corresponding to the reflection intensity value that is greater than or equal to the point cloud reflection threshold, and obtain the edge point cloud.

[0030] S312: Extract the UAV yaw angle from the UAV pose data, obtain the prior building contours attached to the global map, extract the UAV yaw angle and the prior building contours as plane normal vector constraints for the random sampling consensus algorithm, calculate the plane normal vector, plane intercept and plane fitting inlier rate of the edge point cloud in three-dimensional space based on the random sampling consensus algorithm, fit and generate a local space plane model, call the camera intrinsic parameter matrix and the surface feature confidence map, calculate the back projection intersection points of the grid pixels contained in the surface feature confidence map along the camera optical center ray according to the camera intrinsic parameter matrix, project the back projection intersection points to the surface of the local space plane model, and establish a surface blocking layer voxel;

[0031] S313: Extract the three-dimensional depth position data contained in the virtual point cloud frame with the coverage area of ​​the mesh with the penetration misleading conflict identifier, retain the three-dimensional depth position data and convert it to generate a background detection layer voxel, call the surface blocking layer voxel, and merge the spatial coordinates of the surface blocking layer voxel and the background detection layer voxel in the same line of sight to construct a three-dimensional layered octree map.

[0032] The present invention is improved in that the process of extracting the point cloud corresponding to the reflection intensity value that is greater than or equal to the point cloud reflection threshold to obtain the edge point cloud specifically includes:

[0033] Calculate the mean and standard deviation of the reflection intensity values;

[0034] The numerical proportional relationship between the width of the known entity's reflective surface echo intensity distribution interval and the mean offset is used to set the weight for adjusting the reflection intensity.

[0035] The dynamic reflection compensation value is obtained by multiplying the standard deviation of the reflection intensity and the adjustment weight of the reflection intensity.

[0036] The point cloud reflection threshold is obtained by summing the mean reflection intensity and the dynamic reflection compensation value.

[0037] Compare the numerical values ​​between the reflection intensity value and the point cloud reflection threshold;

[0038] Extract the point clouds corresponding to the reflection intensity values ​​that are greater than or equal to the point cloud reflection threshold to generate a set of high-reflectivity candidate point clouds;

[0039] Calculate the Euclidean space distance between point cloud data in the high-reflectivity candidate point cloud set;

[0040] The connectivity distance threshold is set by calculating the arc length product between the radar hardware scanning angular resolution and the current detection space distance.

[0041] The edge point cloud is obtained by performing a spatial aggregation operation on the point cloud data whose Euclidean spatial distance is less than the connectivity distance threshold.

[0042] The present invention is improved in that the specific steps of S4 are as follows:

[0043] S411: Extract the in-plane fitting rate of the local spatial plane model, calculate the product between the mean confidence of the surface features and the in-plane fitting rate to obtain the surface layer confidence score, extract the surface layer confidence score as the negative exponential part of the exponential decay model, calculate the negative exponential power of the natural logarithm base based on the exponential decay model to obtain the decay coefficient, call the background detection layer voxel, obtain the initial confidence of the background detection layer voxel grid corresponding to the background detection layer voxel, and calculate the penalty product result based on the decay coefficient and the initial confidence of the background detection layer voxel grid to generate the background layer confidence score;

[0044] S412: Collect the UAV size radius, current flight speed vector, end-to-end communication delay time, and maximum braking distance contained in the UAV hardware bounding box; calculate the communication delay displacement by multiplying the current flight speed vector and the end-to-end communication delay time; calculate the sum of the lengths corresponding to the communication delay displacement, the UAV size radius, and the maximum braking distance to establish a dynamic safety buffer thickness; call the surface barrier layer voxel; extract the mesh normal vector direction corresponding to the surface barrier layer voxel; expand the barrier boundary range of the surface barrier layer voxel along the mesh normal vector direction according to the dynamic safety buffer thickness to generate a high-risk no-entry boundary.

[0045] S413: Extract the high-risk forbidden boundary, the surface layer credibility score, and the background layer credibility score as semantic attribute labels, assign the semantic attribute labels to the three-dimensional layered octree map, and establish an environment mapping map with risk gradient attributes.

[0046] The present invention is improved in that the specific steps of S5 are as follows:

[0047] S511: Extract the window frame corner points and image stains contained in the visual camera video image as static surface features, calculate the relative displacement vector between each static surface feature in the continuous visual camera video images using optical flow method, calculate the trajectory deviation between the relative displacement vector between the images and the expected projection trajectory calculated from the UAV pose data, determine that the surface barrier layer voxel has temporal continuity when the trajectory deviation is less than the deviation judgment threshold, and accumulate the surface layer confidence score to establish the temporal confidence of the surface barrier layer;

[0048] S512: Calculate the spatial distance clustering degree of the point cloud contained in the voxels of the background detection layer based on the Euclidean clustering algorithm, merge the point clouds whose spatial distance clustering degree is less than the distance clustering threshold to establish cluster centroid nodes, extract the cluster 3D position and cluster 3D velocity contained in the cluster centroid nodes as dynamic state tracking vectors, extract the grid coordinate boundary contained in the voxels of the surface barrier layer to establish an isolation mask matrix, and use a Kalman filter combined with the isolation mask matrix to limit the covariance update range of the dynamic state tracking vector to generate the trajectory of the isolated dynamic target;

[0049] S513: Collect the approach velocity of the UAV, calculate the relative distance between the surface barrier layer voxel and the grid corresponding to the current position of the UAV, calculate the ratio between the grid relative distance and the UAV approach velocity to obtain the relative collision prediction time, when the relative collision prediction time is lower than the emergency obstacle avoidance threshold, combine the temporal confidence of the surface barrier layer, extract the coverage area of ​​the high-risk prohibited boundary and the trajectory of the isolated dynamic target in the environment mapping map with risk gradient attributes, replace the cost value of the coverage area with the penalty constant generation value to establish an elevation risk fusion grid map, search for available grids corresponding to non-obstacle labels in the elevation risk fusion grid map through the optimization algorithm, calculate the minimum detour curvature between the lines connecting the available grids, and generate a collision-free obstacle avoidance flight path.

[0050] The present invention is improved in that the process of accumulating the surface layer confidence score to establish the time-series confidence of the surface barrier layer specifically includes:

[0051] Extract the historical time-series confidence score of the previous image frame data record;

[0052] In statistical historical test data, the probability of stable static surface features between consecutive frames is used to set a time-series smoothing transition coefficient;

[0053] The historical transit confidence level is obtained by calculating the product between the historical time series confidence level and the time series smoothing transition coefficient.

[0054] The initial time-series confidence level is obtained by summing the historical transmission confidence level and the surface layer confidence score.

[0055] Extract the pre-defined obstacle occupancy probability safety boundary value from the global map file as the confidence saturation upper limit threshold;

[0056] Compare the initial time-series confidence level with the upper limit of the confidence level saturation threshold.

[0057] When the initial time series confidence level is greater than the upper limit threshold of the confidence level saturation, the upper limit threshold of the confidence level saturation is extracted as the time series confidence level of the surface barrier layer;

[0058] When the initial time-series confidence level is less than or equal to the confidence level saturation upper limit threshold, the initial time-series confidence level is extracted as the time-series confidence level of the surface barrier layer.

[0059] A drone obstacle avoidance system based on multi-sensor fusion, wherein the multi-sensor fusion-based drone obstacle avoidance system is used to implement the aforementioned multi-sensor fusion-based drone obstacle avoidance method, and the system includes:

[0060] The multi-source sensing access module collects LiDAR point cloud data, visual camera video images, and UAV pose data. It uses the UAV pose data to align the LiDAR point cloud data to obtain a virtual point cloud frame. It uses the virtual point cloud frame and the visual camera video image to perform coordinate transformation and projection association to obtain a sparse depth map. It calculates the point cloud geometric curvature based on the virtual point cloud frame and extracts the surface feature confidence map based on the visual camera video image.

[0061] The cross-domain consistency identification module generates visual surface candidate regions based on the surface feature confidence map and sparse depth map. It combines the point cloud geometric curvature, UAV flight speed, and depth anomaly gradient and echo missing rate calculated based on the sparse depth map and virtual point cloud frame to calculate a comprehensive conflict factor. Based on the comprehensive conflict factor, it identifies medium conflict regions and establishes a penetration misleading conflict identification grid.

[0062] The spatial structure analysis module generates a local spatial planar model based on the medium conflict area and the UAV pose data. It generates surface blocking layer voxels from the surface feature confidence map and extracts background detection layer voxels from the virtual point cloud frame by combining the penetration misleading conflict identification grid. It merges the surface blocking layer voxels and the background detection layer voxels to establish a three-dimensional layered octree map.

[0063] The security situation assignment module calculates the surface layer confidence score based on the local spatial plane model and the surface feature confidence map, calculates the background layer confidence score based on the background detection layer voxels, generates a high-risk no-entry boundary from the surface barrier layer voxels, and assigns the high-risk no-entry boundary, the surface layer confidence score, and the background layer confidence score to the three-dimensional layered octree map to establish an environment mapping map with risk gradient attributes.

[0064] The continuous trajectory decision module extracts the temporal features of the visual camera video images to establish the temporal confidence of the surface barrier layer, and generates the trajectory of the isolated dynamic target based on the voxels of the background detection layer. It establishes an elevation risk fusion grid map by replacing the environment mapping map with risk gradient attributes according to the high-risk no-entry boundary, the temporal confidence of the surface barrier layer, and the cost of the isolated dynamic target trajectory. It then retrieves the elevation risk fusion grid map to generate a collision-free obstacle avoidance flight path.

[0065] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0066] In this invention, a sparse depth map and a surface feature confidence map are generated by aligning poses and transforming coordinates. A comprehensive conflict factor is calculated by combining multi-dimensional data such as depth anomaly gradients. Special medium regions are accurately identified and a penetration-misleading conflict identification grid is constructed. This effectively overcomes the blind spots of conventional perception in identifying physical failure modes. A three-dimensional layered octree map containing surface obstruction and background detection is generated on the basis of a local spatial planar model. This accurately isolates and distinguishes real obstacles from reflection-misleading perception. At the same time, the confidence scores of each layer are introduced to establish high-risk no-entry boundaries. Under this framework, temporal features are extracted to establish an elevation risk fusion grid map, realizing dynamic quantitative assessment of complex environmental risks and outputting a collision-free obstacle avoidance flight path. Attached Figure Description

[0067] Figure 1 This is the main flowchart of the multi-sensor fusion obstacle avoidance method of the present invention;

[0068] Figure 2 This is a flowchart of the multi-source sensing alignment and feature generation process of the present invention;

[0069] Figure 3 This is a flowchart of the media conflict identification and identifier generation process of the present invention;

[0070] Figure 4 This is a flowchart illustrating the hierarchical risk mapping and path generation process of this invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0072] Please see Figures 1 to 4 This embodiment provides a method and system for obstacle avoidance of unmanned aerial vehicles (UAVs) based on multi-sensor fusion. In practical applications, such as during the obstacle avoidance control process of a low-altitude inspection UAV continuously flying near building facades, transparent barriers, and reflective components, the airborne lidar, visual camera, inertial measurement unit, flight control attitude interface, and local map cache continuously generate spatial perception data within the same flight cycle. This method addresses the inconsistency between radar penetration misleading caused by transparent or reflective media and the visible surface, and completes point cloud alignment, visual projection, conflict identification, spatial layer mapping, risk assignment, and path retrieval in a unified time sequence, including the following steps:

[0073] S1: Acquire LiDAR point cloud data, visual camera video images, and UAV pose data. Use the UAV pose data to align the LiDAR point cloud data to obtain a virtual point cloud frame. Perform coordinate transformation and projection association between the virtual point cloud frame and the visual camera video image to obtain a sparse depth map. Calculate the point cloud geometric curvature based on the virtual point cloud frame, and extract a surface feature confidence map from the visual camera video image. The virtual point cloud frame refers to the point cloud frame after compensating the original LiDAR scan points to the same spatial reference according to the UAV pose corresponding to the image exposure time. It carries 3D coordinates, radar echo intensity, acquisition time, and pose alignment status fields. The sparse depth map is a pixel depth index map formed by projecting the pose-aligned 3D depth points onto the camera pixel plane. It includes pixel location, depth source, corresponding point cloud identifier, and effective projection status fields. The surface feature confidence map is a pixel-level or grid-level surface existence state map formed by highlights, linear edges, and continuous textures in the visual image. It carries the source of visual surface features, local confidence level, and spatial index required for subsequent candidate region segmentation.

[0074] During the data acquisition phase, timestamp integrity is verified on data from LiDAR, cameras, and flight control pose interfaces. Data with missing, duplicate, or out-of-bounds timestamps are entered into a calibration buffer and written to the sensor status record. In the event of short-term frame loss within the same period, the pose alignment status of the previous valid frame is retained as a temporary reference, and subsequent risks are marked as pending confirmation. Verified data is entered into a perception synchronization buffer for continuous access from S111 to S113.

[0075] S111: Acquire LiDAR point cloud data, visual camera video images, and UAV pose data. Extract the UAV pose data as the attitude smoothing node for the quaternion spherical linear interpolation algorithm. Calculate the virtual pose deviation corresponding to the exposure time of the LiDAR point cloud data and the visual camera video image based on the quaternion spherical linear interpolation algorithm. Adjust the spatial position of the LiDAR point cloud data according to the virtual pose deviation and perform pose alignment to obtain a virtual point cloud frame. In this method, the quaternion spherical linear interpolation algorithm is used to handle the rotational transition between adjacent attitude smoothing nodes. The attitude smoothing node consists of the attitude quaternion, position state, and acquisition time output by the flight control pose interface. The virtual pose deviation is the attitude difference state between the point cloud scanning time and the image exposure time, carrying the rotation compensation direction, translation compensation direction, and synchronization validity identifier.

[0076] The lidar point cloud data is first divided into corresponding exposure windows according to point-level time stamps, and then the pose smoothing nodes within the same window are read. When a node is missing, the previous valid attitude node and the current inertial state prediction node are called to form compensation input, and the compensation source is written to the synchronization log. After compensation is completed, the spatial position of each point cloud data is transformed to the global coordinate reference corresponding to the camera exposure time, and a virtual point cloud frame is output. The virtual point cloud frame is written to the point cloud alignment buffer, and spatial coordinates, echo intensity, point cloud index, and alignment status are provided to S112.

[0077] S112: Obtain the rotation matrix, translation vector, and camera intrinsic matrix from the joint extrinsic parameter calibration file of the LiDAR and vision camera. Calculate the spatial coordinate transformation result of the virtual point cloud frame in the camera coordinate system using the rotation matrix and translation vector. Use the spatial coordinate transformation result to unify the spatial reference benchmark between the radar and camera. Calculate the geometric curvature of the point cloud frame in the local spherical neighborhood of the point cloud using principal component analysis. Calculate the projection coordinate relationship between the 3D depth points contained in the virtual point cloud frame and the 2D color pixels contained in the vision camera video image based on the camera intrinsic parameter matrix, thereby establishing a sparse depth map. The joint extrinsic parameter calibration file is a configuration file written during the UAV assembly and calibration phase. It contains attitude transformation fields from the radar coordinate system to the camera coordinate system, position offset fields, camera imaging parameter fields, and calibration version fields. If the calibration file reading fails or the version is mismatched, the current cycle enters a conservative obstacle avoidance state, and subsequent maps do not accept new projection points, only retaining the risk boundaries of the previous valid cycle.

[0078] Principal component analysis (PCA) is used in this method to determine the geometric flatness of the point cloud distribution within a local spherical neighborhood. The geometric curvature of the point cloud is a dimensionless geometric state field reflecting the deviation of the local point cloud from the planar structure. The local spherical neighborhood is jointly determined by the radar angular resolution, the current detection range category, and the point cloud density state. When there are insufficient effective points in the neighborhood, the geometric curvature of that point is marked as undeterminable and enters the low-confidence echo branch in S2. During projection association, points whose projections fall outside the camera imaging range, whose depth direction does not meet the forward observation constraints, or whose projections are inconsistent with the image time window are not written into the sparse depth map; only invalid projection records are written. Effective projection points form an index relationship from pixels to the point cloud and serve as inputs for S2 to calculate the effective echo rate, depth anomaly gradient, and candidate region depth state.

[0079] S113: The system calls upon the video image from the visual camera to perform color space conversion, extracting the highlight region of the luminance channel and the high-frequency linear edges of the image. Based on the gray-level co-occurrence matrix, it calculates the continuous surface texture contained in the visual camera video image. A weighted algorithm is then used to calculate the fusion feature map of the highlight region of the luminance channel, the high-frequency linear edges of the image, and the continuous surface texture to generate a surface feature confidence map. The highlight region of the luminance channel is the connected pixel region in the image luminance component that satisfies the reflective surface response rules. The high-frequency linear edges of the image are pixel line segments with continuous geometric directions after edge response filtering. The continuous surface texture is the stable texture state formed by statistically analyzing the local gray-level co-occurrence relationship using the gray-level co-occurrence matrix.

[0080] The weight fields in the weighted algorithm are derived from camera factory calibration, lens exposure status, inspection scene calibration records, and the lighting stability status of the previous valid period. These weight fields are stored in the visual rule cache and refreshed when lighting conditions change, exposure modes switch, or the camera calibration file is updated. Highlights, edges, and textures are first converted to a unified confidence level before entering the fusion process, avoiding direct cross-dimensional comparisons between brightness response, edge response, and texture response. The resulting surface feature confidence map retains the feature source identifier for each grid. When the image has occlusions, exposure anomalies, or motion blur, a visual anomaly identifier is written to the corresponding area, and its direct trigger weight is reduced during the candidate region selection in S2.

[0081] S2: Based on the surface feature confidence map and sparse depth map, visual surface candidate regions are generated. Combining the point cloud geometric curvature, UAV flight speed, and depth anomaly gradient and echo missing rate calculated based on the sparse depth map and virtual point cloud frames, a comprehensive conflict factor is calculated. Based on this comprehensive conflict factor, media conflict regions are identified, and a penetration-misleading conflict marker grid is established. The visual surface candidate region refers to an image grid region in the visual image that possesses surface features and requires consistency verification with radar depth. The depth anomaly gradient is the deviation level field of the depth state within the candidate region relative to the adjacent reference depth state. The echo missing rate is the echo incompleteness state field formed when the visual surface exists but the effective radar echo is insufficient. The comprehensive conflict factor is a conflict level field that unifies the mapping of depth deviation state, echo incompleteness state, point cloud geometric flatness state, and flight speed state, and is subsequently used to generate media conflict regions.

[0082] S211: Based on the multi-scale grid partitioning strategy corresponding to the image pyramid, a spatial grid is generated from the surface feature confidence map to form local grid regions. The global illumination intensity measured by the illumination sensing hardware is obtained. The arithmetic mean of the confidence scores of the pixels within the local grid region is calculated to obtain the mean confidence score of the surface features. The visual surface presence threshold is adjusted according to the global illumination intensity. The difference between the mean confidence score of the surface features and the visual surface presence threshold is calculated. Local grid regions with a difference greater than a preset difference threshold are selected to generate visual surface candidate regions. In this method, the multi-scale grid partitioning strategy corresponding to the image pyramid is used to form mutually inclusive grid indices of the same image according to different spatial granularities. The upper-layer grid is used to stably identify large-area surfaces, while the lower-layer grid is used to preserve edge and local reflective details. The local grid region carries the image coordinate boundaries, the mean confidence score of the features, visual anomaly markers, and candidate state fields.

[0083] Global illumination intensity is output by the onboard illumination sensing hardware. After entering the illumination state cache, it is matched with the baseline illumination environment parameters recorded in the hardware's factory calibration file. The baseline illumination environment parameters include a calibration state field corresponding to the standard illuminance value and an initial response scaling factor field, both of which serve as the source of threshold adjustment. After classifying the deviation between the current illumination state and the baseline illumination state, a nonlinear illumination compensation function mapping table formed by fitting surface feature confidence fluctuation data under the test illumination environment is retrieved to obtain the compensation level corresponding to the dynamic compensation coefficient. The dynamic compensation coefficient and the initial response scaling factor together form the illumination adjustment control weight. This weight is associated with the static surface confidence baseline field obtained statistically under ideal diffuse reflection conditions to generate a visual surface presence threshold. The visual surface presence threshold is stored in the visual candidate area rule cache and is called by the candidate area screening and abnormal illumination verification of the current cycle.

[0084] S212: Calculate the effective echo rate by the ratio of the number of effective depth points within the candidate area coverage of the sparse depth map to the total number of pixels in the grid. When the effective echo rate is higher than a preset echo judgment threshold, extract the average depth value corresponding to the projected point cloud of the sparse depth map within the candidate area of ​​the visual surface. Extract the internal depth data corresponding to the adjacent grids outside the local grid region of the sparse depth map. Calculate the reference depth mean and depth standard deviation based on the internal depth data. Calculate the depth difference between the average depth value and the reference depth mean. Calculate the ratio between the depth difference and the depth standard deviation to obtain the depth anomaly gradient. Call the point cloud geometric curvature. When the point cloud geometric curvature is lower than a preset curvature judgment threshold, calculate the echo difference between the reference value of the numerical complete echo state and the effective echo rate to obtain the echo missing rate. Effective depth points are depth points that have completed projection association, passed depth direction verification, are not covered by repeated projection conflicts, and have valid point cloud indexes. The total number of pixels in the grid is only used as a discrete sampling caliber for the grid coverage area. It is converted into an effective echo level during rule processing and is not output as a data instance.

[0085] The reference depth mean and depth standard deviation are derived from the internal depth data of adjacent grids surrounding the candidate area. These adjacent grids must be of the same scale, exposure window, and projection validity state as the candidate area. When adjacent grid depth data is missing, the depth state of the same neighborhood in the previous valid period is used as a reference, and the depth anomaly gradient is written with an inheritance source identifier. If historical states are also unavailable, the candidate area is transferred to the conservative conflict verification queue. When the point cloud geometric curvature is lower than the boundary corresponding to the curvature determination threshold field, it indicates that the local shape of the point cloud is close to planar echo characteristics. In this case, the visual surface exists, but the echo is insufficient, and this state is recorded as the echo missing rate. The curvature determination threshold field is derived from radar hardware calibration, flight altitude category, and building facade prior type, stored in the point cloud geometric rule cache, and used for S3 edge point cloud extraction and reuse.

[0086] S213: Collect the UAV flight speed, assign depth anomaly weight coefficient and echo missing weight coefficient based on the UAV flight speed, calculate the anomaly product value based on the depth anomaly weight coefficient and depth anomaly gradient, calculate the missing product value based on the echo missing weight coefficient and echo missing rate, and sum the anomaly product value and missing product value to obtain the comprehensive conflict factor. Regions with a comprehensive conflict factor greater than a preset conflict alarm threshold are selected to generate media conflict regions. For media conflict regions, when the depth anomaly gradient is greater than a preset anomaly gradient threshold, the media conflict region is marked and a penetration misleading conflict identification grid is established. The UAV flight speed is output from the flight control interface, first converted into a speed status level, and then entered into the weight allocation rule table. The depth anomaly weight coefficient represents the degree of influence of depth deviation on conflict identification under the current flight state, and the echo missing weight coefficient represents the degree of influence of insufficient radar echo on conflict identification. Both are dimensionless configuration fields, derived from flight control safety policies, radar detection status, and historical inspection calibration rules, and stored in the conflict identification rule cache.

[0087] The comprehensive conflict factor output is a conflict level field, which is compared with the boundary corresponding to the preset conflict alarm threshold field. The conflict alarm threshold field is derived from the UAV safety policy configuration, flight speed status, and obstacle occupancy probability safety boundary, and does not change directly with the instantaneous noise of a single grid. Regions that reach the conflict alarm state are written into the medium conflict region cache. The medium conflict region includes the image grid boundary, the corresponding 3D spatial boundary, the conflict source identifier, the depth anomaly state, and the echo missing state. When the depth anomaly gradient reaches the boundary corresponding to the anomaly gradient threshold field, the corresponding medium conflict region is marked as a penetration misleading conflict identifier grid. The penetration misleading conflict identifier grid is used to indicate that there is a medium layer conflict between the visual surface and the radar background detection, which S3 uses to distinguish the surface blocking layer and the background detection layer.

[0088] S3: A local spatial planar model is generated based on the media conflict area and UAV pose data. Surface blocking layer voxels are generated from the surface feature confidence map, and background detection layer voxels are extracted from the virtual point cloud frame by combining the penetrating misleading conflict marker mesh. The surface blocking layer voxels and background detection layer voxels are merged to establish a 3D hierarchical octree map. The local spatial planar model refers to the local planar structure formed near the outer boundary of the media conflict area by fitting edge point clouds, UAV yaw angle, and prior building contours. It carries the plane normal vector, spatial intercept, fitted inlier rate, and applicable spatial boundary fields. Surface blocking layer voxels are the visual surface occupancy units projected onto the surface of the local spatial planar model along the camera's optical center ray. Background detection layer voxels are the background detection occupancy units located behind the corresponding line of sight of the penetrating misleading conflict marker mesh and retained by the radar virtual point cloud frame. The 3D hierarchical octree map is a spatial index structure that stores surface blocking layer voxels and background detection layer voxels in the same line of sight as hierarchical nodes.

[0089] S311: Extract the coordinate boundaries of the media conflict area in the 3D global coordinate system. Based on the coordinate boundaries, divide the surface blocking layer and the background detection layer. Call the penetrating misleading conflict marker grid, virtual point cloud frame, and reflection intensity value. Extract the reflection intensity value of the outer boundary of the media conflict area without the penetrating misleading conflict marker grid in the virtual point cloud frame. Extract the point cloud corresponding to the reflection intensity value greater than or equal to the point cloud reflection threshold to obtain the edge point cloud. The 3D global coordinate system adopts the current map coordinate reference output by the flight control pose interface. The coordinate boundary is formed by back-projecting the image grid boundary of the media conflict area into space. The surface blocking layer corresponds to the visual surface boundary near the UAV, and the background detection layer corresponds to the radar point cloud distribution boundary behind the penetrating misleading conflict marker grid.

[0090] The point cloud reflection threshold field is jointly determined by the mean reflection intensity field, the standard deviation reflection intensity field, and the reflection intensity adjustment weight field within the current outer boundary region. The reflection intensity adjustment weight field is derived from the calibration relationship between the width of the known entity's reflective surface echo intensity distribution interval and the mean offset, and is stored in the radar echo rule cache. First, the reflection intensity values ​​are checked for integrity and abnormal saturation, and then a high-reflectivity candidate point cloud set is formed. The point clouds in the high-reflectivity candidate point cloud set are spatially clustered according to Euclidean spatial distance. The connectivity distance threshold field is determined by the radar hardware scanning angular resolution and the arc length aperture corresponding to the current detection spatial distance category. Point clouds that satisfy the connectivity rules are clustered into edge point clouds, which are written to the plane fitting input cache for use by S312.

[0091] S312: Extract the UAV yaw angle from the UAV pose data, obtain the prior building contours attached to the global map, and extract the UAV yaw angle and prior building contours as plane normal vector constraints for the random sampling consensus algorithm. Based on the random sampling consensus algorithm, calculate the plane normal vector, plane intercept, and plane fitting inlier rate of the edge point cloud in 3D space, and fit a local spatial plane model. Call the camera intrinsic parameter matrix and surface feature confidence map, and calculate the back projection intersection points of the grid pixels contained in the surface feature confidence map along the camera optical center ray according to the camera intrinsic parameter matrix. Project the back projection intersection points onto the surface of the local spatial plane model to establish a surface blocking layer voxel. In this method, the random sampling consensus algorithm is used to select a set of point clouds consistent with the planar structure from the edge point cloud. The plane normal vector constraint is used to restrict the fitting results to maintain spatial consistency with the current UAV orientation and the building contour direction. The prior building contours are derived from the global map file and carry the building facade boundary, contour direction, and map coordinate reference fields.

[0092] The in-plane fitting point rate is the point cloud occupancy state field in the edge point cloud that conforms to the fitting plane consistency rule, used by S4 to calculate the surface layer confidence score. If the edge point cloud is insufficient, the prior building outline is missing, or the yaw angle state is abnormal, the local spatial plane model enters the pending confirmation state, and a temporary surface projection surface is generated using the spatial boundary of the medium conflict area. The temporary surface projection surface only participates in the generation of high-risk forbidden boundaries and does not participate in background confidence enhancement. After the grid pixels in the surface feature confidence map are back-projected along the camera optical center ray, they form intersections with the surface of the local spatial plane model. The intersections are converted into spatial voxel indices and written to the surface barrier layer voxel cache. Each surface barrier layer voxel records the visual confidence source, projection ray identifier, plane model version, and voxel occupancy state.

[0093] S313: Extract the 3D depth position data contained in the virtual point cloud frame from the area covered by the mesh with penetration misleading conflict markers. Retain the 3D depth position data and convert it into background detection layer voxels. Call the surface blocking layer voxels and merge the spatial coordinates of the surface blocking layer voxels and the background detection layer voxels in the same line of sight to construct a 3D layered octree map. Point clouds within the area covered by the mesh with penetration misleading conflict markers are not directly merged into a single obstacle, but are preserved according to the line of sight direction, depth level, and point cloud index. The 3D depth position data includes the spatial position of the point cloud in the global coordinate system, the corresponding projected pixels, echo intensity, and alignment status.

[0094] The line-of-sight direction is determined by the relationship between the camera's optical center ray and the point cloud projection. Voxels of the surface blocking layer closest to the UAV along the same line-of-sight direction are designated as foreground layer nodes, and voxels of the background detection layer behind them are designated as background layer nodes. Nodes in the 3D layered octree map record the layer identifier, voxel coordinates, occupancy status, source sensor, conflict identifier, and update time. When voxels are repeatedly written, the most recent valid data is retained, and the older status is written to the history cache. When there is a spatial conflict between the foreground and background layers, the layered structure is maintained first, and the background point cloud is not allowed to cover the foreground blocking layer.

[0095] S4: Calculate the surface layer confidence score based on the local spatial planar model and surface feature confidence map. Calculate the background layer confidence score based on the background detection layer voxels. Generate a high-risk no-entry boundary from the surface barrier layer voxels. Assign the high-risk no-entry boundary, surface layer confidence score, and background layer confidence score to a 3D hierarchical octree map to establish an environment mapping map with risk gradient attributes. The surface layer confidence score is a state field describing the spatial confidence level of the visual surface barrier layer in the current period, derived from the consistency of visual surface features and local planar fitting. The background layer confidence score is a state field describing the influence of the background detection layer on obstacle avoidance decisions when surface barriers are present, constrained by the surface layer confidence score. The high-risk no-entry boundary is the flight prohibition space boundary formed by the expansion of the surface barrier layer voxels along the normal vector direction, carrying the dynamic safety buffer thickness, voxel source, and safety status.

[0096] S411: Extract the plane fitting inlier rate from the local spatial plane model. Calculate the product between the mean surface feature confidence score and the plane fitting inlier rate to obtain the surface layer confidence score. Extract the surface layer confidence score as the negative exponential part of the exponential decay model. Calculate the negative exponential power of the natural logarithm base based on the exponential decay model to obtain the decay coefficient. Call the background detection layer voxels to obtain the initial confidence of the background detection layer voxel grid corresponding to the voxels. Calculate the penalty product based on the decay coefficient and the initial confidence of the background detection layer voxel grid to generate the background layer confidence score. The mean surface feature confidence score comes from the local grid region generated in S211, and the plane fitting inlier rate comes from the local spatial plane model generated in S312. Both are first converted to a unified confidence level caliber before being incorporated into the surface layer confidence score rules. The closer the surface layer confidence score is to the surface obstacle confirmation state, the more the direct confidence of the background layer in obstacle avoidance decision-making is suppressed.

[0097] The exponential decay model is used as a risk constraint rule in this method, with the decay coefficient serving as the background layer confidence penalty field, stored in the risk assignment cache. The initial confidence of the background detection layer voxel mesh is derived from the radar point cloud alignment state, echo intensity state, and voxel persistence state. If the background detection layer voxels lack continuous point cloud support, the initial confidence enters a low confidence state; if the surface layer confidence score is in an unconfirmed state, the background layer confidence score is marked as a restricted call state, used only for dynamic target isolation trajectory maintenance, and not for weakening high-risk forbidden boundaries.

[0098] S412: Collect the UAV's size radius, current flight speed vector, end-to-end communication delay, and maximum braking distance contained in the UAV hardware bounding box. Calculate the communication delay displacement by multiplying the current flight speed vector and the end-to-end communication delay. Calculate the sum of the lengths corresponding to the communication delay displacement, UAV size radius, and maximum braking distance to establish a dynamic safety buffer thickness. Call the surface barrier layer voxels, extract the mesh normal vector direction corresponding to the surface barrier layer voxels, and expand the barrier boundary range of the surface barrier layer voxels along the mesh normal vector direction according to the dynamic safety buffer thickness to generate a high-risk no-entry boundary. The UAV hardware bounding box is the airframe space occupancy boundary recorded in the flight control safety configuration, containing the airframe shape safety radius field and the control protection boundary field. The current flight speed vector comes from the flight control status interface, the end-to-end communication delay comes from the control link status record, and the maximum braking distance comes from the flight control braking capability configuration.

[0099] The dynamic safety buffer thickness is a spatial extension field formed by integrating airframe occupancy, communication link response, and braking status. It is stored in the risk boundary cache and updated according to flight speed, control link status, and braking capability configuration. The mesh normal vector direction is provided by the local spatial plane model associated with the surface barrier layer voxels; when the local spatial plane model is in a state of pending confirmation, the conservative normal direction facing the UAV from the medium conflict region is used as the extension direction. The extended high-risk restricted area boundary is written into the risk layer nodes of the 3D hierarchical octree map and provides a restricted area index for path retrieval to S5.

[0100] S413: Extract high-risk forbidden boundaries, surface layer credibility scores, and background layer credibility scores as semantic attribute labels, assign these labels to the 3D hierarchical octree map, and establish an environment mapping map with risk gradient attributes. The semantic attribute labels are risk description fields attached to the octree nodes, including hierarchical identifier, forbidden boundary status, surface layer credibility status, background layer credibility status, update time status, and anomaly source identifier. The environment mapping map with risk gradient attributes is the environment representation result after adding risk attributes to the 3D hierarchical octree map, used for path retrieval and cost replacement.

[0101] During the assignment process, foreground layer nodes receive high-risk forbidden boundaries and surface layer confidence scores, while background layer nodes receive background layer confidence scores and penetration misleading conflict flags. When an attribute write fails, the existing occupying node is not deleted, and the corresponding node is marked as a risk attribute awaiting refresh. The map cache uses periodic version management; the current period's mapping map is validated and replaced with the previous valid version. If validation fails, the previous valid version continues to be used, and the current period's anomaly is recorded for subsequent sensor status traceability.

[0102] S5: Extract temporal features from visual camera video images to establish temporal confidence of the surface barrier layer. Generate isolated dynamic target trajectories based on voxels in the background detection layer. Establish an elevation risk fusion raster map by replacing the environment mapping map with risk gradient attributes based on the high-risk prohibited boundary, the temporal confidence of the surface barrier layer, and the cost of replacing the isolated dynamic target trajectory with the environment mapping map with risk gradient attributes. Retrieve the elevation risk fusion raster map to generate a collision-free obstacle avoidance flight path. The temporal confidence of the surface barrier layer is the cumulative state field of whether the surface barrier layer exists stably in consecutive image frames. The isolated dynamic target trajectory is the clustered trajectory of the moving point cloud isolated by the surface barrier layer in the background detection layer. The elevation risk fusion raster map is the path search input after converting the 3D risk mapping map into a flight altitude-related raster cost.

[0103] S511: Window frame corners and image stains / patches in the visual camera video images are extracted as static surface features. The relative displacement vectors between these static surface features in consecutive visual camera video images are calculated using optical flow. The trajectory deviation between the relative displacement vectors and the UAV pose data used to estimate the expected projection trajectory is calculated. When the trajectory deviation is less than a deviation threshold, the surface barrier layer voxels are determined to have temporal continuity. The surface layer confidence score is then accumulated to establish the temporal confidence score of the surface barrier layer. Static surface features are corners, patches, or local texture regions with fixed surface attachment relationships in consecutive visual images, carrying pixel positions, feature types, tracking states, and associated voxel indices. In this method, optical flow is used to establish the displacement correspondence of static surface features between adjacent image frames. The expected projection trajectory is jointly estimated from the UAV pose data, camera intrinsic parameter matrix, and the voxel positions of the surface barrier layer in the previous cycle.

[0104] When establishing the temporal confidence of the surface barrier layer, the historical temporal confidence recorded in the previous image frame is first read. If there is no historical temporal confidence at the moment of device startup or in the first effective period, the surface layer confidence score is used as the initial temporal source; if there are insufficient consecutive frames, the tracking state of the already obtained effective frames is used to supplement it, and a historical insufficiency flag is written. The temporal smoothing transition coefficient comes from the state transition probability field of the static surface features remaining stable in the historical test data and is stored in the visual temporal rule cache. After smoothing and transitioning, the historical temporal confidence is merged with the current surface layer confidence score to form the initial temporal confidence, which is then compared with the confidence saturation upper limit threshold corresponding to the obstacle occupancy probability safety boundary field preset in the global map file. When the saturation boundary is exceeded, the surface barrier layer temporal confidence remains in the safety boundary state; when the saturation boundary is not exceeded, the current initial temporal confidence is used as the output. The output result is written to the temporal confidence cache and used by the S513 replacement value.

[0105] S512: Based on the Euclidean clustering algorithm, the spatial distance clustering degree of the point cloud contained in the voxels of the background detection layer is calculated. Point clouds with spatial distance clustering degrees less than the distance clustering threshold are merged to establish cluster centroid nodes. The 3D position and 3D velocity of the cluster contained in the cluster centroid nodes are extracted as dynamic state tracking vectors. The grid coordinate boundaries contained in the voxels of the surface barrier layer are extracted to establish an isolation mask matrix. The covariance update range of the dynamic state tracking vector is limited by a Kalman filter combined with the isolation mask matrix to generate the trajectory of the isolated dynamic target. In this method, the Euclidean clustering algorithm is used to form point cloud clusters in the voxels of the background detection layer according to spatial proximity. The distance clustering threshold field is derived from the radar angular resolution, voxel scale configuration, and current detection range category and is stored in the background clustering rule cache. The cluster centroid node carries the cluster spatial position, cluster velocity state, point cloud source index, and trajectory tracking identifier.

[0106] The isolation mask matrix is ​​a spatially constrained data structure generated from the voxel grid coordinate boundaries of the surface barrier layer. It is used to mark which background dynamic point clouds are behind the surface barrier layer and which regions must not directly affect the foreground no-entry boundary. In this method, the Kalman filter is used to maintain the dynamic state tracking vector of the cluster centroid nodes, which includes the cluster's 3D position and 3D velocity. The covariance update range is limited by the isolation mask matrix; dynamic trajectories behind the surface barrier layer are written into the isolated dynamic target trajectory cache, without directly weakening the no-entry property of the surface barrier layer. If a background point cloud cluster disappears within a continuous period, the trajectory enters a pending deletion state; if it reappears and satisfies the spatial continuity rule, the original trajectory label is restored.

[0107] S513: Collect the UAV approach velocity, calculate the relative distance between the surface barrier layer voxels and the corresponding grids of the UAV's current position, and calculate the ratio between the grid relative distance and the UAV approach velocity to obtain the relative collision prediction time. When the relative collision prediction time is lower than the emergency obstacle avoidance threshold, combine the surface barrier layer temporal confidence to extract the coverage areas of high-risk prohibited boundaries and isolated dynamic target trajectories in the environment mapping map with risk gradient attributes. Replace the cost value of the coverage area with the penalty constant value to establish an elevation risk fusion grid map. Use an optimization algorithm to retrieve available grids corresponding to non-obstacle labels in the elevation risk fusion grid map, calculate the minimum detour curvature between the lines connecting available grids, and generate a collision-free obstacle avoidance flight path. The UAV approach velocity is generated by the flight control status interface based on the current motion direction and the relative position of the surface barrier layer voxels. The grid relative distance comes from the spatial relationship between the UAV's current position node and the surface barrier layer voxel node in the environment mapping map. The relative collision prediction time is a risk trigger status field, which is compared with the emergency obstacle avoidance threshold field after entering the emergency obstacle avoidance rule cache.

[0108] The emergency obstacle avoidance threshold field is derived from the flight control safety strategy, dynamic safety buffer thickness, and control link status. When an emergency obstacle avoidance state is reached, the high-risk prohibited boundary coverage area is assigned a penalty constant value, the isolated dynamic target trajectory coverage area is written with a background dynamic risk identifier according to the isolation status, and areas with stable temporal confidence of the surface blocking layer retain the foreground prohibited attribute. The elevation risk fusion grid map uses flight altitude layer, horizontal grid position, obstacle label, penalty value, and source risk fields as data structures. During path retrieval, only available grids corresponding to non-obstacle labels are accessed. The optimization algorithm reads the curvature constraints of available grid connections, UAV dynamics constraints, and prohibited boundary indices, and outputs a collision-free obstacle avoidance flight path composed of waypoint order, heading constraints, risk detour status, and valid path identifiers. Before the path is output, the flight control interface confirms the control command. If the confirmation fails, the previous valid obstacle avoidance path is retained and the current risk map is not cleared.

[0109] The UAV obstacle avoidance system based on multi-sensor fusion operates in conjunction with the aforementioned method within the same embodiment. The system includes a multi-source perception access module, a cross-domain consistency identification module, a spatial structure analysis module, a security situation assignment module, and a continuous trajectory decision module. The multi-source perception access module handles the access of radar point clouds, visual images, and UAV pose in S1, correspondingly outputting virtual point cloud frames, sparse depth maps, point cloud geometric curvature, and surface feature confidence maps. The cross-domain consistency identification module handles candidate region division, depth anomaly state generation, echo missing state generation, and penetration misleading conflict marker mesh establishment in S2, and transmits the medium conflict region to the spatial structure analysis module. The spatial structure analysis module handles local spatial plane model fitting, surface blocking layer voxel generation, background detection layer voxel extraction, and 3D layered octree map establishment in S3. The security situation assignment module handles the writing of confidence scores, dynamic safety buffer thickness, high-risk forbidden boundaries, and risk gradient attribute environment mapping maps in S4. The continuous trajectory decision module carries out the temporal confidence of the surface barrier layer, the trajectory of the isolated dynamic target, the elevation risk fusion grid map, and the generation of collision-free obstacle avoidance flight path in S5.

[0110] The modules are connected via data version number, spatial coordinate reference, and risk status identifier within the same flight cycle. When the data version output by the multi-source sensing access module fails integrity verification, the cross-domain consistency identification module receives the pending confirmation status and enters conservative conflict review; the spatial structure analysis module receives the media conflict area and the penetration misleading conflict identifier grid, and only processes virtual point cloud frames under the same version; the safety situation assignment module only writes risk attributes to voxel nodes that have passed spatial boundary verification; the continuous trajectory decision module only calls the environment mapping map that has passed version verification to generate the flight path. The system operation log records the data acquisition source, rule version, anomaly identifier, map version, and path confirmation status. Access permissions are managed separately by the flight control link and maintenance interface. During tracing, the corresponding records are read according to the flight cycle, sensor source, and risk node index.

[0111] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for obstacle avoidance in unmanned aerial vehicles (UAVs) based on multi-sensor fusion, characterized in that, Includes the following steps: S1: Collect LiDAR point cloud data, visual camera video images, and UAV pose data. Use the UAV pose data to align the LiDAR point cloud data to obtain a virtual point cloud frame. Use the virtual point cloud frame and the visual camera video image to perform coordinate transformation and projection association to obtain a sparse depth map. Calculate the point cloud geometric curvature based on the virtual point cloud frame. Extract the surface feature confidence map based on the visual camera video image. S2: Generate visual surface candidate regions based on the surface feature confidence map and sparse depth map. Combine point cloud geometric curvature, UAV flight speed, and depth anomaly gradient and echo missing rate calculated based on sparse depth map and virtual point cloud frame to calculate comprehensive conflict factor. Identify medium conflict regions based on the comprehensive conflict factor and establish a penetration misleading conflict identification grid. S3: Generate a local spatial planar model based on the medium conflict area and the UAV pose data, generate surface blocking layer voxels from the surface feature confidence map, and extract background detection layer voxels from the virtual point cloud frame by combining the penetration misleading conflict identification grid, and merge the surface blocking layer voxels and the background detection layer voxels to establish a three-dimensional layered octree map. S4: Calculate the surface layer confidence score based on the local spatial plane model and the surface feature confidence map, calculate the background layer confidence score based on the background detection layer voxels, generate a high-risk no-entry boundary from the surface barrier layer voxels, and assign the high-risk no-entry boundary, the surface layer confidence score and the background layer confidence score to the three-dimensional layered octree map to establish an environment mapping map with risk gradient attributes; S5: Extract the temporal features of the visual camera video image to establish the temporal confidence of the surface barrier layer, and generate the trajectory of the isolated dynamic target based on the voxels of the background detection layer. Establish an elevation risk fusion grid map by replacing the environment mapping map with risk gradient attributes according to the high-risk no-entry boundary, the temporal confidence of the surface barrier layer and the trajectory of the isolated dynamic target. Retrieve the elevation risk fusion grid map to generate a collision-free obstacle avoidance flight path.

2. The UAV obstacle avoidance method based on multi-sensor fusion according to claim 1, characterized in that, The specific steps of S1 are as follows: S111: Collect LiDAR point cloud data, visual camera video images, and UAV pose data; extract the UAV pose data as the pose smoothing node of the quaternion spherical linear interpolation algorithm; calculate the virtual pose deviation corresponding to the exposure time of the LiDAR point cloud data and the visual camera video image based on the quaternion spherical linear interpolation algorithm; adjust the spatial position of the LiDAR point cloud data according to the virtual pose deviation; and perform pose alignment to obtain a virtual point cloud frame. S112: Obtain the rotation matrix, translation vector, and camera intrinsic matrix from the joint extrinsic calibration file of the LiDAR and vision camera. Use the rotation matrix and translation vector to calculate the spatial coordinate transformation result of the virtual point cloud frame in the camera coordinate system. Use the spatial coordinate transformation result to unify the spatial reference benchmark of the LiDAR and the camera. Use principal component analysis to calculate the geometric curvature of the point cloud in the local spherical neighborhood of the point cloud. Calculate the projection coordinate relationship between the three-dimensional depth points contained in the virtual point cloud frame and the two-dimensional color pixels contained in the video image of the vision camera according to the camera intrinsic matrix, thereby establishing a sparse depth map. S113: Call the video image from the visual camera to perform image color space conversion operation, extract the highlight area of ​​the brightness channel and the high-frequency linear edge of the image, calculate the continuous surface texture contained in the video image from the visual camera based on the gray-level co-occurrence matrix, and generate a surface feature confidence map by calculating the fusion feature map of the highlight area of ​​the brightness channel, the high-frequency linear edge of the image and the continuous surface texture through a weighted algorithm.

3. The UAV obstacle avoidance method based on multi-sensor fusion according to claim 2, characterized in that, The specific steps of S2 are as follows: S211: Divide the spatial grid of the surface feature confidence map according to the multi-scale grid partitioning strategy corresponding to the image pyramid to generate local grid regions, obtain the global illumination intensity measured by the illumination sensing hardware, calculate the arithmetic mean of the confidence of the pixels covered in the local grid region to obtain the surface feature confidence mean, adjust the visual surface presence threshold according to the global illumination intensity, calculate the difference between the surface feature confidence mean and the visual surface presence threshold, and filter the local grid regions whose difference is greater than the preset difference filtering threshold to generate visual surface candidate regions; S212: Calculate the effective echo rate by the ratio between the number of effective depth points and the total number of pixels in the grid within the coverage area of ​​the candidate region of the sparse depth map. When the effective echo rate is higher than a preset echo judgment threshold, extract the average depth value corresponding to the projected point cloud of the sparse depth map in the candidate region of the visual surface. Extract the internal depth data corresponding to the adjacent grids outside the local grid region of the sparse depth map. Calculate the reference depth mean and depth standard deviation based on the internal depth data. Calculate the depth difference between the average depth value and the reference depth mean. Calculate the depth anomaly gradient by the ratio between the depth difference and the depth standard deviation. Call the point cloud geometric curvature. When the point cloud geometric curvature is lower than a preset curvature judgment threshold, calculate the echo difference between the value 1 and the effective echo rate to obtain the echo missing rate. S213: Collect the UAV flight speed, assign depth anomaly weight coefficient and echo missing weight coefficient based on the UAV flight speed, calculate the anomaly product value based on the depth anomaly weight coefficient and the depth anomaly gradient, calculate the missing product value based on the echo missing weight coefficient and the echo missing rate, calculate the sum of the anomaly product value and the missing product value to obtain the comprehensive conflict factor, filter the regions where the comprehensive conflict factor is greater than the preset conflict alarm threshold to generate media conflict regions, and for the media conflict regions, when the depth anomaly gradient is greater than the preset anomaly gradient threshold, mark the media conflict regions and establish a penetration misleading conflict identification grid.

4. The UAV obstacle avoidance method based on multi-sensor fusion according to claim 3, characterized in that, The process of adjusting the visual surface presence threshold based on the global illumination intensity specifically includes: Obtain the baseline lighting environment parameters recorded in the factory calibration file of the lighting sensing hardware; Extract the standard illuminance value and initial response scaling factor from the reference lighting environment parameters; Calculate the environmental illuminance deviation between the global illumination intensity and the standard illuminance value; A smooth attenuation correspondence dataset generated by fitting multiple sets of surface feature confidence fluctuation data under known test lighting conditions is used to set up a nonlinear lighting compensation function mapping table. The dynamic compensation coefficient corresponding to the environmental illuminance deviation value is retrieved based on the nonlinear illumination compensation function mapping table. The illumination adjustment control weight is obtained by calculating the numerical product between the dynamic compensation coefficient and the initial response scaling factor. Under ideal diffuse reflection conditions without ambient light interference, the mean confidence score of surface features of statistical surface feature samples is used as the static surface confidence benchmark value. The visual surface presence threshold is obtained by calculating the product between the illumination adjustment control weight and the static surface confidence benchmark value.

5. The UAV obstacle avoidance method based on multi-sensor fusion according to claim 3, characterized in that, The specific steps of S3 are as follows: S311: Extract the coordinate boundary of the medium conflict region in the three-dimensional global coordinate system, divide the surface blocking layer and the background detection layer according to the coordinate boundary, call the penetration misleading conflict marker grid, the virtual point cloud frame and the reflection intensity value, extract the reflection intensity value of the outer boundary of the medium conflict region that does not have the penetration misleading conflict marker grid in the virtual point cloud frame, extract the point cloud corresponding to the reflection intensity value that is greater than or equal to the point cloud reflection threshold, and obtain the edge point cloud. S312: Extract the UAV yaw angle from the UAV pose data, obtain the prior building contours attached to the global map, extract the UAV yaw angle and the prior building contours as plane normal vector constraints for the random sampling consensus algorithm, calculate the plane normal vector, plane intercept and plane fitting inlier rate of the edge point cloud in three-dimensional space based on the random sampling consensus algorithm, fit and generate a local space plane model, call the camera intrinsic parameter matrix and the surface feature confidence map, calculate the back projection intersection points of the grid pixels contained in the surface feature confidence map along the camera optical center ray according to the camera intrinsic parameter matrix, project the back projection intersection points to the surface of the local space plane model, and establish a surface blocking layer voxel; S313: Extract the three-dimensional depth position data contained in the virtual point cloud frame with the coverage area of ​​the mesh with the penetration misleading conflict identifier, retain the three-dimensional depth position data and convert it to generate a background detection layer voxel, call the surface blocking layer voxel, and merge the spatial coordinates of the surface blocking layer voxel and the background detection layer voxel in the same line of sight to construct a three-dimensional layered octree map.

6. The UAV obstacle avoidance method based on multi-sensor fusion according to claim 5, characterized in that, The process of extracting the point cloud corresponding to the reflection intensity value that is greater than or equal to the point cloud reflection threshold to obtain the edge point cloud specifically includes: Calculate the mean and standard deviation of the reflection intensity values; The numerical proportional relationship between the width of the known entity's reflective surface echo intensity distribution interval and the mean offset is used to set the weight for adjusting the reflection intensity. The dynamic reflection compensation value is obtained by multiplying the standard deviation of the reflection intensity and the adjustment weight of the reflection intensity. The point cloud reflection threshold is obtained by summing the mean reflection intensity and the dynamic reflection compensation value. Compare the numerical values ​​between the reflection intensity value and the point cloud reflection threshold; Extract the point clouds corresponding to the reflection intensity values ​​that are greater than or equal to the point cloud reflection threshold to generate a set of high-reflectivity candidate point clouds; Calculate the Euclidean space distance between point cloud data in the high-reflectivity candidate point cloud set; The connectivity distance threshold is set by calculating the arc length product between the radar hardware scanning angular resolution and the current detection space distance. The edge point cloud is obtained by performing a spatial aggregation operation on the point cloud data whose Euclidean spatial distance is less than the connectivity distance threshold.

7. The UAV obstacle avoidance method based on multi-sensor fusion according to claim 5, characterized in that, The specific steps of S4 are as follows: S411: Extract the in-plane fitting rate of the local spatial plane model, calculate the product between the mean confidence of the surface features and the in-plane fitting rate to obtain the surface layer confidence score, extract the surface layer confidence score as the negative exponential part of the exponential decay model, calculate the negative exponential power of the natural logarithm base based on the exponential decay model to obtain the decay coefficient, call the background detection layer voxel, obtain the initial confidence of the background detection layer voxel grid corresponding to the background detection layer voxel, and calculate the penalty product result based on the decay coefficient and the initial confidence of the background detection layer voxel grid to generate the background layer confidence score; S412: Collect the UAV size radius, current flight speed vector, end-to-end communication delay time, and maximum braking distance contained in the UAV hardware bounding box; calculate the communication delay displacement by multiplying the current flight speed vector and the end-to-end communication delay time; calculate the sum of the lengths corresponding to the communication delay displacement, the UAV size radius, and the maximum braking distance to establish a dynamic safety buffer thickness; call the surface barrier layer voxel; extract the mesh normal vector direction corresponding to the surface barrier layer voxel; expand the barrier boundary range of the surface barrier layer voxel along the mesh normal vector direction according to the dynamic safety buffer thickness to generate a high-risk no-entry boundary. S413: Extract the high-risk forbidden boundary, the surface layer credibility score, and the background layer credibility score as semantic attribute labels, assign the semantic attribute labels to the three-dimensional layered octree map, and establish an environment mapping map with risk gradient attributes.

8. The UAV obstacle avoidance method based on multi-sensor fusion according to claim 7, characterized in that, The specific steps of S5 are as follows: S511: Extract the window frame corner points and image stains contained in the visual camera video image as static surface features, calculate the relative displacement vector between each static surface feature in the continuous visual camera video images using optical flow method, calculate the trajectory deviation between the relative displacement vector between the images and the expected projection trajectory calculated from the UAV pose data, determine that the surface barrier layer voxel has temporal continuity when the trajectory deviation is less than the deviation judgment threshold, and accumulate the surface layer confidence score to establish the temporal confidence of the surface barrier layer; S512: Calculate the spatial distance clustering degree of the point cloud contained in the voxels of the background detection layer based on the Euclidean clustering algorithm, merge the point clouds whose spatial distance clustering degree is less than the distance clustering threshold to establish cluster centroid nodes, extract the cluster 3D position and cluster 3D velocity contained in the cluster centroid nodes as dynamic state tracking vectors, extract the grid coordinate boundary contained in the voxels of the surface barrier layer to establish an isolation mask matrix, and use a Kalman filter combined with the isolation mask matrix to limit the covariance update range of the dynamic state tracking vector to generate the trajectory of the isolated dynamic target; S513: Collect the approach velocity of the UAV, calculate the relative distance between the surface barrier layer voxel and the grid corresponding to the current position of the UAV, calculate the ratio between the grid relative distance and the UAV approach velocity to obtain the relative collision prediction time, when the relative collision prediction time is lower than the emergency obstacle avoidance threshold, combine the temporal confidence of the surface barrier layer, extract the coverage area of ​​the high-risk prohibited boundary and the trajectory of the isolated dynamic target in the environment mapping map with risk gradient attributes, replace the cost value of the coverage area with the penalty constant generation value to establish an elevation risk fusion grid map, search for available grids corresponding to non-obstacle labels in the elevation risk fusion grid map through the optimization algorithm, calculate the minimum detour curvature between the lines connecting the available grids, and generate a collision-free obstacle avoidance flight path.

9. The UAV obstacle avoidance method based on multi-sensor fusion according to claim 8, characterized in that, The process of accumulating the surface layer confidence score to establish the time-series confidence of the surface barrier layer specifically includes: Extract the historical time-series confidence score of the previous image frame data record; In statistical historical test data, the probability of stable static surface features between consecutive frames is used to set a time-series smoothing transition coefficient; The historical transit confidence level is obtained by calculating the product between the historical time series confidence level and the time series smoothing transition coefficient. The initial time-series confidence level is obtained by summing the historical transmission confidence level and the surface layer confidence score. Extract the pre-defined obstacle occupancy probability safety boundary value from the global map file as the confidence saturation upper limit threshold; Compare the initial time-series confidence level with the upper limit of the confidence level saturation threshold. When the initial time series confidence level is greater than the upper limit threshold of the confidence level saturation, the upper limit threshold of the confidence level saturation is extracted as the time series confidence level of the surface barrier layer; When the initial time-series confidence level is less than or equal to the confidence level saturation upper limit threshold, the initial time-series confidence level is extracted as the time-series confidence level of the surface barrier layer.

10. A drone obstacle avoidance system based on multi-sensor fusion, characterized in that, The system is used to implement the UAV obstacle avoidance method based on multi-sensor fusion as described in any one of claims 1-9, and the system includes: The multi-source sensing access module collects LiDAR point cloud data, visual camera video images, and UAV pose data. It uses the UAV pose data to align the LiDAR point cloud data to obtain a virtual point cloud frame. It uses the virtual point cloud frame and the visual camera video image to perform coordinate transformation and projection association to obtain a sparse depth map. It calculates the point cloud geometric curvature based on the virtual point cloud frame and extracts the surface feature confidence map based on the visual camera video image. The cross-domain consistency identification module generates visual surface candidate regions based on the surface feature confidence map and sparse depth map. It combines the point cloud geometric curvature, UAV flight speed, and depth anomaly gradient and echo missing rate calculated based on the sparse depth map and virtual point cloud frame to calculate a comprehensive conflict factor. Based on the comprehensive conflict factor, it identifies medium conflict regions and establishes a penetration misleading conflict identification grid. The spatial structure analysis module generates a local spatial planar model based on the medium conflict area and the UAV pose data. It generates surface blocking layer voxels from the surface feature confidence map and extracts background detection layer voxels from the virtual point cloud frame by combining the penetration misleading conflict identification grid. It merges the surface blocking layer voxels and the background detection layer voxels to establish a three-dimensional layered octree map. The security situation assignment module calculates the surface layer confidence score based on the local spatial plane model and the surface feature confidence map, calculates the background layer confidence score based on the background detection layer voxels, generates a high-risk no-entry boundary from the surface barrier layer voxels, and assigns the high-risk no-entry boundary, the surface layer confidence score, and the background layer confidence score to the three-dimensional layered octree map to establish an environment mapping map with risk gradient attributes. The continuous trajectory decision module extracts the temporal features of the visual camera video images to establish the temporal confidence of the surface barrier layer, and generates the trajectory of the isolated dynamic target based on the voxels of the background detection layer. It establishes an elevation risk fusion grid map by replacing the environment mapping map with risk gradient attributes according to the high-risk no-entry boundary, the temporal confidence of the surface barrier layer, and the cost of the isolated dynamic target trajectory. It then retrieves the elevation risk fusion grid map to generate a collision-free obstacle avoidance flight path.

Images