A Dynamic Obstacle Avoidance Method for Unmanned Aerial Vehicles Based on Multi-Sensor Fusion

Abstract

This invention relates to the field of unmanned aerial vehicle (UAV) control technology, specifically to a dynamic obstacle avoidance method for UAVs based on multi-sensor fusion. The method includes receiving image sequences, depth point clouds, pose parameters, and flight velocity parameters; calculating illumination distortion gradient values ​​and speckle noise entropy values ​​based on the image sequences and depth point clouds; performing coordinate registration and feature extraction on the image sequences, depth point clouds, and pose parameters to output visual and radar feature sets; generating weighted visual and radar feature sets through a weight adjustment function, and generating a joint heterogeneous feature tensor through alignment and concatenation; and generating airframe attitude and thrust control commands by applying the joint heterogeneous feature tensor to a probabilistic prediction model and a model predictive control algorithm. This invention avoids error accumulation and decision delays caused by multi-level serial computations through cross-domain reuse and deep coupling of airframe physical parameters in the data stream, thus improving the global executability of obstacle avoidance tasks.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, specifically to a dynamic obstacle avoidance method for UAVs based on multi-sensor fusion. Background Technology

[0002] Autonomous navigation and dynamic obstacle avoidance for drones in complex physical environments (such as urban low-rise building clusters or mountainous and forested areas) is an important technological application area. Existing technologies generally employ multimodal data fusion methods combining visual cameras and radar sensors. This involves monitoring changes in the three-dimensional physical contours of the external environment, and when encountering potential spatial obstacles, automatically extracting data features from heterogeneous sensors and planning spatial avoidance routes within a pre-defined environmental perception computing framework. This is currently the mainstream technology for enabling drones to fly in complex environments.

[0003] However, existing technical solutions have significant shortcomings in perception robustness and spatial reconstruction continuity. Their weight allocation mechanisms for heterogeneous data fusion suffer from staticity and environmental decoupling. Existing fusion frameworks are mostly based on fixed prior confidence levels or historical observation noise covariance, and their fusion weights cannot be adjusted nonlinearly with real-time environmental changes. This makes it difficult to match extreme perception areas that require degradation compensation (such as backlight blind zones and radar speckle noise zones), resulting in severe perception biases in the final synthesized spatial features. Furthermore, when processing heterogeneous perspectives, only coarse two-dimensional stitching is performed, which cannot achieve rigorous reconstruction of physical space. It cannot eliminate the geometric scale misalignment between perspective and bird's-eye view, nor can it restore the true three-dimensional voxel height, causing the prediction of obstacle's three-dimensional volume and spatial occupancy probability to completely fail. In addition, existing obstacle avoidance technologies cannot eliminate multimodal perception errors in complex environments and output control strategies that conform to the physical motion constraints of the aircraft, resulting in a lack of executability of the final flight trajectory.

[0004] To address this, a dynamic obstacle avoidance method for unmanned aerial vehicles (UAVs) based on multi-sensor fusion is proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a dynamic obstacle avoidance method for unmanned aerial vehicles (UAVs) based on multi-sensor fusion, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a dynamic obstacle avoidance method for unmanned aerial vehicles based on multi-sensor fusion, comprising:

[0007] The system receives image sequences and depth point clouds acquired by airborne sensors, and pose and flight speed parameters acquired by the airborne measurement unit. Based on the image sequences, it calculates the illumination distortion gradient value through contrast attenuation. Based on the depth point clouds, it calculates the speckle noise entropy value through local point cloud density.

[0008] The image sequence, depth point cloud, and pose state parameters are coordinate registered to output the registered image sequence and registration point cloud. The visual feature set and radar feature set are then output through a feature extraction network.

[0009] The illumination distortion gradient value, speckle noise entropy value, and flight speed parameter are input into the weight adjustment function, and the visual weight coefficient and radar weight coefficient are output. The visual weight coefficient and radar weight coefficient are multiplied with the visual feature set and radar feature set respectively to generate a weighted visual feature set and a weighted radar feature set. A joint heterogeneous feature tensor is generated by alignment and splicing.

[0010] The joint heterogeneous feature tensor is input into the probabilistic prediction model, which outputs a spatial probability occupancy map. This map, along with the pose state parameters and flight velocity parameters, is then input into the model predictive control algorithm to generate airframe attitude and thrust control commands, which are transmitted to the underlying flight controller to execute physical obstacle avoidance.

[0011] Preferably, the specific generation process of the illumination distortion gradient value and speckle noise entropy value includes: the image sequence uses an illumination intensity extractor to output an ambient brightness distribution matrix, and uses a low-pass smoothing filter to filter out high-frequency texture details to generate a global illumination component matrix; the global illumination component matrix separates the reflection background using an illumination reflection decoupling operator to output an ambient illumination distribution map; the ambient illumination distribution map uses a second-order Laplacian differentiator to extract brightness abrupt change boundaries to output an illumination abrupt change feature set; the illumination abrupt change feature set uses an extremum calculation function to obtain local regional maxima to generate an extremum feature map, and uses a linear normalization operator to compress the numerical space to output the illumination distortion gradient value;

[0012] The depth point cloud set is downsampled to output a sparse point cloud tensor. The sparse point cloud tensor is then processed by a Euclidean clusterer to calculate spatial distance, resulting in a point cloud cluster set. The point cloud cluster set is then used to calculate the number of points per unit volume using a local density estimation operator, resulting in a point cloud density distribution matrix. The point cloud density distribution matrix is ​​then processed by an outlier detector to extract the spatial coordinates of density anomalies, resulting in an outlier speckle coordinate set. Based on the Shannon entropy formula, the outlier speckle coordinate set is subjected to logarithmic probability distribution calculation within a preset discrete spatial grid to generate a speckle information entropy matrix. The speckle information entropy matrix is ​​then processed through global convergence calculation to extract a global speckle information entropy scalar. The global speckle information entropy scalar is input into a smoothing mapping function to output speckle noise entropy values.

[0013] Preferably, the specific generation process of the registered image sequence and registration point cloud includes: clock matching between the image sequence and the depth point cloud to output a synchronized image sequence and a synchronized point cloud; temporal optimization of the pose state parameters based on the timestamp of the synchronized image sequence to output a synchronized reference pose matrix; inputting the synchronized reference pose matrix into a quaternion analyzer to solve the transient attitude fluctuations of the body, outputting a rotational perturbation matrix, and fusing it with the fixed installation offset of the heterogeneous sensor through a coordinate transformer to generate an extrinsic parameter transformation matrix; inputting the synchronized image sequence into a camera calibration model to perform optical distortion correction, outputting a distortion-free image sequence; obtaining a pre-calibrated intrinsic parameter feature matrix, combining it with the synchronized point cloud, the extrinsic parameter transformation matrix, and the distortion-free image sequence into a spatial reprojection model for mapping transformation, extracting the common-view frustum region, and outputting the registered image sequence and registration point cloud.

[0014] Preferably, the specific generation process of the visual feature set and radar feature set includes: inputting the registered image sequence into a multi-scale residual convolutional network to extract multi-level spatial topological information and outputting a visual semantic feature tensor; calculating pixel-level weights through a spatial attention layer to output a visual feature set; inputting the registered point cloud into a spatial bounding box generator for recursive partitioning to output a three-dimensional subspace array, and extracting local geometric topological information through a sparse three-dimensional convolutional kernel to output a three-dimensional geometric feature matrix; inputting the three-dimensional geometric feature matrix into a bird's-eye view projection operator to perform height-dimensional orthogonal compression transformation and output a radar feature set.

[0015] Preferably, the specific generation process of the visual weight coefficient and radar weight coefficient includes: inputting the illumination distortion gradient value into a one-dimensional convolutional layer and outputting a smooth gradient vector; generating a standard entropy vector from the speckle noise entropy value through a constraint operator; filtering out physical jitter from the flight speed parameter and outputting a smooth speed vector; merging the smooth gradient vector, standard entropy vector, and smooth speed vector into a splicing layer and outputting a state matrix; inputting the state matrix into a fully connected network to extract nonlinear mapping relationships and outputting a hidden layer state vector; and performing scalar mapping on the hidden layer state vector through a normalization operator to output the visual weight coefficient and radar weight coefficient.

[0016] Preferably, the specific generation process of the joint heterogeneous feature tensor includes: the visual feature set and visual weight coefficients are subjected to dimensional copying and element-wise weighted multiplication through a tensor broadcast multiplier to output a weighted visual feature set; the radar feature set and radar weight coefficients are input into a tensor broadcast multiplier for dimensional copying and element-wise weighted multiplication to output a weighted radar feature set; the weighted visual feature set is input into an inverse perspective mapping operator to perform perspective transformation from pixel space to physical overhead space to output a bird's-eye view visual feature tensor; the bird's-eye view visual feature tensor and the weighted radar feature set are subjected to resolution interpolation and coordinate system origin alignment through a spatial grid sampler to generate a pair of feature matrices of the same dimension; the pair of feature matrices of the same dimension are subjected to channel dimension superposition through a deep cascaded layer to output a spliced ​​feature volume; the spliced ​​feature volume is subjected to cross-channel nonlinear fusion extraction using a pointwise convolutional network to output a joint heterogeneous feature tensor.

[0017] Preferably, the specific generation process of the spatial probability occupancy map includes: performing feature channel dimensionality reduction on the joint heterogeneous feature tensor through a multilayer perceptron model to output two-dimensional compressed features; calculating the discrete height layer probability of the two-dimensional compressed features to output a height probability distribution vector set; reconstructing the two-dimensional compressed features and the height probability distribution vector set through orthogonal dimensions to output initial voxel features; expanding the spatial resolution of the initial voxel features using a three-dimensional deconvolution network model to output a three-dimensional upsampled tensor; inputting the three-dimensional upsampled tensor into a normalized exponential activation function to output a spatial probability distribution matrix; using the spatial probability distribution matrix to output an initial continuous occupancy map using a voxel occupancy grid generator; and performing diffusion smoothing expansion on the initial continuous occupancy map to output the spatial probability occupancy map.

[0018] Preferably, the specific generation process of the airframe attitude and thrust control commands includes: calculating the multirotor thrust boundary using flight speed parameters and attitude state parameters through dynamic equations, and outputting a feasible control input set; inputting the feasible control input set into the state transition model to output a three-dimensional spatial sampling node sequence; calculating the collision expectation risk using the three-dimensional spatial sampling node sequence and the spatial probability occupancy map through a Bayesian cost function to generate a continuous generation value matrix; iteratively solving the minimum cost sequence using a gradient descent optimization solver to output the optimal spatial avoidance trajectory; inputting the optimal spatial avoidance trajectory into an inverse dynamics solver to extract lateral acceleration and vertical acceleration components, and generating airframe attitude and thrust control commands; transmitting the airframe attitude and thrust control commands to the underlying flight controller to convert them into independent control signals for the quadcopter motors to execute physical obstacle avoidance actions.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0020] 1. Based on image sequences, the illumination distortion gradient is calculated through contrast attenuation; based on depth point clouds, the speckle noise entropy is calculated through local point cloud density; the illumination distortion gradient, speckle noise entropy, and flight velocity parameters are input into a weighting adjustment function, outputting visual and radar weight coefficients. Subsequently, the visual and radar weight coefficients are multiplied by tensors with the visual and radar feature sets, respectively, to generate weighted visual and radar feature sets. These steps, through the coupling weighting of environmental degradation factors and motion parameters, achieve objective quantification of environmental degradation states and, combined with aircraft dynamic parameters, complete feature-level pre-fusion weighting, reducing the interference weight of a single sensor in complex environments, improving the accuracy of heterogeneous data extraction, and providing a reliable and executable data foundation from the perception source for subsequent obstacle avoidance trajectory generation.

[0021] 2. The image sequence, depth point cloud, and pose state parameters are coordinate registered, outputting a registered image sequence and a registration point cloud. A feature extraction network is then used to output visual and radar feature sets. Based on this, a joint heterogeneous feature tensor is generated by stitching these features together. This joint heterogeneous feature tensor is then input into a probabilistic prediction model, outputting a spatial probability occupancy map. These steps, through the registration and prediction of heterogeneous features, avoid spatial misalignment and scale distortion caused by heterogeneous data conversion. Simultaneously, the prediction model constructs a continuous spatial probability occupancy map, objectively reproducing the three-dimensional distribution of the physical environment. This allows the planned obstacle avoidance route to more accurately match the real physical space, ensuring the executability of obstacle avoidance actions from a geometric perspective.

[0022] 3. The generated spatial probability occupancy map, along with the pose and flight velocity parameters, is synchronously input into the model predictive control algorithm to generate attitude and thrust control commands, which are then transmitted to the underlying flight controller to execute physical obstacle avoidance maneuvers. In the final control command generation stage, the above steps synchronously couple and solve the occupancy map representing environmental obstacles with the parameters representing the UAV's current motion state. This ensures that the output control commands are constrained by the actual dynamic constraints of the multi-rotor UAV, avoiding the planning of flight paths that exceed the aircraft's physical limits or are impossible to complete. This contributes to the dynamic safety and physical feasibility of the underlying aircraft when performing obstacle avoidance maneuvers.

[0023] 4. The system receives image sequences and depth point clouds from airborne sensors, as well as pose and flight velocity parameters from the airborne measurement unit. In the front-end computation, flight velocity parameters and environmental distortion features are input into a weighting function; in the middle stage, a spatial probability occupancy map is output; in the back-end solution, the spatial probability occupancy map, pose and flight velocity parameters are synchronously input into the algorithm. This breaks down the barriers between independent modules of perception, mapping, and control in traditional obstacle avoidance systems, enabling cross-domain reuse and deep coupling of the aircraft's physical parameters in the data stream. This end-to-end fusion architecture avoids error accumulation and decision delays caused by multi-level serial computation, ensuring that the final generated instructions are the optimal solution based on the collaborative deduction of the global environmental state and the aircraft's global capabilities. From the perspective of the overall system architecture, this contributes to the global executability of the obstacle avoidance task. Attached Figure Description

[0024] Figure 1 This is a flowchart of a dynamic obstacle avoidance method for unmanned aerial vehicles based on multi-sensor fusion proposed in this invention;

[0025] Figure 2 This is a flowchart of the dynamic weight generation and cross-modal feature fusion proposed in this invention;

[0026] Figure 3 This is a flowchart of the spatial probability prediction and underlying physical thrust control proposed in this invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Please see Figures 1-3 The present invention provides a dynamic obstacle avoidance method for unmanned aerial vehicles based on multi-sensor fusion, the specific steps of which are as follows:

[0029] The system receives image sequences and depth point clouds acquired by airborne sensors, and pose and flight speed parameters acquired by the airborne measurement unit. Based on the image sequences, it calculates the illumination distortion gradient value through contrast attenuation. Based on the depth point clouds, it calculates the speckle noise entropy value through local point cloud density.

[0030] The image sequence, depth point cloud, and pose state parameters are coordinate registered to output the registered image sequence and registration point cloud. The visual feature set and radar feature set are then output through a feature extraction network.

[0031] The illumination distortion gradient value, speckle noise entropy value, and flight speed parameter are input into the weight adjustment function, and the visual weight coefficient and radar weight coefficient are output. The visual weight coefficient and radar weight coefficient are multiplied with the visual feature set and radar feature set respectively to generate a weighted visual feature set and a weighted radar feature set. A joint heterogeneous feature tensor is generated by alignment and splicing.

[0032] The joint heterogeneous feature tensor is input into the probabilistic prediction model, which outputs a spatial probability occupancy map. This map, along with the pose state parameters and flight velocity parameters, is then input into the model predictive control algorithm to generate airframe attitude and thrust control commands, which are transmitted to the underlying flight controller to execute physical obstacle avoidance.

[0033] The technical solution of the present invention will be further described in detail below with reference to specific embodiments.

[0034] Example 1

[0035] This application discloses a dynamic obstacle avoidance method for unmanned aerial vehicles (UAVs) based on multi-sensor fusion. (See attached document.) Figure 1 The specific steps proposed in this invention include: S1, receiving image sequences and depth point clouds acquired by airborne sensors, and receiving pose state parameters and flight speed parameters acquired by airborne measurement units; calculating the illumination distortion gradient value based on the image sequence through contrast attenuation; calculating the speckle noise entropy value based on the depth point cloud through local point cloud density; S2, performing coordinate registration of the image sequence, depth point cloud, and pose state parameters, outputting the registered image sequence and registered point cloud, and outputting a visual feature set and a radar feature set through a feature extraction network; S3, processing the illumination distortion gradient value and speckle noise... The acoustic entropy value and flight speed parameter are input into the weight adjustment function, which outputs visual weight coefficients and radar weight coefficients. The visual weight coefficients and radar weight coefficients are multiplied by the visual feature set and radar feature set respectively to generate a weighted visual feature set and a weighted radar feature set. A joint heterogeneous feature tensor is generated by aligning and stitching. S4. The joint heterogeneous feature tensor is input into the probability prediction model, which outputs a spatial probability occupancy map. The map is then input into the model predictive control algorithm synchronously with the pose state parameter and flight speed parameter to generate airframe attitude and thrust control commands, which are transmitted to the underlying flight controller to perform physical obstacle avoidance.

[0036] Further, the system receives image sequences and depth point clouds acquired by airborne sensors, and pose and flight velocity parameters acquired by the airborne measurement unit; based on the image sequences, it calculates the illumination distortion gradient value through contrast attenuation; based on the depth point cloud, it calculates the speckle noise entropy value through local point cloud density; corresponding to step S1 above; the specific implementation process includes:

[0037] The image sequence utilizes an illumination intensity extractor to output an ambient brightness distribution matrix, and uses a low-pass smoothing filter to filter out high-frequency texture details, generating a global illumination component matrix. The global illumination component matrix separates the reflected background using an illumination reflection decoupling operator, outputting an ambient illumination distribution map. The ambient illumination distribution map uses a second-order Laplacian differentiator to extract brightness abrupt change boundaries, outputting an illumination abrupt change feature set. The illumination abrupt change feature set uses an extremum calculation function to obtain local regional maxima, generating an extremum feature map. The numerical space is compressed using a linear normalization operator, outputting the illumination distortion gradient value.

[0038] This implementation relies on a rotary-wing unmanned aerial vehicle (UAV) platform, equipped with a heterogeneous multi-sensor fusion sensing system and a high-performance edge computing unit. The multi-sensor fusion architecture employs a global shutter depth camera. The camera outputs raw image sequences with a resolution of 1920 x 1080 pixels, a sampling frequency of 30 frames per second, and a field of view of 87° horizontally and 58° vertically. The lidar sensing module uses a non-repetitive scanning solid-state lidar. Under standard daylight conditions of 100,000 lux, this lidar has a range of up to 90 meters (for targets with 10% reflectivity), a near-field blind zone of 0.05 meters, a circular field of view of 70.4°, and a ranging random error controlled within 2 cm. The UAV's attitude and flight speed parameters are provided by the onboard flight controller and the built-in microelectromechanical system (MEMS) inertial measurement unit, with a data sampling output frequency set to 500 Hz to ensure low latency in status feedback. The computing center uses a high-performance heterogeneous computing core board to perform high-performance tasks such as feature extraction, tensor calculation, and model predictive control.

[0039] Specifically, the generation process of the illumination distortion gradient value is as follows:

[0040] High-resolution raw data collected by airborne sensors is used as an image sequence. A light intensity extractor is used to perform spatial color conversion, mapping and converting the data of the three primary color spaces of red, green and blue to hue, saturation and lightness color spaces. The channel data representing lightness is separated out separately, and the ambient light distribution matrix is ​​output.

[0041] Subsequently, the ambient luminance distribution matrix is ​​filtered out for high-frequency texture details using a low-pass smoothing filter. In this embodiment, the low-pass smoothing filter employs a two-dimensional Gaussian smoothing kernel with a specific window size (e.g., 5×5). Specifically, the scalar value of each central pixel in the ambient luminance distribution matrix is ​​replaced by the weighted arithmetic mean of its surrounding neighboring pixel values. The weighting coefficients strictly follow a two-dimensional Gaussian normal distribution function, with the weight of pixels farther from the center decreasing exponentially. This generates the global illumination component matrix.

[0042] Based on the Retinex theory in visual perception, the natural logarithm of each pixel in the original ambient brightness distribution matrix is ​​subtracted from the natural logarithm of the corresponding pixel in the global illumination component matrix. This process extracts high-frequency details that contain only the inherent reflective properties of the object's surface and low-frequency information that reflects the distribution of external light source projection, thus outputting an ambient lighting distribution map.

[0043] In this embodiment, the local variance of the ambient lighting distribution map is calculated to generate compensation coefficient values, and nonlinear compensation is performed on overexposed or underexposed areas. Specifically, a 9×9 sliding calculation window is set for the ambient lighting distribution map, and the local mean and local variance of pixel brightness within the window are traversed and statistically analyzed. When the local mean is greater than a certain high percentage of the camera's maximum quantization level (e.g., greater than 78% of the upper limit of quantization level 255 in an 8-bit camera, i.e., approximately equal to 200) or less than a certain low percentage (e.g., less than 20% of the upper limit of quantization level), and the local variance is less than 10, the area is determined to be an overexposed blind spot or an underexposed dark area lacking detail. For the above-mentioned abnormal areas, pixel-level compensation is performed using a nonlinear correction function. The compensation coefficient value is dynamically calculated from the local mean, specifically: using 128 as a reference constant, dividing by the current local mean, and then taking the square root of the result to the power of 0.5. The calculated compensation coefficient value is used as an exponent to perform a power function operation on the normalized pixel values ​​of the ambient lighting distribution map. Finally, the result is multiplied by the maximum quantization level to restore the original numerical space, and this is used for updating. This process restores hidden textures under extreme lighting and shadows, preventing the missed detection of obstacles caused by overexposure or underexposure.

[0044] To further capture the boundaries of drastic changes in illumination in complex environments (such as the light-dark boundary when a drone passes through an underpass, tunnel, or encounters strong backlight), a second-order Laplace differentiator is used to perform spatial second-order derivative calculations on the ambient lighting distribution map. The mathematical logic is as follows: solve for the sum of the second-order partial derivatives of the two-dimensional distribution map in the horizontal and vertical directions. When the calculated result of the second derivative crosses a zero point (i.e., a zero cross occurs), it indicates that there is a drastic change in illumination intensity at that spatial location, thus outputting a set of illumination abrupt change feature sets.

[0045] The illumination abrupt change feature set is then used to find the local maxima through an extremum calculation function. The calculation process sets a sliding window with a specific step size (such as an 8×8 pixel area). In each spatial sliding traversal, the maximum absolute value of the derivative within the local window area is compared and selected, and this value is used as the unique representative value of the entire area to generate the dimensionality-reduced extremum feature map.

[0046] Finally, for each value in the extreme value feature map, subtract the global minimum value in the matrix from it, and then divide the difference by the difference between the global maximum and minimum values. This maps all values ​​representing photometric distortion to an absolute floating-point range of 0 to 1, outputting the illumination distortion gradient value.

[0047] The depth point cloud set is downsampled to output a sparse point cloud tensor. The sparse point cloud tensor is then processed by a Euclidean clusterer to calculate spatial distance, resulting in a point cloud cluster set. The point cloud cluster set is then used to calculate the number of points per unit volume using a local density estimation operator, resulting in a point cloud density distribution matrix. The point cloud density distribution matrix is ​​then processed by an outlier detector to extract the spatial coordinates of density anomalies, resulting in an outlier speckle coordinate set. Based on the Shannon entropy formula, the outlier speckle coordinate set is subjected to logarithmic probability distribution calculation within a preset discrete spatial grid to generate a speckle information entropy matrix. The speckle information entropy matrix is ​​then processed through global convergence calculation to extract a global speckle information entropy scalar. The global speckle information entropy scalar is input into a smoothing mapping function to output speckle noise entropy values.

[0048] Specifically, the generation process of speckle noise entropy values:

[0049] The high-frequency 3D spatial coordinates and reflection intensity data collected by the airborne lidar are used as depth point cloud data. Due to the extremely large amount of raw point cloud data and the presence of speckle noise caused by a large number of suspended particles under adverse weather conditions (such as dust, rain, fog, and snow), the depth point cloud is first subjected to voxelization spatial filtering by downsampling operators. The physical side length parameter of the 3D voxel grid is set (e.g., 0.05 meters), and the continuous 3D physical space is cut into discrete voxel grids. In each 3D voxel space containing laser reflection points, the 3D geometric centroid of all internal point coordinates is calculated, and the calculated single centroid coordinates are used to replace all the original dense points in the voxel, outputting a sparse point cloud tensor.

[0050] The sparse point cloud tensor is then processed using a Euclidean clusterer to calculate spatial distances. Starting with any valid point in 3D space as the initial seed point, the cluster searches for other valid points within a spherical neighborhood with a radius equal to a Euclidean spatial distance threshold (e.g., 0.2 meters). If any exist, they are grouped into the same data cluster. This search process is then recursively performed, using newly added points as new seed points, until the cluster can no longer absorb any new points. This threshold (the spherical neighborhood is determined by forward derivation based on the angular resolution of the mounted LiDAR and the minimum physical size of the obstacle to be avoided by the target) is then processed. After traversing all unclassified points, the output is a set of point cloud clusters containing multiple independent geometric stacks.

[0051] For any independent cluster in the point cloud cluster set, with each single point in the cluster as the center, the average Euclidean physical distance of the cluster within the preset K nearest neighbors (e.g., the fifteen nearest points) is calculated. The reciprocal of the average Euclidean distance is used as the density proxy index of the local point distribution density. Thus, the density mapping table of each point is output, which is the point cloud density distribution matrix.

[0052] The point cloud density distribution matrix is ​​calculated using a Gaussian statistical model to determine the global mean and standard deviation of the entire density distribution matrix, and a dynamic detection lower bound threshold is set (in this embodiment, the global mean minus twice the standard deviation). The judgment logic is as follows: when the local density value of a spatial point is strictly lower than the detection threshold, the point is determined to lack local smooth support and belong to unstructured isolated noise points caused by reflection of airborne particles. The three-dimensional coordinates of these outlier points that meet the conditions are extracted, and the outlier speckle coordinate set is output.

[0053] The spatial distribution probability of outlier speckle coordinate sets appearing within a set of preset discrete 3D spatial grids is calculated as follows: The number of outlier speckle coordinate points within a single discrete 3D spatial grid is counted, divided by the total number of point clouds extracted from the outlier speckle coordinate set at the current moment, to calculate the spatial distribution probability of that grid. Then, the distribution probability value of each grid is multiplied by the base-2 logarithm of that probability, and the product is negative to obtain the independent speckle information entropy of each grid. The independent speckle information entropies of all grids are arranged in a two-dimensional or three-dimensional structure according to their corresponding spatial topological positions to generate a speckle information entropy matrix. Subsequently, a global convergence operation of arithmetic mean summation is performed on all values ​​in the speckle information entropy matrix to extract the global speckle information entropy scalar representing the degree of global speckle interference.

[0054] The global speckle information entropy scalar is ultimately input to the smoothing mapping function. This embodiment uses a sigmoid-type nonlinear activation function, which utilizes the nonlinear saturation characteristics of the exponential function to smoothly and continuously map the discrete entropy value to a closed interval between 0 and 1, outputting the speckle noise entropy value.

[0055] By separating the reflected background and extracting illumination distortion using Laplace's derivative, and combining point cloud density with the Shannon entropy formula to output speckle noise entropy, the degree of environmental interference is quantified, providing reliable data support for dynamic weight allocation. This helps to identify perception degradation in scenarios with sudden changes in lighting or dust, and suppresses error amplification caused by local interference.

[0056] Furthermore, the image sequence, depth point cloud, and pose state parameters are coordinate registered to output the registered image sequence and registration point cloud. A feature extraction network is then used to output the visual feature set and radar feature set; this corresponds to step S2 above. The specific implementation process includes:

[0057] The image sequence is clock-matched with the depth point cloud to output a synchronized image sequence and a synchronized point cloud. The pose state parameters are optimized in the time domain based on the timestamp of the synchronized image sequence to output a synchronized reference pose matrix. The synchronized reference pose matrix is ​​input into a quaternion analyzer to solve for transient attitude fluctuations of the machine body, outputting a rotational perturbation matrix. This matrix is ​​then fused with the fixed installation offset of the heterogeneous sensor through a coordinate transformer to generate an extrinsic parameter transformation matrix. The synchronized image sequence is input into a camera calibration model to perform optical distortion correction, outputting a distortion-free image sequence. A pre-calibrated intrinsic parameter feature matrix is ​​obtained and combined with the synchronized point cloud, the extrinsic parameter transformation matrix, and the distortion-free image sequence to input into a spatial reprojection model for mapping transformation. The common-view frustum region is extracted, and the registered image sequence and the registration point cloud are output.

[0058] Specifically, the generation process of the registered image sequence and registration point cloud is as follows:

[0059] After receiving the image sequence and depth point cloud, the trigger timestamp of the camera's global shutter exposure is used as the absolute reference timeline. In the radar point cloud data stream, the frame of complete point cloud data with the smallest absolute value of the difference between the preceding and following timestamps (and the difference threshold must be less than or equal to 2ms) is retrieved in both directions. This paired data is extracted, and the synchronized image sequence and synchronized point cloud are output.

[0060] Simultaneously, the pose state parameters (including quaternions representing the attitude and three-dimensional coordinates representing the position) transmitted back from the airborne measurement unit are received. These parameters are optimized in the time domain based on the timestamps of the synchronized image sequence. Since the IMU sampling frequency is much higher than the camera frame rate, two adjacent IMU sampling points before and after the camera exposure timestamp are located. Using a spherical linear interpolation algorithm based on the kinematic model, the spatial six-degree-of-freedom state of the UAV body at the moment of camera exposure is calculated, and the synchronized reference pose matrix is ​​output.

[0061] The synchronous reference pose matrix is ​​then input into the quaternion analyzer, which expands the four-dimensional real quaternion vector (including a scalar part and three imaginary vector parts) through a nonlinear trigonometric equation system of arctangent and arcsine functions to calculate the roll angle, pitch angle and yaw angle of the UAV, and outputs an orthogonal 3×3 rotational disturbance matrix.

[0062] Furthermore, the rotational perturbation matrix is ​​fused using a coordinate transformer. Combined with the fixed mechanical installation offset from the LiDAR optical center to the camera optical center, pre-obtained via an offline calibration board (e.g., Zhang Zhengyou calibration method combined with laser point cloud plane fitting), matrix multiplication and addition operations are performed to generate a unified, 4×4 homogeneous transformation matrix, i.e., the extrinsic parameter transformation matrix. This matrix contains all external relative spatial information between the heterogeneous sensors.

[0063] In this embodiment, the transient angular velocity change rate of the airborne microelectromechanical system (MEMS) is introduced to perform high-frequency jitter compensation updates on the extrinsic parameter transformation matrix. Specifically, after generating the extrinsic parameter transformation matrix, the high-frequency angular velocity sequence of the airborne measurement unit is extracted from 2 milliseconds before to 2 milliseconds after camera exposure. The transient angular velocity change rate of this sequence in the three rotational directions of roll, pitch, and yaw is calculated. If the absolute value of the angular velocity change rate in any direction exceeds 1.5 radians per second, the airframe is determined to be in a high-frequency mechanical resonance state. At this time, the extracted angular velocity change rate is multiplied by the camera exposure time constant, and transformed into a 3×3 small rotational perturbation matrix using the first-order Taylor series expansion formula. Subsequently, this small rotational perturbation matrix is ​​fine-tuned by matrix multiplication with the rotated part of the original extrinsic parameter transformation matrix. The camera exposure time constant is obtained in real time by reading the exposure duration parameter through the camera's underlying application programming interface. The above process eliminates the pixel-level projection geometric offset error caused by high-frequency rotor vibration, improving the spatial alignment accuracy of heterogeneous data.

[0064] The synchronous image sequence is input into the camera calibration model to perform optical distortion correction. The correction process is as follows: based on the lens distortion coefficients (such as radial and tangential distortion coefficients) obtained from calibration, a nonlinear reverse offset compensation operation is performed on the physical coordinates of each pixel on the image plane according to the polynomial optical distortion model to eliminate barrel or pincushion distortion of the image and output a distortion-free image sequence; at the same time, an intrinsic parameter feature matrix containing camera focal length parameters and principal optical center coordinate parameters is obtained.

[0065] Finally, using the extrinsic transformation matrix, the three-dimensional physical coordinates of the synchronization point cloud are rotated and translated from the radar local coordinate system to the camera local coordinate system. Then, they are multiplied by the intrinsic feature matrix, and the calculated horizontal and vertical coordinates in the three-dimensional coordinate system are divided by the depth value Z, thereby projecting the three-dimensional spatial point cloud onto the pixel plane of the two-dimensional camera. Next, the common-view frustum region is extracted, specifically: all invalid point clouds whose projected horizontal and vertical coordinate values ​​exceed the actual physical resolution boundaries of the image (in this embodiment, exceeding 1920 and 1080) are removed, retaining only the data within the camera's field of view. This data is then compared with the distortion-reduced image sequence for extraction, ultimately outputting a spatially perfectly aligned and complementary registered image sequence and a registration point cloud.

[0066] By using clock matching, pose temporal domain optimization, and spatial reprojection to extract the common-view frustum, spatiotemporal alignment of heterogeneous sensors was achieved, reducing the risk of misalignment caused by asynchronous acquisition and installation deviations. This provides a unified physical benchmark for multimodal feature fusion and improves the spatial consistency of joint sensing.

[0067] The registered image sequence is input into a multi-scale residual convolutional network to extract multi-level spatial topological information and output a visual semantic feature tensor. The visual semantic feature tensor is then processed by a spatial attention layer to calculate pixel-level weights and output a visual feature set. The registered point cloud is input into a spatial bounding box generator for recursive partitioning and outputs a 3D subspace array. Local geometric topological information is then extracted using sparse 3D convolutional kernels, and a 3D geometric feature matrix is ​​output. The 3D geometric feature matrix is ​​input into a bird's-eye view projection operator to perform a height-dimensional orthogonal compression transformation and output a radar feature set.

[0068] Specifically, the generation process of the visual feature set and the radar feature set is as follows:

[0069] The registered image sequence is first input into a multi-scale residual convolutional network (based on the ResNet architecture in this embodiment) to extract multi-level spatial topological information. It consists of four residual stages, each containing 3, 4, 6, and 3 residual blocks respectively, with output channels of 256, 512, 1024, and 2048 respectively. At the first residual block of stages 2 through 4, the network performs spatial downsampling using convolutional layers with a stride of 2, ultimately outputting a visual semantic feature tensor with a spatial size of 1 / 16th of the input and 2048 channels.

[0070] The computational logic for each residual block is as follows: the input multidimensional tensor is passed through three consecutive convolutional layers (including pointwise convolutions for spatial dimension compression and expansion, and standard convolutions for feature extraction), and linear convolution dot product operations are performed on it. Simultaneously, through an identity mapping jump connection spanning these convolutional layers, the original input tensor and the output tensor after convolution are added element-wise. The result of the addition is then input into a modified linear unit activation function to introduce a nonlinear response.

[0071] As network depth increases, residual convolutional networks perform spatial downsampling by increasing the convolution stride, expanding the receptive field of neurons layer by layer, capturing local texture details and global semantic contours of the image, and outputting a visual semantic feature tensor with compressed spatial dimension and expanded channel dimension.

[0072] Global max pooling and global mean pooling are performed on the visual semantic feature tensor in the depth channel dimension to generate two refined two-dimensional spatial feature distribution maps. These two two-dimensional maps are merged and stitched together in the channel dimension, and then fused through a standard convolutional layer with a large receptive field. The Sigmoid non-linear activation function is applied to generate a normalized weight matrix with values ​​strictly limited to 0 to 1.

[0073] Finally, the normalized weight matrix is ​​multiplied element-wise along the tensor dimension by the original visual semantic feature tensor. The physical meaning of this operation is to adaptively enhance the high-frequency feature responses of potential dynamic obstacles and significant geometric edges in the image, while significantly suppressing invalid low-frequency interference from solid-color backgrounds such as the sky and flat roads, ultimately outputting a visual feature set filtered by the attention mechanism.

[0074] In parallel with the visual flow, since the disorder and continuity of point clouds are not suitable for direct convolution, the bounding box generator of the registration point cloud set input space is recursively partitioned. The operation process is as follows: First, a 3D physical monitoring bounding space is defined with the current geometric center of the UAV as the origin, and the length, width, and height have fixed physical dimensions (such as covering 100 meters in front and several meters above and below). Then, according to the preset high-precision spatial resolution, the 3D physical monitoring bounding space is divided into hundreds of thousands of small cubic grids of equal size. Next, the spatial boundary to which the 3D absolute coordinates of each point in the registration point cloud set belong is determined, and the point cloud data is mapped and filled into the corresponding 3D index of the grid. For non-empty grids containing multiple point clouds, the arithmetic mean of the coordinates and reflection intensity of all points in the grid is calculated, which is used as the local representative feature of the grid. Finally, a 3D subspace array with a structured matrix arrangement is output.

[0075] Due to the openness of the real environment, the 3D subspace array exhibits extremely high sparsity (i.e., the vast majority of grid cells are null). To avoid inefficient consumption of computing power and GPU memory, the 3D subspace array extracts local geometric topology information through sparse 3D convolutional kernels. This embodiment employs a submanifold sparse convolutional network to construct a 3D convolutional weight kernel with a preset size (e.g., 3×3×3). When traversing the 3D subspace array, this convolutional kernel is only activated and performs dot product multiplication and weight gradient updates when the input grid cell corresponding to the current center position contains valid non-empty point clouds; if the center is empty, the operation is skipped and the output remains 0. This constraint mechanism not only extracts the 3D geometric contours of the obstacle surface but also ensures that the output tensor strictly maintains the same extremely sparse structure as the input tensor, preventing spatial expansion caused by feature diffusion, thereby outputting a high-dimensional 3D geometric feature matrix.

[0076] The four-dimensional three-dimensional geometric feature matrix (containing three spatial dimensions: length, width, and height, and one feature channel dimension) is forcibly reshaped and merged along the Z-axis dimension representing physical height. The original height dimension and channel dimension are algebraically multiplied, thus flattening the four-dimensional spatial tensor into a three-dimensional tensor containing only the length, width, and dilated channel dimensions. Subsequently, a series of two-dimensional pointwise convolutional layers (i.e., convolutional kernel size of 1×1) are used to linearly superimpose and compress the dilated channel dimensions across channels, ultimately outputting a radar feature set with a physical top-down view directly above the UAV.

[0077] By utilizing residual networks and attention mechanisms to extract visual semantic features, and combining sparse 3D convolution and bird's-eye view projection to extract radar geometric features, a dimensionality reduction transformation from 3D to 2D was achieved, providing high-quality structured data for cross-modal feature stitching.

[0078] Further, the illumination distortion gradient value, speckle noise entropy value, and flight speed parameter are input into the weight adjustment function, which outputs visual weight coefficients and radar weight coefficients. The visual weight coefficients and radar weight coefficients are then multiplied by tensors with the visual feature set and radar feature set, respectively, to generate a weighted visual feature set and a weighted radar feature set. These are then aligned and stitched together to generate a joint heterogeneous feature tensor; this corresponds to step S3 above; see [link to relevant documentation]. Figure 2 The specific implementation process includes:

[0079] The illumination distortion gradient value is input into a one-dimensional convolutional layer, which outputs a smooth gradient vector; the speckle noise entropy value is used to generate a standard entropy vector through a constraint operator; the flight velocity parameter is filtered to remove physical jitter, and a smooth velocity vector is output; the smooth gradient vector, standard entropy vector, and smooth velocity vector are input into a concatenation layer for merging, and a state matrix is ​​output; the state matrix is ​​input into a fully connected network to extract nonlinear mapping relationships, and a hidden layer state vector is output; the hidden layer state vector is passed through a normalization operator for scalar mapping, and visual weight coefficients and radar weight coefficients are output.

[0080] Specifically, the generation process of the visual weighting coefficient and the radar weighting coefficient is as follows:

[0081] First, for the photometric environment, the gradient value of the illumination distortion is input into a one-dimensional convolutional layer. Using a fixed-length one-dimensional sliding time window, the gradient value sequence containing historical multi-frame temporal illumination information is linearly convolved and low-pass filtered to eliminate the transient, pseudo-random photometric abrupt changes caused by the periodic occlusion of light due to the high-frequency mechanical vibration of the UAV body or the high-speed rotation of the rotor. The output is a smooth gradient vector that continuously reflects the trend of ambient illumination.

[0082] Meanwhile, regarding point cloud quality, speckle noise entropy values ​​are standardized using a constraint operator. The specific process involves pre-setting upper and lower bounds for an empirically safe entropy value (e.g., the lower bound is set to the entropy value of an extremely clean environment, and the upper bound to the entropy value of a severely disturbed environment). When the input real-time entropy value exceeds this range, it is forcibly truncated and pulled back to the boundary extreme value. Subsequently, using the range normalization mathematical formula, it is linearly scaled and mapped to a standard 0-to-1 floating-point space to prevent numerical divergence in subsequent neural network weight calculations due to individual extreme abnormal noise points, outputting a standard entropy vector.

[0083] Considering that the high-speed maneuvering of the UAV will exacerbate motion ambiguity and time asynchrony of sensor data, this embodiment uses a digital second-order Butterworth low-pass filter to establish a frequency domain transfer function based on a second-order constant coefficient linear difference equation and set a low cutoff frequency (such as 12 Hz). This cutoff frequency parameter is selected and set according to the inherent mechanical resonance frequency of the UAV frame structure and the lower bound of the multi-rotor blade pass frequency in this embodiment, so as to ensure that the amplitude of the input triaxial flight speed raw digital signal is attenuated by high frequency stopband and output a smooth velocity vector.

[0084] The smoothed gradient vector, standard entropy vector, and smoothed velocity vector are simultaneously input into the concatenation layer. Each of these three feature streams has a fixed one-dimensional feature length (e.g., the smoothed gradient vector extracts 64 feature values, the standard entropy vector extracts 64 feature values, and the smoothed velocity vector contains 3 feature values ​​along the X, Y, and Z axes). In the concatenation layer, these three vectors are directly concatenated at the end of the data dimension to output a state matrix with a total feature length equal to the sum of the lengths of the three vectors (i.e., containing 131 feature nodes).

[0085] The state matrix is ​​then input into a fully connected network to extract nonlinear mappings. This network consists of multiple layers of neurons. Its first hidden layer nonlinearly compresses the input high-dimensional state matrix (i.e., 131 nodes) to a suitable number of intermediate feature nodes (e.g., 32 hidden nodes) through matrix multiplication and activation functions, thereby initially filtering out redundant environmental state information. Subsequently, the second output layer maps and compresses these 32 hidden nodes into a one-dimensional vector containing only two numerical outputs, i.e., the output hidden state vector.

[0086] Finally, the hidden state vector is scalar-mapped using the Softmax exponential normalization function. Specifically, the natural exponent values ​​of the two elements representing the visual branch response and the radar branch response in the hidden state vector are calculated separately, and these two exponent values ​​are added together as the global denominator. Then, each independent exponent value is divided by this global denominator to ensure that the two output weight coefficients are always between 0 and 1, and that their arithmetic sum is absolutely equal to 1. This outputs the visual weight coefficient and the radar weight coefficient. When visual lighting conditions deteriorate, the visual weight coefficient is automatically reduced by the algorithm, while the radar weight coefficient increases accordingly; conversely, the opposite is also true.

[0087] By merging the gradients and entropy vectors related to environmental disturbances with the body velocity parameters, and extracting the nonlinear mapping output weight coefficients through a fully connected network, the perception fluctuations caused by the degradation of single sensor data are alleviated, thereby improving the adaptability to complex environments.

[0088] The visual feature set and visual weight coefficients are subjected to dimensional copying and element-wise weighted multiplication through a tensor broadcast multiplier, outputting a weighted visual feature set. The radar feature set and radar weight coefficients are input into a tensor broadcast multiplier for dimensional copying and element-wise weighted multiplication, outputting a weighted radar feature set. The weighted visual feature set is input into an inverse perspective mapping operator to perform perspective transformation from pixel space to physical overhead space, outputting a bird's-eye view visual feature tensor. The bird's-eye view visual feature tensor and the weighted radar feature set are subjected to resolution interpolation and coordinate system origin alignment through a spatial grid sampler to generate a pair of feature matrices of the same dimension. The pair of feature matrices of the same dimension are subjected to channel dimension superposition through a deep cascaded layer, outputting a stitched feature volume. The stitched feature volume is subjected to cross-channel nonlinear fusion extraction using a pointwise convolutional network, outputting a joint heterogeneous feature tensor.

[0089] Specifically, the generation process of the joint heterogeneous feature tensor is as follows:

[0090] The visual feature set and visual weight coefficients are first computed and adapted using a tensor broadcast multiplier. Specifically, since the visual weight coefficients are only single scalar values, while the visual feature set is a three-dimensional tensor, the multiplier first copies and expands the scalar coefficients in memory along the length, width, and depth dimensions of the two-dimensional space of the feature set, giving it a tensor shape completely consistent with the visual feature set. Then, the copied weight matrix is ​​multiplied element-wise with the corresponding spatial elements of the original visual feature matrix, ensuring that the activation response values ​​throughout the feature space are scaled proportionally according to the environmental quality weights, outputting a weighted visual feature set.

[0091] Similarly, in parallel processing, the radar feature set and radar weight coefficients are input into another parallel tensor broadcast multiplier, which performs the same dimension copying and element-wise weighted multiplication operations, and outputs a weighted radar feature set whose response is scaled accordingly.

[0092] Since the weighted visual feature set retains the original camera's two-dimensional perspective pixel spatial attributes (i.e., perspective distortion where near pixels appear larger than distant pixels), while the weighted radar feature set has already been transformed into a three-dimensional orthogonally compressed physical bird's-eye view space through the previous operators, an inverse perspective mapping operator needs to be introduced to perform geometric perspective transformation. The logic of this mapping operator is as follows: First, extract the real-time camera pitch angle, roll angle, and ground clearance from the synchronous reference pose matrix. Using the intersection theory of the pinhole imaging geometric model and the local actual spatial surface, calculate the 3×3 dimension homography projection transformation matrix. The specific mathematical derivation process is as follows: Extract the camera's intrinsic parameter feature matrix, and combine it with the camera's pitch and roll angles to construct a local rotation matrix from the body coordinate system to the world coordinate system; construct a three-dimensional translation vector using the camera's ground clearance; subsequently, extract the unit normal vector of the local actual spatial surface based on the synchronous point cloud, and perform matrix algebra operations on the camera's intrinsic parameter feature matrix with the local rotation matrix, translation vector, and the unit normal vector of the local actual spatial surface in sequence to generate a homography projection transformation matrix from the two-dimensional perspective image plane to the physical bird's-eye view plane.

[0093] By applying a homography projection transformation matrix to perform linear fractional projection algebra operations on the coordinates of each pixel in the weighted visual feature set, perspective distortion that causes distant objects to appear smaller in the image is eliminated, outputting a bird's-eye view feature tensor. To achieve feature channel fusion, a uniform-sized grid map is constructed on a virtual two-dimensional bird's-eye view plane with the origin of the UAV's body coordinate system as the absolute geometric center. Using a bilinear spatial interpolation algorithm, the feature floating-point values ​​of each coordinate point on the unified reference grid are re-estimated and assigned based on the spatial resolution and geometric offset distance of both the bird's-eye view feature tensor and the weighted radar feature set, ensuring that the length and width dimensions of the matrices are completely identical, generating a pair of feature matrices of the same dimension.

[0094] At this point, the same-dimensional feature matrices are superimposed through the channel dimension of the deep cascaded layer, keeping the aligned bird's-eye view features and radar features unchanged in planar space size, while in the depth of the feature channel dimension, the two are directly concatenated and spliced ​​(for example, the visual features with 256 channels and the radar features with 256 channels are concatenated end to end and fused into a tensor with 512 channels), and the spliced ​​feature volume is output.

[0095] Finally, to eliminate redundant computations caused by simple splicing, several 1×1 two-dimensional convolutional kernels are used to perform full-coverage traversal scanning dot product operations on the thick spliced ​​feature volume. Through the linear combination mechanism of weights within the convolutional kernels and the nonlinear mapping filtering of the activation function, the number of channels after fusion is recompressed and reduced in dimensionality (e.g., re-extracted and condensed to 256 core channels), outputting the final joint heterogeneous feature tensor.

[0096] By using inverse perspective mapping to transform weighted visual features into a bird's-eye view space, and aligning them with radar features through spatial grid sampling, and outputting a joint tensor through pointwise convolution, the spatial heterogeneity caused by perspective distortion is improved, enabling heterogeneous features to achieve deep fusion under a unified coordinate system and extracting a more complete global environmental representation.

[0097] Furthermore, the joint heterogeneous feature tensor is input into the probabilistic prediction model, outputting a spatial probability occupancy map. This map, along with the pose state parameters and flight velocity parameters, is then input into the model predictive control algorithm to generate airframe attitude and thrust control commands. These commands are then transmitted to the underlying flight controller to execute physical obstacle avoidance; this corresponds to step S4 above; see [link to relevant documentation]. Figure 3 The specific implementation process includes:

[0098] The joint heterogeneous feature tensor is used to reduce the dimensionality of feature channels through a multilayer perceptron model, outputting a two-dimensional compressed feature. The two-dimensional compressed feature is used to calculate the discrete height layer probability, outputting a height probability distribution vector set. The two-dimensional compressed feature and the height probability distribution vector set are reconstructed through orthogonal dimension reconstruction, outputting an initial voxel feature. The initial voxel feature is used to expand the spatial resolution using a three-dimensional deconvolutional network model, outputting a three-dimensional upsampled tensor. The three-dimensional upsampled tensor is input into a normalized exponential activation function, outputting a spatial probability distribution matrix. The spatial probability distribution matrix is ​​used to generate an initial continuous occupancy map using a voxel occupancy grid generator. The initial continuous occupancy map is then diffusely smoothed and expanded, outputting a spatial probability occupancy map.

[0099] Specifically, the generation process of the spatial probability occupancy map is as follows:

[0100] Each feature vector in the two-dimensional joint heterogeneous feature tensor is sequentially input into a fully connected neural network containing multiple hidden layers. Utilizing the matrix multiplication property of the neural network, redundant feature dimensions with low correlation are eliminated, resulting in a more streamlined two-dimensional compressed feature output with fewer channels.

[0101] Subsequently, using a dedicated depth prediction convolutional head, at each independent grid spatial location in the 2D feature map, a classification prediction network calculates the confidence score of the presence of physical obstacles at each of the preset discrete height intervals (e.g., dividing the physical space from 3 meters above to 3 meters below the drone into 16 equally spaced height probability layers). The output is a set of height probability distribution vectors representing the likelihood of an obstacle occupying a height direction. The depth prediction convolutional head consists of two consecutive 2D convolutional layers: the first layer uses a 3×3 convolutional kernel to perform spatial feature fusion on the 2D compressed features, introducing nonlinearity through the ReLU activation function; the second layer uses a 1×1 convolutional kernel to map the number of channels to a preset number of discrete height layers (16 layers in this embodiment), outputting the unnormalized score of the presence of obstacles at each plane grid location in each height layer. The 3D deconvolutional network consists of two consecutive cascaded 3D transposed convolutional layers, with each layer configured as follows: kernel size 4×4×4, stride 2, and edge padding 1, progressively increasing the voxel spatial resolution by a factor of 4.

[0102] The two-dimensional compressed feature matrix in the XY plane is subjected to a tensor cross product operation with the height probability distribution vector set representing the spatial distribution along the Z-axis. In the mathematical matrix space, the originally flat two-dimensional features are physically expanded and vertically extended along the Z-axis, thereby restoring the initial voxel features with a coarse three-dimensional spatial structure array.

[0103] To meet the need for drones to avoid small obstacles (such as power lines and tree branches) during high-speed flight, the initial voxel features are expanded in spatial resolution using a 3D deconvolutional network model. The core of this step is to upsample the coarse, low-resolution features into a high-precision occupancy grid. In this embodiment, the deconvolutional network uses specific parameter configurations: for example, a 3D transposed convolutional kernel with a kernel size of 4×4×4, a stride of 2, and an edge padding parameter of 1. The calculation process is as follows: Zero values ​​are forcibly inserted between adjacent element matrices of the input initial voxel feature tensor to expand the physical memory dimension. Subsequently, the 3D convolutional kernel performs sliding matrix multiplication and addition operations on the expanded, massive tensor, filling in gaps based on trained weights, thereby smoothly amplifying the 3D spatial resolution of the voxels by a factor of 1, and outputting a reconstructed 3D upsampled tensor with geometric details and edge contours.

[0104] Subsequently, along the channel dimensions that characterize the semantic category of the environment and the physical occupancy state, the natural index of each channel value within each three-dimensional voxel is calculated; then, the natural index is divided by the algebraic sum of the index values ​​of all channels within that voxel, thereby transforming the dimensionless numerical value output by the network into a physical distribution value, and outputting a spatial probability distribution matrix.

[0105] In this embodiment, the spatial probability distribution matrix is ​​subjected to temporal recursive filtering, and the obstacle occupancy probabilities of historical multiple frames are fused.

[0106] Specifically, after obtaining the spatial probability distribution matrix generated at the current moment, the historical spatial probability data of the previous 5 frames are extracted from the memory cache. Temporal fusion is performed using an exponentially weighted moving average algorithm. The specific calculation is as follows: for each voxel in the 3D space of the current frame, a current confidence weight of 0.7 is assigned; the historical probability values ​​of voxels with the same coordinates from the previous 1 to the previous 5 frames are arithmetically averaged to obtain the historical average probability value, which is then multiplied by a historical decay weight of 0.3. The current confidence weighted value and the historical decay weighted value are added together to update the final probability of the voxel. If the initial probability of a voxel is below 10% for 3 consecutive frames, its historical accumulated weight is forcibly reset to zero. This process reduces the probability flickering caused by obstacles due to brief occlusion or dust interference in a single frame, improving the safety of drone flight.

[0107] The spatial probability distribution matrix utilizes a voxel occupancy grid generator to iterate through each voxel, comparing the occupancy probability in the probability distribution matrix with the absolute value of a collision determination probability threshold (e.g., a probability greater than 50%). If the probability is greater than this safety threshold, the voxel space is determined to be occupied by a physical obstacle, and its value is discretely assigned to 1; conversely, if the probability is less than the threshold, it is determined to be free and passable space, and its value is assigned to 0, thus outputting the initial continuous occupancy map.

[0108] Finally, considering the physical frame size of the UAV (including rotor wheelbase) and the aerodynamic boundary effects during flight (such as the influence of rotor downwash), the initial continuous occupancy map uses the dilation operation in three-dimensional mathematical morphology to forcibly expand the occupied voxels in the map that are judged to have a value of 1 outward to their 26 adjacent grids in three-dimensional space by a certain safe physical margin (e.g., expanding outward by 0.3 meters in all directions), reserving physical fault tolerance space for the UAV. Finally, a spatial probability occupancy map that is dedicated to dynamic trajectory planning and includes the equivalent dilation size of the UAV body is output.

[0109] By orthogonally reconstructing two-dimensional compressed features and height probability vectors, and generating a spatial probability occupancy map through three-dimensional deconvolution and smooth expansion, the system effectively incorporates body size and safety margin, providing an obstacle model for the control algorithm that balances accuracy and fault tolerance.

[0110] Flight speed and attitude parameters are used to solve the multi-rotor thrust boundary through dynamic equations, outputting a feasible control input set. The feasible control input set is input into the state transition model to output a three-dimensional spatial sampling node sequence. The three-dimensional spatial sampling node sequence and the spatial probability occupancy map are used to calculate the expected collision risk through a Bayesian cost function, generating a continuous generation value matrix. The continuous generation value matrix is ​​iteratively solved by a gradient descent optimization solver to find the minimum cost sequence, outputting the optimal spatial avoidance trajectory. The optimal spatial avoidance trajectory is input into an inverse dynamics solver to extract lateral and vertical acceleration components, generating airframe attitude and thrust control commands. The airframe attitude and thrust control commands are transmitted to the underlying flight controller and converted into independent control signals for the quadcopter motors to execute physical obstacle avoidance actions.

[0111] Specifically, the process of performing physical obstacle avoidance actions is as follows:

[0112] First, using flight speed and attitude parameters, and based on the Newton-Euler equations in rigid body dynamics, combined with the input of the current UAV mass, real-time air density, and motor rotor tension coefficient (for example, setting the tension coefficient constant to 1.4865 × 10⁻⁶), -7 The constants are calculated by statically calibrating a specific model of brushless motor and propeller combination on a ground test bench, and by linear regression fitting of the square of the rotational speed and the generated lift. These constants include the system constant, moment of inertia, and the mechanical limit of the maximum tilt angle. By simultaneously solving the nonlinear equations, the maximum vertical absolute lift of the multirotor system under the current flight attitude and velocity inertia is calculated as the sum of the products of the squares of the maximum rotational speeds of the four motors and their corresponding rotor thrust coefficients. The maximum horizontal limit acceleration is calculated by multiplying the maximum vertical lift by the sine of the maximum tilt angle of the airframe and subtracting the aerodynamic drag under real-time air density. The output is a feasible control input set composed of these physical extreme values.

[0113] The feasible control input set is then input into the state transition model to extrapolate future trajectories. Using the currently perceived 3D position, 3D linear velocity, and 3Axis acceleration of the UAV as the initial state of the dynamic differential equations, and based on an extremely small time discretization step (e.g., 0.1 seconds), the ordinary differential equations are iterated forward using the fourth-order Runge-Kutta numerical integration method. Within the line-of-sight range predicted by the control command extrapolation (e.g., predicting the next 15 discrete time steps), all possible candidate trajectory point sequences satisfying the aforementioned dynamic thrust boundary limits are calculated, and a tree-like diverging sequence of 3D spatial sampling nodes is output.

[0114] The sequence of 3D spatial sampling nodes and the spatial probability occupancy map are synchronously input into the Bayesian cost function. The 3D continuous physical coordinates of each predicted trajectory node in the sequence are extracted, and then converted into discrete grid indices using a preset voxel grid physical side length resolution. In this embodiment, the voxel grid physical side length resolution is set to 0.05 meters. The selection of this physical side length resolution is based on a comprehensive consideration of the physical size limits of the smallest typical obstacles that the UAV needs to avoid in complex low-altitude environments (such as construction tunnels and forests) (e.g., suspended construction cables, outstretched tree branches, whose diameters are usually between 0.05 meters and 0.1 meters). In the specific mapping calculation, the 3D continuous physical coordinates of the predicted trajectory nodes are subtracted from the origin physical coordinates of the spatial probability occupancy map, divided by the 0.05-meter physical side length resolution, and the calculation result is rounded down, thereby converting continuous physical spatial points into 3D discrete integer indices in the spatial probability occupancy map. Subsequently, the discrete integer index is used to retrieve the occupancy map, extract the collision probability value of the corresponding voxel grid, and use it as the geometric spatial overlap of the node.

[0115] Subsequently, the overlap penalty value is multiplied by the collision risk weight parameter dynamically adjusted through Bayesian online optimization learning (for example, when a high-speed oncoming obstacle is detected ahead, the Bayesian optimizer will continuously adjust the collision penalty weight from the normal value of 100 to 5000). Simultaneously, the yaw penalty cost for deviation of the flight trajectory from the predetermined global target route is added, which is obtained by calculating the square of the Euclidean spatial distance between the predicted trajectory node and the nearest reference point on the predetermined global target route; and the control energy cost incurred to perform obstacle avoidance, which is obtained by calculating the sum of the squares of the total thrust control inputs over all prediction time steps. These cost functions are then multiplied by preset risk adjustment weights, yaw adjustment weights, and energy adjustment weights (for example, to emphasize obstacle avoidance safety, the preferred weight ratio of the three is 0.7:0.2:0.1), and after unifying the dimensions, a weighted sum is performed to output a continuous generation value matrix. The specific derivation process of this Bayesian online optimization learning is as follows: First, a prior probability model of collision risk based on a sparse Gaussian process is established to control computational complexity; then, the relative approach velocity and relative acceleration of obstacles obtained at the current time and within a set historical observation sliding window are used as observation samples, substituted into the likelihood function for updating, and the posterior distribution representing the true collision probability is calculated; finally, an acquisition function is constructed with the goal of maximizing the expected safety margin, and the optimal penalty weight multiplier is output under the guidance of this function, so that when the relative approach velocity is detected to increase, the collision penalty weight is nonlinearly and significantly increased to a specific high threshold (e.g., 5000).

[0116] In this embodiment, a nonlinear, nonconvex optimization engine (such as the PANOC algorithm solver) is used to find the minimum cost trajectory for the continuous cost matrix. Specifically, within the feasible region defined by the cost matrix, the engine calculates the partial derivative of the cost function with respect to the control input and performs multiple rapid fine-tuning iterations along the mathematical direction where the gradient descent of the cost function is steepest and fastest. This iterative process is subject to strict constraints on the feasible control input set until the algorithm converges and finds a three-dimensional spatial position sequence with the absolute minimum total cost (meaning the lowest collision risk, the most stable control input, and the closest proximity to the target route), outputting the optimal spatial avoidance trajectory.

[0117] For the target point on the optimal spatial avoidance trajectory, by taking the second-order numerical derivative of the spatial position sequence with respect to the time variable twice consecutively, the lateral and vertical acceleration components that the UAV must possess at each moment to complete the smooth trajectory are extracted. Subsequently, using the quadrotor rigid body attitude mapping matrix, these abstract acceleration component matrices are nonlinearly transformed into a digital command set containing the target roll angle, target pitch angle, target yaw angle angular velocity, and the percentage of the system's total thrust, thus generating the airframe attitude and thrust control commands.

[0118] Finally, the airframe attitude and thrust control commands are transmitted to the underlying flight controller via a serial bus. The flight control system incorporates a high-frequency proportional-integral-derivative inner-loop attitude control algorithm. This algorithm compares the received target attitude angle command with the actual airframe attitude angle measured and fed back by the onboard IMU sensor at ultra-high frequency in real time. Through mathematical operations such as proportional amplification, error integral accumulation, and differential prediction of the difference, it generates adjustment torque commands to resist drag and change the airframe attitude.

[0119] Finally, combining the overall thrust command, the torque is converted into the square of the target rotational speed for the four independent brushless DC motors using a hybrid control matrix solver. Specifically, for an X-shaped quadcopter, the square of the target rotational speed for the first motor equals the overall thrust command plus the roll torque, minus the pitch and yaw torques; the square of the target rotational speed for the second motor equals the overall thrust command minus the roll and pitch torques, plus the yaw torque; and so on for the other motors, using linear addition and subtraction based on a preset algebraic relationship of symmetrical frame space geometry. Finally, the square root of the target rotational speed is taken and converted into a duty cycle pulse width modulation (PWM) electrical signal, outputting independent control signals for the four quadcopter motors. These four signals independently regulate the physical rotational speed of each rotor, driving the rotary-wing UAV to perform physical dynamic obstacle avoidance maneuvers.

[0120] By using dynamic equations to solve the thrust boundary to generate a feasible control input set, and combining it with the Bayesian cost function to calculate the collision risk, the body attitude and thrust control commands are finally output through the inverse dynamics solver. By placing the dynamic thrust limit of the UAV in the trajectory optimization model, the optimal spatial avoidance trajectory is strictly constrained within the actual physical dynamic range of the body, thus improving the physical executability of dynamic obstacle avoidance actions at the underlying motor end.

[0121] Before implementing the aforementioned online dynamic obstacle avoidance, the fully connected network, multi-scale residual convolutional network, sparse 3D convolutional network, and 3D deconvolutional network models need to be jointly trained offline. The training process specifically includes:

[0122] We collected multimodal perception datasets of UAVs under different lighting and dust concentrations, and simultaneously recorded the high-frequency pose and flight velocity parameters of the onboard inertial measurement unit at corresponding moments. We then used high-precision LiDAR scanning to acquire realistic physical space point clouds, which were voxelized and used as true labels for the 3D spatial occupancy map.

[0123] Considering that the number of voxels occupied by physical obstacles in the real environment is far less than the number of idle voxels, a focus loss function is used to alleviate the extreme imbalance between positive and negative samples. At the same time, for the height probability distribution vector set, a cross-entropy loss function is introduced to constrain the deviation between the predicted height distribution and the true height.

[0124] The feature extraction network and the probabilistic prediction model are concatenated into an end-to-end training network, and gradient updates are performed using the AdamW optimizer with weight decay. The initial learning rate is set to 10. -4 Meanwhile, the weight decay coefficient is set to 10. -4 This prevents the model from overfitting when processing sparse 3D point cloud data. The optimal batch size for training is 16. The entire end-to-end network is trained for a total of 200 epochs, and a cosine annealing decay strategy is used for learning rate scheduling, ensuring that the learning rate smoothly transitions and converges to a minimum of 10 at the end of training. -6 When calculating the total loss, the weight allocation ratios for the focus loss function and the cross-entropy loss function are set to 0.7 and 0.3, respectively. The weight parameters in all convolutional kernels and fully connected layer nodes are updated through backpropagation of the optimizer until the joint loss function converges, at which point the trained network weights are output.

[0125] This invention provides a dynamic obstacle avoidance method for unmanned aerial vehicles (UAVs) based on multi-sensor fusion. By deeply fusing multi-sensor information and introducing physical constraints, a closed-loop processing framework for coordinated perception and control is constructed. First, illumination distortion gradient values ​​and speckle noise entropy values ​​are introduced, and combined with flight speed parameters to dynamically generate weight coefficients for vision and radar. Based on changes in ambient illumination and interference such as dust, the confidence levels of different sensors are adaptively adjusted, effectively suppressing the accumulation and propagation of multimodal perception errors in complex environments and improving the accuracy of the heterogeneous data fusion process. Second, this invention transforms the joint heterogeneous feature tensor generated after multimodal data processing into a spatial probability occupancy map, and synchronously inputs it, along with the current pose and flight speed of the UAV, into a model predictive control algorithm to directly generate attitude and thrust control commands. This organically incorporates the underlying dynamic physical constraints of the UAV into the spatial obstacle avoidance planning, ensuring that the output control strategy matches the actual kinematic response and dynamic boundaries of the UAV, improving the physical executability of the final flight trajectory and the safety of the flight process.

[0126] Example 2

[0127] This embodiment describes a dynamic obstacle avoidance method for UAVs based on multi-sensor fusion, applied to UAVs performing low-altitude intelligent inspection tasks. The specific implementation process is as follows:

[0128] A drone was performing a low-altitude intelligent inspection mission in an open outdoor area, cruising at 10 m / s along a predetermined straight trajectory. A dark tunnel entrance was located 50 meters directly ahead. Inside the tunnel, due to large-scale drilling operations, the area was filled with high concentrations of suspended dust. As the drone's nose rapidly entered the shadow boundary of the tunnel entrance, the ambient brightness captured by the onboard camera experienced a precipitous drop, with the physical brightness plummeting from 100,000 lx in standard outdoor daytime conditions to less than 200 lx inside the tunnel. This resulted in severe motion blur and color distortion in the images. At this moment, the illumination abrupt change feature set fluctuated drastically. After local extremum calculation and range linear normalization, the output illumination distortion gradient value abruptly increased from 0.15 under normal outdoor conditions to 0.88.

[0129] Meanwhile, the 905-nanometer wavelength laser beam emitted by the airborne lidar entered the high-concentration dust area inside the tunnel. Due to the multiple scattering and reflection of laser photons by a large number of tiny dust particles in the air, the radar receiver triggered a large number of false alarm signals, causing the volume of outlier speckle coordinate sets to increase exponentially. The local density estimation operator detected a large number of disordered and isolated noise clusters in three-dimensional space, with their local average density being significantly lower than the normal solid surface density of 15 points / m. 3 The health threshold, speckle noise entropy value rapidly increased from 0.22 (when the outdoor environment is clear) to 0.74.

[0130] The illumination distortion gradient and speckle noise entropy values ​​were synchronously input into a weight adjustment function composed of a fully connected network as constraints. Through a nonlinear mapping relationship and a Softmax normalization operator, the visual weight coefficient was suppressed to 0.31, while the radar weight coefficient was increased to 0.69. A joint heterogeneous feature tensor was generated based on the weight coefficients, and a binarized spatial probability occupancy map was output through a 3D deconvolution network. Based on this, a construction vehicle was identified 15 meters directly ahead in the tunnel, although obscured by dust, and moving slowly towards it at a speed of 2 m / s. Faced with this imminent collision threat, a forward inference solution involving 15 predicted vision steps was completed through the calculation of the MPC model predictive controller and Bayesian cost function. The collision avoidance penalty term output by the continuous generation value matrix increased sharply due to the decreasing distance, forcing the control model to abandon the dangerous collision avoidance strategy of direct emergency braking inside the tunnel. Conversely, the solver rapidly generates a cubic spline trajectory with a smooth, continuous curvature that yaws 1.5 meters to the upper left safe space within the feasible control set along the cost gradient descent direction.

[0131] Based on the optimal obstacle avoidance trajectory, the inverse dynamics solver extracted the lateral acceleration command to change the aircraft's heading, generating an attitude control command that tilted 25 degrees to the left and correspondingly increased vertical thrust. After this command was sent to the underlying flight control system via serial port, the electronic speed controller changed the PWM duty cycle, causing the speed signals of the two right motors to rise sharply to approximately 6500 RPM, while the speeds of the two left motors dropped to approximately 4000 RPM. This resulted in a physical obstacle avoidance maneuver, successfully clearing the upper left of the obstacle vehicle and continuing to fly along a safe trajectory within the limited tunnel space.

[0132] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A dynamic obstacle avoidance method for unmanned aerial vehicles (UAVs) based on multi-sensor fusion, characterized in that, include: The system receives image sequences and depth point clouds acquired by airborne sensors, and pose and flight speed parameters acquired by the airborne measurement unit. Based on the image sequences, it calculates the illumination distortion gradient value through contrast attenuation. Based on the depth point clouds, it calculates the speckle noise entropy value through local point cloud density. The image sequence, depth point cloud, and pose state parameters are coordinate registered to output the registered image sequence and registration point cloud. The visual feature set and radar feature set are then output through a feature extraction network. The illumination distortion gradient value, speckle noise entropy value, and flight speed parameter are input into the weight adjustment function, and the visual weight coefficient and radar weight coefficient are output. The visual weight coefficient and radar weight coefficient are multiplied with the visual feature set and radar feature set respectively to generate a weighted visual feature set and a weighted radar feature set. A joint heterogeneous feature tensor is generated by alignment and splicing. The specific generation process of the visual weight coefficient and radar weight coefficient includes: inputting the illumination distortion gradient value into a one-dimensional convolutional layer and outputting a smooth gradient vector; generating a standard entropy vector from the speckle noise entropy value through a constraint operator; filtering out physical jitter from the flight speed parameter and outputting a smooth speed vector; merging the smooth gradient vector, standard entropy vector, and smooth speed vector into a splicing layer and outputting a state matrix; inputting the state matrix into a fully connected network to extract nonlinear mapping relationships and outputting a hidden layer state vector; and performing scalar mapping on the hidden layer state vector through a normalization operator to output the visual weight coefficient and radar weight coefficient. The specific generation process of the joint heterogeneous feature tensor includes: visual feature set and visual weight coefficients are subjected to dimensional copying and element-wise weighted multiplication through a tensor broadcast multiplier to output a weighted visual feature set; radar feature set and radar weight coefficients are input into a tensor broadcast multiplier for dimensional copying and element-wise weighted multiplication to output a weighted radar feature set; the weighted visual feature set is input into an inverse perspective mapping operator to perform perspective transformation from pixel space to physical overhead space to output a bird's-eye view feature tensor; the bird's-eye view feature tensor and the weighted radar feature set are subjected to resolution interpolation and coordinate system origin alignment through a spatial grid sampler to generate a pair of feature matrices of the same dimension; the pair of feature matrices of the same dimension are subjected to channel dimension superposition through a deep cascaded layer to output a stitched feature volume; the stitched feature volume is subjected to cross-channel nonlinear fusion extraction using a pointwise convolutional network to output a joint heterogeneous feature tensor. The joint heterogeneous feature tensor is input into the probabilistic prediction model, which outputs a spatial probability occupancy map. This map, along with the pose state parameters and flight velocity parameters, is then input into the model predictive control algorithm to generate airframe attitude and thrust control commands, which are transmitted to the underlying flight controller to execute physical obstacle avoidance.

2. The UAV dynamic obstacle avoidance method based on multi-sensor fusion according to claim 1, characterized in that, The specific generation process of the illumination distortion gradient value and speckle noise entropy value includes: the image sequence uses an illumination intensity extractor to output an ambient brightness distribution matrix, and uses a low-pass smoothing filter to filter out high-frequency texture details to generate a global illumination component matrix; the global illumination component matrix separates the reflection background through an illumination reflection decoupling operator to output an ambient illumination distribution map; the ambient illumination distribution map uses a second-order Laplacian differentiator to extract the brightness abrupt change boundary to output an illumination abrupt change feature set; the illumination abrupt change feature set uses an extreme value calculation function to obtain the local regional maxima to generate an extreme value feature map, and uses a linear normalization operator to compress the numerical space to output the illumination distortion gradient value; The depth point cloud set is downsampled to output a sparse point cloud tensor. The sparse point cloud tensor is then processed by a Euclidean clusterer to calculate spatial distance, resulting in a point cloud cluster set. The point cloud cluster set is then used to calculate the number of points per unit volume using a local density estimation operator, resulting in a point cloud density distribution matrix. The point cloud density distribution matrix is ​​then processed by an outlier detector to extract the spatial coordinates of density anomalies, resulting in an outlier speckle coordinate set. Based on the Shannon entropy formula, the outlier speckle coordinate set is subjected to logarithmic probability distribution calculation within a preset discrete spatial grid to generate a speckle information entropy matrix. The speckle information entropy matrix is ​​then processed through global convergence calculation to extract a global speckle information entropy scalar. The global speckle information entropy scalar is input into a smoothing mapping function to output speckle noise entropy values.

3. The UAV dynamic obstacle avoidance method based on multi-sensor fusion according to claim 1, characterized in that, The specific generation process of the registered image sequence and registration point cloud includes: clock matching between the image sequence and the depth point cloud to output a synchronized image sequence and a synchronized point cloud; temporal optimization of the pose state parameters based on the timestamp of the synchronized image sequence to output a synchronized reference pose matrix; inputting the synchronized reference pose matrix into a quaternion analyzer to solve the transient attitude fluctuations of the machine body, outputting a rotational perturbation matrix, and fusing it with the fixed installation offset of the heterogeneous sensor through a coordinate transformer to generate an extrinsic parameter transformation matrix; inputting the synchronized image sequence into a camera calibration model to perform optical distortion correction, outputting a distortion-free image sequence; obtaining a pre-calibrated intrinsic parameter feature matrix, combining it with the synchronized point cloud, the extrinsic parameter transformation matrix, and the distortion-free image sequence into a spatial reprojection model for mapping transformation, extracting the common-view frustum region, and outputting the registered image sequence and registration point cloud.

4. The UAV dynamic obstacle avoidance method based on multi-sensor fusion according to claim 1, characterized in that, The specific generation process of the visual feature set and radar feature set includes: the registered image sequence is input into a multi-scale residual convolutional network to extract multi-level spatial topological information and output a visual semantic feature tensor; the visual semantic feature tensor is used to calculate pixel-level weights through a spatial attention layer to output a visual feature set; the registered point cloud is input into a spatial bounding box generator for recursive partitioning to output a three-dimensional subspace array, and local geometric topological information is extracted through a sparse three-dimensional convolutional kernel to output a three-dimensional geometric feature matrix; the three-dimensional geometric feature matrix is ​​input into a bird's-eye view projection operator to perform a height-dimensional orthogonal compression transformation and output a radar feature set.

5. The UAV dynamic obstacle avoidance method based on multi-sensor fusion according to claim 1, characterized in that, The specific generation process of the spatial probability occupancy map includes: dimensionality reduction of the feature channels of the joint heterogeneous feature tensor through a multilayer perceptron model to output two-dimensional compressed features; the two-dimensional compressed features are used to calculate discrete height layer probabilities to output a height probability distribution vector set; the two-dimensional compressed features and the height probability distribution vector set are reconstructed through orthogonal dimensions to output initial voxel features; the initial voxel features are used to expand the spatial resolution using a three-dimensional deconvolutional network model to output a three-dimensional upsampled tensor; the three-dimensional upsampled tensor is input into a normalized exponential activation function to output a spatial probability distribution matrix; the spatial probability distribution matrix is ​​used to output an initial continuous occupancy map using a voxel occupancy grid generator; the initial continuous occupancy map is then diffusely smoothed to output the spatial probability occupancy map.

6. The UAV dynamic obstacle avoidance method based on multi-sensor fusion according to claim 1, characterized in that, The specific generation process of the airframe attitude and thrust control commands includes: solving the multi-rotor thrust boundary using dynamic equations based on flight speed and attitude parameters, and outputting a feasible control input set; inputting the feasible control input set into the state transition model to output a three-dimensional spatial sampling node sequence; calculating the collision expectation risk using a Bayesian cost function based on the three-dimensional spatial sampling node sequence and the spatial probability occupancy map, and generating a continuous generation value matrix; iteratively solving the minimum cost sequence using a gradient descent optimization solver to output the optimal spatial avoidance trajectory; inputting the optimal spatial avoidance trajectory into an inverse dynamics solver to extract lateral and vertical acceleration components, and generating airframe attitude and thrust control commands; transmitting the airframe attitude and thrust control commands to the underlying flight controller to convert them into independent control signals for the quadcopter motors to execute physical obstacle avoidance actions.

Images