Unmanned aerial vehicle autonomous obstacle avoidance method and system based on vision-laser radar fusion

Abstract

The application discloses a UAV autonomous obstacle avoidance method and system based on vision-laser radar fusion, belongs to the technical field of UAV autonomous flight, and comprises the following steps: S1, collecting vision image data and laser point cloud data of the environment around a UAV through a vision sensor and a laser radar synchronously; adopting a hardware trigger and a timestamp double synchronization mechanism to control the time synchronization error of the vision image data and the laser point cloud data within + / -1 mu s; respectively carrying out denoising, enhancement and distortion correction processing on the vision image data; and adopting a bilateral filtering algorithm to remove image noise. The hardware trigger and the timestamp double synchronization mechanism are adopted to control the time synchronization error of the vision and the laser radar data within + / -1 mu s, the problems of fusion feature space dislocation and obstacle calibration distortion caused by a traditional synchronization mode are solved, a high-precision data foundation is laid for multi-sensor feature fusion, and the real-time sensing demand of high-speed flight of the UAV is met.

Description

Technical Field

[0001] This invention relates to the field of autonomous flight technology for unmanned aerial vehicles (UAVs), and more specifically, to an autonomous obstacle avoidance method and system for UAVs based on vision-lidar fusion. Background Technology

[0002] Autonomous obstacle avoidance for drones is a core technology for enabling autonomous drone operations and ensuring low-altitude flight safety. Compared to single-sensor obstacle avoidance solutions, vision-LiDAR fusion perception, which combines the semantic texture information of visual sensors with the high-precision depth information of LiDAR, has become the mainstream technical approach for autonomous obstacle avoidance in complex scenarios.

[0003] However, in practical engineering applications, this integrated obstacle avoidance solution still has the following technical shortcomings: First, existing visual and lidar data acquisition methods mostly use single software timestamp alignment or general hardware triggering, without establishing a high-precision synchronization strategy that combines software and hardware. This easily leads to significant deviations between image data and point cloud data in the time domain, directly causing spatial misalignment of fused features and distortion of obstacle position calibration, which cannot meet the real-time fusion perception requirements under high-speed flight conditions.

[0004] Second, existing visual-LiDAR feature fusion methods generally adopt a fixed weight ratio, without adaptively adjusting the fusion strategy based on environmental texture complexity, illumination changes, and the real-time operating status of the sensors. In extreme scenarios such as strong light interference, low-texture open areas, or sparse point clouds, a single weight cannot leverage the complementary advantages of the two sensors, resulting in a significant decrease in the overall environmental perception robustness.

[0005] Third, existing fusion perception networks lack dedicated feature enhancement mechanisms for small obstacles such as thin wires and branches, which can easily lead to missed detections. At the same time, trajectory prediction for dynamic obstacles often relies on a single deep learning model or a single physical motion model, without achieving model complementarity. This results in a large deviation between the predicted trajectory and the actual motion, leading to path planning failure and delayed obstacle avoidance decisions, making it difficult to ensure flight safety in complex dynamic environments.

[0006] In view of this, the present invention is proposed to solve the above-mentioned technical problems. Summary of the Invention

[0007] The purpose of this invention is to provide an autonomous obstacle avoidance method and system for unmanned aerial vehicles based on vision-lidar fusion, so as to solve the technical problems existing in the prior art, such as low sensor data synchronization accuracy, poor adaptability of fusion strategy, weak ability to identify small obstacles, and inaccurate prediction of dynamic obstacle trajectories.

[0008] To achieve the above objectives, the present invention provides the following technical solution: The primary objective of this application is to provide an autonomous obstacle avoidance method for unmanned aerial vehicles (UAVs) based on vision-LiDAR fusion, comprising the following steps: S1. Simultaneously acquire visual image data and laser point cloud data of the environment surrounding the drone through visual sensors and lidar; S2. A dual synchronization mechanism of hardware triggering and timestamp is adopted to control the time synchronization error of visual image data and laser point cloud data within ±1μs. S3. Denoising, enhancement, and distortion correction are performed on the visual image data respectively. Bilateral filtering algorithm is used to remove image noise, adaptive histogram equalization algorithm is used to enhance image contrast, and Zhang's calibration method is used for distortion correction. Denoising, filtering, and coordinate transformation are performed on the laser point cloud data. Statistical filtering algorithm is used to remove outliers, voxel filtering algorithm is used to reduce point cloud density, and rigid body transformation algorithm is used to transform the point cloud data in the laser radar coordinate system to the UAV body coordinate system. S4. Based on the environmental complexity assessment results and sensor state parameters, a dynamic hierarchical fusion strategy is adopted to adaptively adjust the fusion weights of visual features and LiDAR features to generate a fused feature map. S5. Using fused feature maps, obstacles are identified through multi-scale feature extraction and attention mechanisms, and the future trajectory of dynamic obstacles is predicted by combining deep learning models and physical models. S6. Based on the location, type, size and dynamic trajectory information of obstacles, a path planning method combining the improved A* algorithm and the dynamic window method is adopted to generate the optimal obstacle avoidance path and update it in real time. S7. Convert the optimal obstacle avoidance path into flight control commands to control the UAV to perform obstacle avoidance actions.

[0009] Furthermore, in S2, the dual synchronization mechanism of hardware triggering and timestamps is executed according to the logic of hardware triggering taking precedence and timestamp calibration and correction, including: Hardware-triggered synchronization: A 100MHz synchronization trigger signal is generated by the FPGA core board of the UAV flight control system, which simultaneously triggers the visual sensor and LiDAR to collect data. Timestamp synchronization: High-precision timestamps are added to both visual image data and laser point cloud data, and the data is further synchronized and calibrated through a linear interpolation timestamp alignment algorithm.

[0010] Furthermore, in S4, the dynamic layered fusion strategy includes: Environmental complexity assessment: Based on the texture features of the visual image and the density features of the laser point cloud, the environmental complexity assessment value C is calculated; Sensor status monitoring: Real-time monitoring of the exposure status of the vision sensor, the scanning frequency of the LiDAR, and data integrity; Dynamic adjustment of fusion weights: Based on the environmental complexity assessment value C and the sensor state parameter S, an adaptive weight adjustment algorithm is adopted to dynamically adjust the weight coefficients of visual features and LiDAR features in the feature extraction layer, feature fusion layer and feature output layer. The weight coefficient of visual features is α∈[0.1,0.9], the weight coefficient of LiDAR features is β∈[0.1,0.9], and α+β=1.

[0011] Furthermore, multi-scale feature extraction and attention mechanisms include: By performing multi-scale convolution operations on the fused feature map using a convolutional neural network, obstacle features at different scales can be extracted. By employing spatial attention and channel attention mechanisms, multi-scale features are weighted and fused to enhance the ability to represent the features of small obstacles.

[0012] Furthermore, combining deep learning models with physical models to predict the future trajectory of dynamic obstacles includes: Deep learning prediction: Using a Transformer-based trajectory prediction model, the future trajectory of dynamic obstacles is predicted using historical trajectory data; Physical model prediction: Based on Newton's laws of motion, a physical motion model of dynamic obstacles is established to predict their future trajectory; Trajectory fusion: The Kalman filter algorithm is used to fuse the deep learning prediction results and the physical model prediction results to generate the final dynamic obstacle trajectory prediction results.

[0013] Furthermore, the improved path planning method combining the A* algorithm and the dynamic window method includes: Global path planning: An improved A* algorithm is used to generate the globally optimal path with the shortest flight time and the highest obstacle avoidance safety as optimization objectives; Local path replanning: Using the dynamic window method, the global path is locally adjusted based on real-time obstacle information and UAV motion status to generate a real-time obstacle avoidance path; Path smoothing: The generated obstacle avoidance path is smoothed using a 3rd-order B-spline curve.

[0014] The second objective of this application is to provide an obstacle avoidance system for UAVs employing any of the aforementioned vision-LiDAR fusion-based autonomous obstacle avoidance methods, with the embedded main control board of the UAV flight control system as the hardware core. This system includes: The perception module includes a visual sensor and a lidar. The visual sensor and lidar are respectively connected to the acquisition board for communication. The acquisition board is connected to the main control board via the SPI bus to synchronously acquire visual image data and lidar point cloud data of the environment around the UAV. Data synchronization module: Integrated into the FPGA core board of the flight control system, it is connected to the sensing module via the ETH bus, receives the collected data transmitted by the sensing module, and adopts a dual synchronization mechanism of hardware triggering and timestamp to achieve high-precision synchronization of visual image data and laser point cloud data. Data preprocessing module: Built on the GPU coprocessing unit of the main control board, it is connected to the data synchronization module via the ETH bus, receives the data synchronized by the data synchronization module, and is used to preprocess the visual image data and laser point cloud data respectively. Dynamic hierarchical fusion module: Embedded in the neural network acceleration unit of the main control board, it is connected to the data preprocessing module via CAN bus, receives the data processed by the data preprocessing module, and uses it to generate fused feature maps based on the environmental complexity assessment results and sensor state parameters, using a dynamic hierarchical fusion strategy. Obstacle recognition and trajectory prediction module: Built on a deep learning accelerator card, it is connected to the dynamic hierarchical fusion module via the ETH bus. It receives the fusion feature map generated by the dynamic hierarchical fusion module and uses the fusion feature map to identify obstacles and predict the future trajectory of dynamic obstacles. Path planning module: Integrated into the main control board, it is connected to the obstacle recognition and trajectory prediction module via CAN bus, receives the prediction results and obstacle information from the obstacle recognition and trajectory prediction module, and is used to generate the optimal obstacle avoidance path based on the obstacle information; Flight control module: Connected to the ESC and motor actuators of the flight control system via a PWM interface, it receives path information transmitted by the path planning module and converts the optimal obstacle avoidance path into flight control commands to control the UAV to perform obstacle avoidance actions.

[0015] Furthermore, the vision sensor is a binocular vision camera with a 120°×90° field of view and 1080P resolution, a frame rate of 30fps, and is connected to the acquisition board of the perception module via a MIPI interface; the lidar is a solid-state lidar with a 360° horizontal field of view and a 25° vertical field of view, a ranging accuracy of ±2cm, a scanning frequency of 1000Hz, and is connected to the acquisition board of the perception module via a CAN FD bus.

[0016] Furthermore, the data synchronization module includes a hardware triggering unit and a timestamp synchronization unit. The hardware triggering unit is the pulse generation circuit of the FPGA core board, which simultaneously triggers the vision sensor and the LiDAR to acquire data through the 100MHz synchronous trigger signal of the flight control system. The timestamp synchronization unit is a high-precision clock module and a data calibration circuit, which achieves further synchronization calibration of the data through a linear interpolation timestamp alignment algorithm. The hardware triggering unit and the timestamp synchronization unit realize data interaction through an internal bus.

[0017] Furthermore, the dynamic hierarchical fusion module includes an environmental complexity assessment unit, a sensor status monitoring unit, and an adaptive weight adjustment unit, all of which are embedded in the neural network acceleration unit of the flight control system. Data interaction is achieved through an on-chip bus. The environmental complexity assessment unit calculates the environmental complexity assessment value based on the texture features of the visual image and the density features of the laser point cloud. The sensor status monitoring unit monitors the exposure status of the visual sensor, the scanning frequency of the LiDAR, and the data integrity in real time through the sensor status feedback interface, and outputs normalized sensor status parameters. The adaptive weight adjustment unit dynamically adjusts the fusion weights according to the environmental complexity assessment value and the sensor status parameters through an adaptive weight adjustment algorithm.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By using a dual synchronization mechanism of hardware triggering and timestamp, the time synchronization error of visual and lidar data is controlled within ±1μs, which solves the problems of spatial misalignment of fused features and distortion of obstacle calibration caused by traditional synchronization methods. This lays a high-precision data foundation for multi-sensor feature fusion and meets the real-time perception requirements of UAVs flying at high speed.

[0019] 2. The dynamic hierarchical fusion strategy based on environmental complexity and sensor status can adaptively adjust the fusion weights of visual and LiDAR features in the feature extraction, fusion, and output layers, breaking through the limitations of traditional fixed-weight fusion. It can fully leverage the complementary advantages of dual sensors in extreme scenarios such as strong light, low texture, and sparse point clouds, significantly improving the robustness of environmental perception.

[0020] 3. By integrating multi-scale feature extraction and spatial-channel dual attention mechanism, the feature representation of small obstacles such as thin wires and thin branches is specifically enhanced, effectively solving the problem of missed detection of small obstacles; at the same time, a trajectory prediction method that integrates the Transformer deep learning model and the Newton physics model is adopted, combined with the optimization results of Kalman filtering, which takes into account the correlation and physical rationality of dynamic trajectory prediction, greatly improving the prediction accuracy and solving the core defect of large prediction deviation of single model.

[0021] 4. By adaptively adjusting the heuristic function weights based on environmental complexity, global optimal path planning is achieved. Combined with real-time obstacle information, local path replanning is completed quickly. With the addition of 3rd-order B-spline curve smoothing, the organic unity of global path optimality and local obstacle avoidance real-time performance is achieved, taking into account both obstacle avoidance safety and UAV flight stability.

[0022] 5. Construct a modular system architecture that deeply integrates software and hardware, precisely matching core algorithms with hardware such as FPGA, neural network acceleration units, and deep learning accelerator cards. Each functional module interacts efficiently through a dedicated bus, forming a closed loop from data acquisition to flight control. This ensures that all innovative algorithms are efficiently implemented on the UAV embedded hardware platform, guaranteeing the real-time performance and engineering feasibility of the entire autonomous obstacle avoidance process. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention, but do not constitute an undue limitation of the invention. Obviously, the drawings described below are merely some embodiments, and those skilled in the art can obtain other drawings based on these drawings without creative effort. In the drawings: Figure 1 A flowchart of an autonomous obstacle avoidance method for unmanned aerial vehicles based on vision-lidar fusion provided in this application embodiment; Figure 2 This is a structural framework diagram of an autonomous obstacle avoidance system for unmanned aerial vehicles based on vision-lidar fusion, provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0025] See Figure 1 and Figure 2 As shown, the autonomous obstacle avoidance method for UAVs based on vision-LiDAR fusion includes the following steps: S1. Simultaneously acquire visual image data and laser point cloud data of the environment surrounding the drone through visual sensors and lidar; S2. A dual synchronization mechanism of hardware triggering and timestamp is adopted to control the time synchronization error of visual image data and laser point cloud data within ±1μs. S3. Denoising, enhancement, and distortion correction are performed on the visual image data respectively. A bilateral filtering algorithm with a kernel size of 5×5, a spatial domain standard deviation of 2, and a gray-level domain standard deviation of 0.5 is used to remove image noise. An adaptive histogram equalization algorithm is used to enhance image contrast. Zhang's calibration method is used for distortion correction. Denoising, filtering, and coordinate transformation are performed on the laser point cloud data. A statistical filtering algorithm is used to remove outliers. A voxel filtering algorithm with a voxel resolution of 0.05m×0.05m×0.05m is used to reduce the point cloud density. A rigid body transformation algorithm is used to transform the point cloud data in the laser radar coordinate system to the UAV body coordinate system. S4. Based on the environmental complexity assessment results and sensor status parameters, a dynamic hierarchical fusion strategy is adopted to adaptively adjust the fusion weights of visual features and LiDAR features to generate a fused feature map; the environmental complexity assessment unit calculates the environmental complexity assessment value based on the texture features of the visual image and the density features of the LiDAR point cloud; the sensor status monitoring unit monitors the exposure status of the visual sensor, the scanning frequency of the LiDAR, and the data integrity in real time. S5. Using the fused feature map, obstacles are identified through multi-scale feature extraction and attention mechanism. The future trajectory of dynamic obstacles is predicted by combining deep learning model and physical model. Multi-scale feature extraction is performed on the fused feature map through multi-scale convolution operation by convolutional neural network to extract obstacle features at different scales. The attention mechanism adopts spatial attention mechanism and channel attention mechanism to perform weighted fusion of multi-scale features to enhance the feature expression ability of small obstacles. S6. Based on the location, type, size and dynamic trajectory information of obstacles, a path planning method combining the improved A* algorithm and the dynamic window method is adopted to generate the optimal obstacle avoidance path and update it in real time. S7. The optimal obstacle avoidance path is converted into flight control commands to control the UAV to perform obstacle avoidance actions. A cascaded PID control algorithm is used to control the attitude and position of the UAV. The cascaded PID control algorithm includes a position loop PID and an attitude loop PID. The position loop controls the UAV's three-dimensional spatial position (x / y / z axes), and the attitude loop controls the UAV's roll, pitch, and yaw angles to ensure that the UAV flies stably according to the generated obstacle avoidance path. The proportional coefficient K of the position loop PID is... p =0.8, integral coefficient K i =0.1, differential coefficient Kd=0.2, proportional coefficient K of attitude loop PID p =1.0, integral coefficient K i =0.05, differential coefficient Kd=0.3; the control range of roll and pitch angles is ±30°, the control range of yaw angle is ±180°, the position control accuracy is ±0.1m, ensuring that the deviation of the UAV in tracking and avoiding obstacles is ≤0.2m.

[0026] In S2, the dual synchronization mechanism of hardware triggering and timestamps is executed according to the logic of hardware triggering taking precedence and timestamp calibration and correction, including: Hardware-triggered synchronization: A 100MHz synchronization trigger signal is generated by the FPGA core board of the UAV flight control system, which simultaneously triggers the visual sensor and lidar to collect data, ensuring that the acquisition start time deviation of the two sensors is ≤0.5μs; Timestamp synchronization: Nanosecond-level high-precision timestamps generated by the high-precision clock module of the flight control system are added to the acquisition frame headers of visual image data and laser point cloud data respectively. A linear interpolation timestamp alignment algorithm is used to further synchronize and calibrate the two data streams. For acquisition frames with time deviations, interpolation frame supplementation or frame removal processing is performed. Finally, the time synchronization error of visual image data and laser point cloud data is controlled within ±1μs.

[0027] The criteria for interpolation or frame rejection during timestamp synchronization are as follows: when the time deviation between visual image data and laser point cloud data is between 0.5μs and 1μs, linear interpolation is used to supplement the missing feature data; when the time deviation exceeds 1μs, the acquired data of that frame is directly rejected to avoid invalid data from participating in the fusion process.

[0028] In S4, the dynamic layered fusion strategy includes: Environmental complexity assessment: Based on the texture features of the visual image and the density features of the laser point cloud, the environmental complexity assessment value C is calculated. The texture features are extracted using a gray-level co-occurrence matrix with a step size of 1 and directions of 0° / 45° / 90° / 135° to extract the contrast, energy, and entropy of the image and then normalized to obtain the visual texture feature value T∈[0,1]. The point cloud density features are calculated by normalizing the number of laser point clouds per unit three-dimensional space to obtain the point cloud density feature value D∈[0,1]. The environmental complexity assessment value is calculated using the formula C=0.5×T+0.5×D, where C∈[0,1]. The larger the C value, the higher the environmental complexity. Sensor status monitoring: Real-time monitoring of the exposure status of the visual sensor (exposure value E∈[0,255]), the scanning frequency of the lidar (the ratio of the actual scanning frequency to the rated 1000Hz F∈[0,1]) and data integrity (the ratio of the effective point cloud number to the total point cloud number I∈[0,1]). E is normalized according to the formula S_E=E / 255, and F and I are directly used as normalized parameters to obtain the sensor status parameters S=(S_E,S_F,S_I), S_E,S_F,S_I∈[0,1]; Dynamic adjustment of fusion weights: Based on the environmental complexity assessment value C and the sensor state parameter S, an adaptive weight adjustment algorithm is adopted to dynamically adjust the weight coefficients of visual features and LiDAR features in the feature extraction layer, feature fusion layer, and feature output layer. The weight coefficient of visual features α∈[0.1,0.9], the weight coefficient of LiDAR features β∈[0.1,0.9], and α+β=1. The core formula of the adaptive weight adjustment algorithm is α=0.5×C+0.2×S_E+0.3×S_F, β=1-α.

[0029] Multi-scale feature extraction and attention mechanisms include: By performing multi-scale convolution operations on the fused feature map using a convolutional neural network, obstacle features at different scales can be extracted. By employing spatial attention and channel attention mechanisms, multi-scale features are weighted and fused to enhance the ability to represent the features of small obstacles.

[0030] Predicting the future trajectory of dynamic obstacles by combining deep learning models with physical models includes: Deep learning prediction: Using a Transformer-based trajectory prediction model, the future trajectory of dynamic obstacles is predicted using historical trajectory data. The input of the Transformer model is the position, velocity, and acceleration data of the obstacle for 10 consecutive frames with a collection period of 10ms, based on the coordinates of the UAV body. The output is the trajectory prediction value for the next 5-20 frames. Physical model prediction: Based on Newton's laws of motion, a physical motion model of the dynamic obstacle is established, and its future trajectory is predicted according to the obstacle's current motion state parameters (position, velocity, acceleration). Trajectory Fusion: The Kalman filter algorithm is used to fuse the deep learning prediction results and the physical model prediction results to generate the final dynamic obstacle trajectory prediction result. The state equation of the Kalman filter algorithm uses the position and velocity of the obstacle as state variables, and the observation equation uses the deep learning prediction value and the physical model prediction value as observation inputs. The process noise covariance Q is set as a diagonal matrix [0.01, 0.01], and the observation noise covariance R is set as a diagonal matrix [0.05, 0.05]. The state equation of the Kalman filter algorithm is X(k) = A X(k-1)+W(k), the observation equation is Z(k)=H X(k) + V(k); where the state variable X = [x, ,y, ]^T, where x and y are the two-dimensional positions of the obstacle in the UAV's body coordinate system. , Let be the velocity in the corresponding direction; the state transition matrix A = [[1,Δt,0,0],[0,1,0,0],[0,0,1,Δt],[0,0,0,1]], where Δt is the data acquisition period; the observation matrix H is the identity matrix; the process noise covariance Q is set as a diagonal matrix [0.01,0.01,0.01,0.01], and the observation noise covariance R is set as a diagonal matrix [0.05,0.05,0.05,0.05]; W(k) is the process noise, and V(k) is the observation noise, both of which follow a Gaussian distribution.

[0031] Improved path planning methods combining the A* algorithm and the dynamic window method include: Global Path Planning: An improved A* algorithm is adopted, with the optimization objectives of minimizing flight time and maximizing obstacle avoidance safety, to generate the globally optimal path. The improved A* algorithm optimizes the heuristic function of the traditional A* algorithm. The heuristic function is f(n) = g(n) + ω × h(n), where g(n) is the actual flight cost from the starting point to node n, h(n) is the Euclidean distance cost from node n to the destination, and ω is a dynamic weight coefficient with a value range of 0.6-1.2, adaptively adjusted according to the environmental complexity evaluation value C. When C∈[0,0.3], ω=0.6; when C∈(0.3,0.7], ω=0.9; and when C∈(0.7,1], ω=1.2. Local path replanning: The dynamic window method is adopted to make local adjustments to the global path based on real-time obstacle information and UAV motion status (speed, angular velocity, turning angle) to generate a real-time obstacle avoidance path. The speed window of the dynamic window method has a value range of 0-5m / s, the angular velocity window has a value range of -30° / s-30° / s, and the window range is adaptively cropped according to the current flight speed of the UAV.

[0032] Path smoothing: The generated obstacle avoidance path is smoothed using a 3rd-order B-spline curve. The planned waypoints of the obstacle avoidance path are used as interpolation nodes, and adjacent waypoints are used as continuous spline curve fitting nodes. Interpolation fitting is performed on discrete points on the path to ensure that the rate of change of the UAV's angular velocity is ≤10° / s, thereby improving flight stability.

[0033] The specific implementation steps for smoothing 3rd-order B-spline curves are as follows: First, set the planned waypoints of the obstacle avoidance path as interpolation nodes, and adjacent waypoints as continuous spline curve fitting nodes, denoted as P0, P1, P2, ..., P... n The second step is to use the basis functions N of a 3rd-order B-spline curve. i3 Interpolating and fitting discrete nodes with (t) (t∈[0,1]) yields the continuous spline curve equation P(t)=ΣP i N i3 (t); The third step is to verify the angular velocity of the fitted curve. If the rate of change of the UAV's flight angular velocity in a certain segment of the curve exceeds 10° / s, the node spacing is readjusted and the curve is fitted again until the flight stability requirements are met.

[0034] The UAV autonomous obstacle avoidance system based on vision-LiDAR fusion uses the embedded main control board of the UAV flight control system (with an ARM Cortex-A72 processor as its core) as its hardware core. The system includes: The perception module includes a visual sensor and a lidar. The visual sensor and lidar are respectively connected to the acquisition board for communication. The acquisition board is connected to the main control board via the SPI bus to synchronously acquire visual image data and lidar point cloud data of the environment around the UAV. Data synchronization module: Integrated into the FPGA core board of the flight control system, it is connected to the sensing module via the ETH bus, receives the collected data transmitted by the sensing module, and adopts a dual synchronization mechanism of hardware triggering and timestamp to achieve high-precision synchronization of visual image data and laser point cloud data. Data preprocessing module: Built on the GPU coprocessing unit of the main control board, it is connected to the data synchronization module via the ETH bus, receives the data synchronized by the data synchronization module, and is used to preprocess the visual image data and laser point cloud data respectively. Dynamic hierarchical fusion module: Embedded in the neural network acceleration unit of the main control board, it is connected to the data preprocessing module via CAN bus, receives the data processed by the data preprocessing module, and uses it to generate fused feature maps based on the environmental complexity assessment results and sensor state parameters, using a dynamic hierarchical fusion strategy. Obstacle recognition and trajectory prediction module: Built on a deep learning accelerator card, it is connected to the dynamic hierarchical fusion module via the ETH bus. It receives the fusion feature map generated by the dynamic hierarchical fusion module and uses the fusion feature map to identify obstacles and predict the future trajectory of dynamic obstacles. Path planning module: Integrated into the main control board, it is connected to the obstacle recognition and trajectory prediction module via CAN bus, receives the prediction results and obstacle information from the obstacle recognition and trajectory prediction module, and is used to generate the optimal obstacle avoidance path based on the obstacle information; Flight control module: Connected to the ESC and motor actuators of the flight control system via a PWM interface, it receives path information transmitted by the path planning module and converts the optimal obstacle avoidance path into flight control commands to control the UAV to perform obstacle avoidance actions.

[0035] The vision sensor is a binocular vision camera with a 120°×90° field of view and 1080P resolution, a frame rate of 30fps, and is connected to the acquisition board of the perception module via a MIPI interface; the lidar is a solid-state lidar with a 360° horizontal field of view and a 25° vertical field of view, a ranging accuracy of ±2cm, a scanning frequency of 1000Hz, and is connected to the acquisition board of the perception module via a CAN FD bus.

[0036] The data synchronization module includes a hardware triggering unit and a timestamp synchronization unit. The hardware triggering unit is a pulse generation circuit of the FPGA core board, which simultaneously triggers the vision sensor and LiDAR to acquire data through the 100MHz synchronous trigger signal of the flight control system. The timestamp synchronization unit is a high-precision clock module and a data calibration circuit, which achieves further synchronization calibration of the data through a linear interpolation timestamp alignment algorithm. The hardware triggering unit and the timestamp synchronization unit realize data interaction through an internal bus.

[0037] The dynamic hierarchical fusion module includes an environmental complexity assessment unit, a sensor status monitoring unit, and an adaptive weight adjustment unit, all of which are embedded in the neural network acceleration unit of the flight control system. Data interaction is achieved through an on-chip bus. The environmental complexity assessment unit calculates the environmental complexity assessment value based on the texture features of the visual image and the density features of the laser point cloud. The sensor status monitoring unit monitors the exposure status of the visual sensor, the scanning frequency of the LiDAR, and the data integrity in real time through the sensor status feedback interface, and outputs normalized sensor status parameters. The adaptive weight adjustment unit dynamically adjusts the fusion weights according to the environmental complexity assessment value and the sensor status parameters through an adaptive weight adjustment algorithm.

[0038] The inter-layer application of the adaptive weight adjustment algorithm is as follows: In the feature extraction layer, the weight coefficient is used to allocate the proportion of extraction computing power for visual features and LiDAR features; in the feature fusion layer, the weight coefficient is used to weight and superimpose the extracted single-sensor features; in the feature output layer, the weight coefficient is used to normalize and correct the fused features to ensure the feature effectiveness of the output feature map.

[0039] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An autonomous obstacle avoidance method for unmanned aerial vehicles (UAVs) based on vision-lidar fusion, characterized in that, Includes the following steps: S1. Simultaneously acquire visual image data and laser point cloud data of the environment surrounding the drone through visual sensors and lidar; S2. A dual synchronization mechanism of hardware triggering and timestamp is adopted to control the time synchronization error of visual image data and laser point cloud data within ±1μs. S3. Denoising, enhancement, and distortion correction are performed on the visual image data respectively. Bilateral filtering algorithm is used to remove image noise, adaptive histogram equalization algorithm is used to enhance image contrast, and Zhang's calibration method is used for distortion correction. Denoising, filtering, and coordinate transformation are performed on the laser point cloud data. Statistical filtering algorithm is used to remove outliers, voxel filtering algorithm is used to reduce point cloud density, and rigid body transformation algorithm is used to transform the point cloud data in the laser radar coordinate system to the UAV body coordinate system. S4. Based on the environmental complexity assessment results and sensor state parameters, a dynamic hierarchical fusion strategy is adopted to adaptively adjust the fusion weights of visual features and LiDAR features to generate a fused feature map. S5. Using fused feature maps, obstacles are identified through multi-scale feature extraction and attention mechanisms, and the future trajectory of dynamic obstacles is predicted by combining deep learning models and physical models. S6. Based on the location, type, size and dynamic trajectory information of obstacles, a path planning method combining the improved A* algorithm and the dynamic window method is adopted to generate the optimal obstacle avoidance path and update it in real time. S7. Convert the optimal obstacle avoidance path into flight control commands to control the UAV to perform obstacle avoidance actions.

2. The autonomous obstacle avoidance method for unmanned aerial vehicles according to claim 1, characterized in that, In S2, the hardware trigger and timestamp dual synchronization mechanism is executed according to the logic of hardware trigger priority and timestamp calibration and correction, including: Hardware-triggered synchronization: A 100MHz synchronization trigger signal is generated by the FPGA core board of the UAV flight control system, which simultaneously triggers the visual sensor and LiDAR to collect data. Timestamp synchronization: High-precision timestamps are added to both visual image data and laser point cloud data, and the data is further synchronized and calibrated through a linear interpolation timestamp alignment algorithm.

3. The autonomous obstacle avoidance method for unmanned aerial vehicles according to claim 2, characterized in that, In S4, the dynamic layered fusion strategy includes: Environmental complexity assessment: Based on the texture features of the visual image and the density features of the laser point cloud, the environmental complexity assessment value C is calculated; Sensor status monitoring: Real-time monitoring of the exposure status of the vision sensor, the scanning frequency of the LiDAR, and data integrity; Dynamic adjustment of fusion weights: Based on the environmental complexity assessment value C and the sensor state parameter S, an adaptive weight adjustment algorithm is adopted to dynamically adjust the weight coefficients of visual features and LiDAR features in the feature extraction layer, feature fusion layer and feature output layer. The weight coefficient of visual features is α∈[0.1,0.9], the weight coefficient of LiDAR features is β∈[0.1,0.9], and α+β=1.

4. The autonomous obstacle avoidance method for unmanned aerial vehicles according to claim 3, characterized in that, The multi-scale feature extraction and attention mechanism includes: By performing multi-scale convolution operations on the fused feature map using a convolutional neural network, obstacle features at different scales can be extracted. By employing spatial attention and channel attention mechanisms, multi-scale features are weighted and fused to enhance the ability to represent the features of small obstacles.

5. The autonomous obstacle avoidance method for unmanned aerial vehicles according to claim 4, characterized in that, The method of combining deep learning models and physical models to predict the future trajectory of dynamic obstacles includes: Deep learning prediction: Using a Transformer-based trajectory prediction model, the future trajectory of dynamic obstacles is predicted using historical trajectory data; Physical model prediction: Based on Newton's laws of motion, a physical motion model of dynamic obstacles is established to predict their future trajectory; Trajectory fusion: The Kalman filter algorithm is used to fuse the deep learning prediction results and the physical model prediction results to generate the final dynamic obstacle trajectory prediction results.

6. The autonomous obstacle avoidance method for unmanned aerial vehicles according to claim 5, characterized in that, The improved path planning method combining the A* algorithm and the dynamic window method includes: Global path planning: An improved A* algorithm is used to generate the globally optimal path with the shortest flight time and the highest obstacle avoidance safety as optimization objectives; Local path replanning: Using the dynamic window method, the global path is locally adjusted based on real-time obstacle information and UAV motion status to generate a real-time obstacle avoidance path; Path smoothing: The generated obstacle avoidance path is smoothed using a 3rd-order B-spline curve.

7. A UAV autonomous obstacle avoidance system based on vision-lidar fusion, employing the UAV autonomous obstacle avoidance method according to any one of claims 1-6, characterized in that, The system, with its embedded main control board as the hardware core, comprises: The perception module includes a visual sensor and a lidar. The visual sensor and lidar are respectively connected to the acquisition board, which is connected to the main control board via an SPI bus to synchronously acquire visual image data and lidar point cloud data of the environment around the UAV. Data synchronization module: Integrated into the FPGA core board of the flight control system, it is connected to the sensing module via the ETH bus, receives the collected data transmitted by the sensing module, and adopts a dual synchronization mechanism of hardware triggering and timestamp to achieve high-precision synchronization of visual image data and laser point cloud data. Data preprocessing module: Built on the GPU coprocessing unit of the main control board, it is connected to the data synchronization module via the ETH bus, receives the data synchronized by the data synchronization module, and is used to preprocess the visual image data and laser point cloud data respectively. Dynamic hierarchical fusion module: Embedded in the neural network acceleration unit of the main control board, it is connected to the data preprocessing module via CAN bus, receives the data processed by the data preprocessing module, and uses it to generate fused feature maps based on the environmental complexity assessment results and sensor state parameters, using a dynamic hierarchical fusion strategy. Obstacle recognition and trajectory prediction module: Built on a deep learning accelerator card, it is connected to the dynamic hierarchical fusion module via the ETH bus. It receives the fusion feature map generated by the dynamic hierarchical fusion module and uses the fusion feature map to identify obstacles and predict the future trajectory of dynamic obstacles. Path planning module: Integrated into the main control board, it is connected to the obstacle recognition and trajectory prediction module via CAN bus, receives the prediction results and obstacle information from the obstacle recognition and trajectory prediction module, and is used to generate the optimal obstacle avoidance path based on the obstacle information; Flight control module: Connected to the ESC and motor actuators of the flight control system via a PWM interface, it receives path information transmitted by the path planning module and converts the optimal obstacle avoidance path into flight control commands to control the UAV to perform obstacle avoidance actions.

8. The UAV autonomous obstacle avoidance system based on vision-lidar fusion according to claim 7, characterized in that, The vision sensor is a binocular vision camera with a 120°×90° field of view and 1080P resolution, a frame rate of 30fps, and is connected to the acquisition board of the perception module via a MIPI interface; the lidar is a solid-state lidar with a 360° horizontal field of view and a 25° vertical field of view, a ranging accuracy of ±2cm, a scanning frequency of 1000Hz, and is connected to the acquisition board of the perception module via a CAN FD bus.

9. The UAV autonomous obstacle avoidance system based on vision-lidar fusion according to claim 7, characterized in that, The data synchronization module includes a hardware triggering unit and a timestamp synchronization unit. The hardware triggering unit is a pulse generation circuit of the FPGA core board, which simultaneously triggers the visual sensor and the lidar to collect data through the 100MHz synchronization trigger signal of the flight control system. The timestamp synchronization unit is a high-precision clock module and a data calibration circuit, which achieves further synchronization calibration of the data through a linear interpolation timestamp alignment algorithm. The hardware triggering unit and the timestamp synchronization unit realize data interaction through an internal bus.

10. The UAV autonomous obstacle avoidance system based on vision-lidar fusion according to claim 7, characterized in that, The dynamic hierarchical fusion module includes an environmental complexity assessment unit, a sensor status monitoring unit, and an adaptive weight adjustment unit, all of which are embedded in the neural network acceleration unit of the flight control system. Data interaction is achieved through an on-chip bus. The environmental complexity assessment unit calculates the environmental complexity assessment value based on the texture features of the visual image and the density features of the laser point cloud. The sensor status monitoring unit monitors the exposure status of the visual sensor, the scanning frequency of the LiDAR, and the data integrity in real time through the sensor status feedback interface, and outputs normalized sensor status parameters. The adaptive weight adjustment unit dynamically adjusts the fusion weights according to the environmental complexity assessment value and the sensor status parameters through an adaptive weight adjustment algorithm.

Images