Unmanned aerial vehicle obstacle avoidance method fusing vision and laser in weak light environment

Abstract

The invention relates to the technical field of unmanned aerial vehicle autonomous obstacle avoidance, and discloses an unmanned aerial vehicle obstacle avoidance method fusing vision and laser in a weak light environment, and the method comprises the following steps: obtaining a visible light image, depth information, sparse laser point cloud and real-time illumination intensity; processing the image and the point cloud data, and fusing to generate the position, size and category of the obstacle; based on the obstacle information, constructing a risk assessment model, and assessing a flight situation risk level; according to the risk level, generating an obstacle avoidance behavior sequence including speed, height and scanning adjustment; based on the obstacle avoidance behavior sequence, the flight path of the unmanned aerial vehicle is updated and flight is controlled; during flight, the visual depth and the laser point cloud are utilized to iterate and self-calibrate the external parameters of the sensor online. According to the method, the exposure time and the gain are adaptively adjusted, the sparse point cloud is acquired in combination with the laser radar, and deep processing and fusion are performed on the image and the point cloud data in the multi-scale cross-modal fusion network, so that the perception performance of a single sensor in a weak light environment is effectively overcome.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of unmanned aerial vehicle autonomous obstacle avoidance, in particular to an unmanned aerial vehicle obstacle avoidance method fusing vision and laser in a weak light environment. BACKGROUND

[0002] In recent years, unmanned aerial vehicle technology has developed rapidly and has shown wide application potential in many fields such as military and civilian use. In particular, in the aspects of inspection, surveying and mapping, agricultural plant protection and logistics distribution, unmanned aerial vehicles have become important tools due to their efficiency and flexibility. However, as the application scenarios become more complex, the demand for unmanned aerial vehicles to operate in low visibility or insufficient light environments is increasing.

[0003] Existing unmanned aerial vehicle obstacle avoidance systems mainly rely on various sensors carried to perceive the environment. For example, visible light cameras can capture environmental images for identifying and locating obstacles. Laser radars obtain the distance and three-dimensional spatial information of obstacles by emitting laser beams and receiving echoes. In addition, some systems may also integrate ultrasonic sensors or infrared sensors to provide near-distance obstacle detection capability.

[0004] However, in the existing technology, in a weak light environment, the image quality of the visible light camera is severely degraded when the light is insufficient, and the noise interference is large, resulting in a significant reduction in obstacle recognition accuracy and robustness. When obtaining sparse point cloud data, the recognition ability for small or irregular obstacles is limited. At the same time, the data of a single sensor is often difficult to fully represent a complex environment. In addition, the existing risk assessment model is usually simple and does not fully consider the motion characteristics of the dynamic obstacles and the state of the unmanned aerial vehicle itself, resulting in inaccurate risk assessment. Therefore, the present application provides an unmanned aerial vehicle obstacle avoidance method fusing vision and laser in a weak light environment to solve the deficiencies in the prior art. SUMMARY

[0005] In view of the deficiencies of the prior art, the purpose of the present application is to provide an unmanned aerial vehicle obstacle avoidance method fusing vision and laser in a weak light environment, which solves the problems of poor perception data quality, inaccurate obstacle avoidance decision and difficulty in multi-sensor cooperation in the existing unmanned aerial vehicle obstacle avoidance process in a weak light environment.

[0006] To achieve the above purpose, the present application is realized by the following technical scheme: The first aspect of the present invention provides a method for obstacle avoidance of a UAV in a low-light environment that integrates vision and laser, comprising the following steps: in the environmental perception data acquisition stage, a low-light visual perception device, a lidar device, and a light intensity sensor carried by the UAV work together. The low-light visual perception device is used to acquire visible light images and depth information, wherein the visible light image adaptively adjusts the exposure time and gain according to the real-time light intensity; the lidar device is used to acquire a sparse laser point cloud, which contains the three-dimensional coordinates and echo intensity of each point; and the light intensity sensor is responsible for acquiring real-time light intensity data. Through the above-mentioned devices, the system can comprehensively acquire environmental perception data including visible light images, depth information, sparse laser point clouds, and real-time light intensity.

[0007] During the multi-physical modality feature processing and fusion stage, the acquired visible light image data and sparse laser point cloud data are processed to extract visual features and laser features. Specifically, the visible light image data undergoes deep learning-based image denoising, contrast enhancement, and HDR reconstruction to generate enhanced image features; the sparse laser point cloud data undergoes point cloud clustering, normal estimation, and key point extraction to generate geometric features. These visual features and laser features are then input into a multi-scale cross-modal fusion network for fusion. The network adopts a multi-branch pyramid structure and performs cross-attention fusion at different feature levels to ultimately generate the location, size, and category information of the obstacle.

[0008] In the scenario and risk assessment phase, the system constructs a comprehensive risk assessment model based on the obstacle location, size, and category generated by the fusion network, as well as the drone's own real-time position, speed, and battery level. This model considers the distance between the drone and static obstacles and the collision risk with dynamic obstacles. The comprehensive risk assessment model determines the risk level of the current flight scenario by calculating the weighted sum of multiple risk factors. Specifically, the scenario risk is calculated as follows: Risk(o k )=w1·exp(-α1·d k )+w2·exp(-α2·σ k )+w3·f dyn (o k ,v d ); Where, d k It is a drone and an obstacle. k distance; σ k It is an obstacle k Perceived uncertainty; f dyn (o k ,v d ) is related to the dynamic obstacle o k and the drone speed v dThe related dynamic risk term; w1, w2, w3, a1, a2 are preset weight coefficients.

[0009] By introducing a dynamic collision probability formula, the dynamic risk is more accurately measured: In the formula, P collision (o k ,v d ,v k ) represents the probability of collision between the UAV and the dynamic obstacle o k at a future time; pos d (t) is the predicted position of the UAV at the future time t, determined by the current position and speed v d of the UAV; pos k (t) is the predicted position of the dynamic obstacle o k at the future time t, determined by the current position and speed v k of the obstacle; ||·|| represents the Euclidean distance; τ is a scale parameter for adjusting the speed of probability decay, representing the uncertainty of the predicted position.

[0010] In the intelligent behavior generation stage, according to the risk level evaluation result determined by the risk evaluation module, the UAV obstacle avoidance behavior sequence is generated through the situation-risk adaptive decision network. The situation-risk adaptive decision network adopts a deep reinforcement learning framework, taking the risk level evaluation result and the UAV situation state as input. A pre-trained policy network outputs the target speed, flight height and sensor scanning mode adjustment amount of the UAV, thereby forming the obstacle avoidance behavior sequence. The adjustment of the sensor scanning mode includes increasing the laser radar scanning frequency, adjusting the light sensitivity or capture mode of the weak light vision perception device. Specifically, the state-action mapping of the situation-risk adaptive decision network is represented as: A=π(S); In the formula, S=(P d ,Q d ,v d ,O,Risk(O)) is the UAV situation state, including the real-time pose P d (position) and Q d (orientation) of the UAV, the speed vector v d of the UAV, the obstacle detection result O and the evaluation risk Risk(O); A is the obstacle avoidance behavior vector, including the target speed adjustment amount, the flight height adjustment amount and the sensor scanning mode adjustment amount; π(·) is the pre-trained policy network.

[0011] In the path planning and flight control stage, based on the generated obstacle avoidance behavior sequence, the system updates the UAV flight path in real time. The path planning minimizes the flight path length under the condition of meeting the safety distance constraint between the UAV and the obstacles, and considers the real-time position, size and motion trajectory of the obstacles, as well as the kinematics and dynamics constraints of the UAV itself. Subsequently, the flight control system generates precise control instructions according to the updated flight path, including the pitch angle, roll angle, yaw angle and thrust of the UAV, to realize autonomous obstacle avoidance.

[0012] In the multi-modal collaborative self-calibration stage, during the flight of the UAV, the depth information provided by the weak light visual perception device and the point cloud data provided by the laser radar device are continuously utilized, and the data registration is carried out in combination with the inertial measurement unit (IMU) data of the UAV. By minimizing the data deviation or re-projection error between multiple sensors, the system iteratively updates the external parameters between the weak light visual perception device and the laser radar device online to ensure that all sensor data are fused in the same accurate coordinate system.

[0013] The second aspect of the present application provides a UAV obstacle avoidance system fusing vision and laser in a weak light environment, comprising: A perception module for acquiring environmental perception data including visible light images, depth information, sparse laser point clouds and real-time illumination intensity through a weak light visual perception device, a laser radar device and an illumination intensity sensor.

[0014] A data processing and fusion module for receiving the environmental perception data output by the perception module and constructing a multi-scale cross-modal fusion network for generating position, size and category information of obstacles. Based on the visible light image data and sparse laser point cloud data, the module extracts visual features and laser features and inputs these features into the fusion network to generate spatial information of obstacles.

[0015] A risk assessment module for constructing a comprehensive risk assessment model based on the position, size and category of obstacles output by the data processing and fusion module, and the real-time pose, speed and battery level of the UAV itself, to determine the risk level of the current flight situation. The module quantifies the flight risk by calculating risk factors such as the distance to static obstacles and the collision risk with dynamic obstacles, and then performing weighted summation.

[0016] A behavior generation module for constructing a situation-risk adaptive decision network to generate a UAV obstacle avoidance behavior sequence based on the risk level evaluation results determined by the risk assessment module, the behavior sequence including target speed adjustment, flight height adjustment and sensor scanning mode adjustment. The module uses a deep reinforcement learning framework, taking the risk level and the UAV situation state as input, and outputs specific obstacle avoidance instructions through a pre-trained policy network.

[0017] a flight control module for updating the flight path of the UAV in real time based on the obstacle avoidance behavior sequence generated by the behavior generation module, and controlling the UAV to fly along the updated flight path. This module calculates the minimum path through an optimization algorithm, and generates precise pitch, roll, yaw and thrust instructions based on the obstacles and the UAV's own state.

[0018] a multi-modal self-calibration module for continuously performing online iterative self-calibration of multi-sensor extrinsic parameters using the depth information provided by the weak light visual perception device and the point cloud data provided by the laser radar device during the flight of the UAV. This module ensures the accuracy of sensor data fusion by data registration and minimizing data deviation or re-projection error.

[0019] In summary, the present application includes at least one of the following beneficial technical effects: 1. The present application adaptively adjusts the exposure time and gain through the weak light visual perception device, combines with the laser radar to obtain sparse point cloud, and performs deep processing and fusion on image and point cloud data in a multi-scale cross-modal fusion network, effectively overcoming the problem of limited perception performance of single sensor in weak light environment. This multi-modal fusion mechanism can obtain more comprehensive environmental information and generate accurate obstacle position, size and category, thereby improving the environmental perception accuracy and robustness of the UAV in complex weak light scenarios.

[0020] 2. The present application constructs a comprehensive risk assessment model, which not only considers the distance between the UAV and the obstacles and the dynamic collision risk, but also comprehensively assesses the current flight risk level according to the UAV's own state. On this basis, the situation-risk adaptive decision network can dynamically generate an obstacle avoidance behavior sequence including flight speed, height and sensor scanning mode according to the real-time risk level and situation state, so that the UAV can flexibly adjust the obstacle avoidance strategy according to the actual danger level to avoid overreaction or insufficient reaction.

[0021] 3. The present application introduces a multi-modal collaborative self-calibration mechanism that continuously uses visual depth information and laser point cloud data for online iterative sensor extrinsic parameter calibration during the flight of the UAV. By minimizing the data deviation or re-projection error between multiple sensors, this mechanism can dynamically correct the relative pose between sensors, effectively solving the problem of calibration drift that may occur in sensors during long-term use or vibration. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a method step flowchart of the present application; Figure 2 is a fusion perception and data processing flowchart of the present application; Figure 3is a risk assessment and obstacle avoidance decision-making process schematic diagram of the present application; Figure 4 is a weak light environment fusion vision and laser unmanned aerial vehicle obstacle avoidance system architecture diagram of the present application.

[0023] Among them, 10, perception module; 20, data processing and fusion module; 30, risk assessment module; 40, behavior generation module; 50, flight control module; 60, multi-modal self-calibration module. DETAILED DESCRIPTION

[0024] The following will be combined with the Figure 1 -Appendix Figure 4 , the present application is further described in detail.

[0025] Please refer to the Figure 4 , Figure 4 is a weak light environment fusion vision and laser unmanned aerial vehicle obstacle avoidance system architecture diagram according to an embodiment of the present application, a weak light environment fusion vision and laser unmanned aerial vehicle obstacle avoidance system is provided, including perception module 10, data processing and fusion module 20, risk assessment module 30, behavior generation module 40, flight control module 50 and multi-modal self-calibration module 60.

[0026] The perception module 10 is used to perform the function of environment perception data acquisition, and the acquired environment perception data is output to the data processing and fusion module 20. The data processing and fusion module 20 processes and fuses the received environment perception data, and outputs the generated obstacle position, size and category information to the risk assessment module 30. The risk assessment module 30 calculates and determines the risk level of the flight situation according to the received obstacle information and the state of the unmanned aerial vehicle itself, and outputs the result to the behavior generation module 40. The behavior generation module 40 generates an obstacle avoidance behavior sequence based on the risk assessment result, and outputs the sequence to the flight control module 50. The flight control module 50 updates the flight path and generates control instructions according to the received obstacle avoidance behavior sequence, so as to realize the autonomous obstacle avoidance of the unmanned aerial vehicle. At the same time, the multi-modal self-calibration module 60 works continuously in the whole flight process, and the data of the perception module is used to calibrate the sensor external parameter online.

[0027] Refer to the Figure 1 -Appendix Figure 3 , the present application provides a weak light environment fusion vision and laser unmanned aerial vehicle obstacle avoidance method, including the following steps: S1, environment perception data acquisition, through a variety of sensor devices carried by the unmanned aerial vehicle, acquiring various key data of the flight environment; S2, multi-physical modal feature processing and fusion, pre-processing, feature extraction are carried out on the acquired original perception data, and the features of different modalities are integrated through the fusion network; S3, Situation and Risk Assessment, builds a risk assessment model based on the fused obstacle information and the drone’s own status and calculates the risk level of the current flight situation; S4, intelligent behavior generation, uses the decision network to generate the obstacle avoidance behavior instruction sequence of the UAV in the current situation based on the assessed risk level; S5, path planning and flight control, updates the UAV’s flight path in real time based on the generated obstacle avoidance behavior sequence, and generates corresponding control instructions to guide the UAV’s flight; S6, multimodal collaborative self-calibration, is performed continuously during the UAV flight and is used to iteratively update the external parameters between multiple sensors online to maintain the accuracy of data fusion.

[0028] During the environmental perception data acquisition phase, the drone acquires perception data of the flight environment through the onboard perception module 10. The perception module 10 includes at least a low-light vision perception device, a laser radar device, and a light intensity sensor.

[0029] The low-light visual perception device is used to acquire visible light images and depth information. Under varying light intensities, the device adaptively adjusts its exposure time and gain to ensure that visible light images with sufficient detail can be captured even in low-light environments. Simultaneously, the device acquires depth information corresponding to the visible light image, providing three-dimensional structural data of the scene.

[0030] The LiDAR device is used to acquire a sparse laser point cloud. This sparse laser point cloud contains the three-dimensional coordinates and echo intensity information of each point. The three-dimensional coordinates accurately represent the spatial position of objects in the environment, while the echo intensity can be used to help distinguish obstacles of different materials.

[0031] The light intensity sensor is used to obtain real-time light intensity data. This light intensity data can be used as input for environmental conditions to guide exposure and gain adjustments for low-light visual perception devices, and can also serve as an auxiliary environmental parameter for subsequent risk assessment and behavior generation stages.

[0032] The perception module 10 can comprehensively acquire multi-physical modal environment perception data including visible light images, depth information, sparse laser point clouds and real-time light intensity. These data will serve as the basis for subsequent data processing and fusion.

[0033] In the multi-physical modality feature processing and fusion stage, the data processing and fusion module 20 processes the acquired visible light image data and sparse laser point cloud data to extract the features of each modality and then fuse them.

[0034] The visible light image data is pre-processed to generate enhanced image features. This pre-processing step includes deep learning based image denoising, contrast enhancement, and HDR reconstruction. Image denoising aims to eliminate the noise commonly seen in low-light environments, such as filtering the image through a convolutional neural network (CNN). Contrast enhancement adjusts the brightness range of image pixels, making image details clearer. HDR (High Dynamic Range) reconstruction handles areas with large brightness differences in the image, merging them into one image with a wider dynamic range, thus preserving more bright and dark details. Through these processes, the original visible light image is converted into enhanced image features more suitable for feature extraction.

[0035] Sparse laser point cloud data is processed to generate geometric features. This processing step includes point cloud clustering, normal estimation, and key point extraction. Point cloud clustering groups spatially adjacent points to identify independent objects or regions. Normal estimation calculates the surface normal vector of each point, providing local geometric direction information. Key point extraction identifies representative and stable feature points from the point cloud, which are crucial for subsequent registration and fusion.

[0036] These extracted enhanced image features and geometric features are input into a multi-scale cross-modal fusion network for fusion. The multi-scale cross-modal fusion network adopts a multi-branch pyramid structure and performs cross-attention mechanism fusion at different feature levels. This network structure allows capturing feature information at different scales and achieving information interaction and complementarity between different modal features through cross-attention mechanism. For example, visual features can enhance the recognition of object boundaries in point clouds, while the accurate depth information of point clouds can correct the uncertainty of visual features in depth. The fusion process can be represented as: fused F image = FusionNet(F pointcloud ); where F fused represents the fused feature representation; F image represents the enhanced image features extracted from visible light image data; F pointcloud represents the geometric features extracted from sparse laser point cloud data; and FusionNet(·,·) represents the multi-scale cross-modal fusion network.

[0037] Through this fusion method, the network can comprehensively utilize the advantages of both visual and laser modalities to ultimately generate the position, size, and category information of obstacles. For example, the category recognition of obstacles can rely more on the texture and shape features of images, while the accurate position and size are mainly provided by point cloud data. The fusion result can be represented as an obstacle set O = {o1, o2, …, o N}, where each obstacle o kmay be defined as: o k = (x k , y k , z k , l k , w k , h k , c k ); where (x k , y k , z k ) represents the three-dimensional center coordinates of the obstacle o k ; (l k , w k , h k ) represents the length, width, and height dimensions of the obstacle o k ; c k represents the category of the obstacle o k (e.g., pedestrian, vehicle, tree, etc.).

[0038] In the situation and risk assessment phase, the risk assessment module 30 constructs a comprehensive risk assessment model based on the obstacle position, size, and category information output by the data processing and fusion module 20, combined with the real-time pose, speed, and battery level of the UAV, etc. parameters, to determine the risk level of the current flight situation.

[0039] The comprehensive risk assessment model aims to quantify the potential danger degree of the current flight state of the UAV. The model considers multiple key risk factors, including the distance between the UAV and static obstacles, and the possibility of collision between the UAV and dynamic obstacles. The real-time pose (including position P d and attitude Q d ) of the UAV, speed v d , and battery level, etc. information provides support data for the risk assessment of the UAV itself state.

[0040] Specifically, the comprehensive risk assessment model determines the risk level of the current flight situation by calculating the weighted sum of multiple risk factors. For each detected obstacle o k , the system calculates its risk value to the UAV. This risk value integrates the obstacle distance, and the dynamic interaction with dynamic obstacles. The calculation method of the situation risk is as follows: Risk(o k ) = w1·exp(-α1·d k ) + w2·exp(-α2·σ k ) + w3·f dyn (o k , v d ); where d k is the distance between the UAV and the obstacle ok distance; σ k It is an obstacle k Perceived uncertainty; f dyn (o k ,v d ) is related to the dynamic obstacle o k and the drone speed v d Related dynamic risk items; w1, w2, w3, α1, α2 are preset weight coefficients.

[0041] The risk assessment module 30 converts complex flight scenarios into quantified risk levels, providing a basis for decision-making in the subsequent intelligent behavior generation phase. This risk level output is continuously updated to reflect the drone's changing environment and state during flight.

[0042] During the intelligent behavior generation phase, the behavior generation module 40 constructs a context-risk adaptive decision network based on the risk level assessment results determined by the risk assessment module 30 to generate a sequence of obstacle avoidance behaviors for the drone. This sequence includes adjustments to flight speed, altitude, and sensor scanning patterns.

[0043] The context-risk adaptive decision network adopts a deep reinforcement learning framework. The framework takes the risk level assessment results and the drone's context state as input and outputs the drone's target speed, flight altitude, and sensor scanning mode adjustment through a pre-trained policy network. The drone's context state S includes the drone's real-time pose P d (Position) and Q d (attitude), UAV velocity vector v d , obstacle detection result O and estimated risk Risk(O). Based on these inputs, the policy network learns how to choose the best obstacle avoidance behavior in different risk scenarios.

[0044] Specifically, the state-action mapping of the context-risk adaptive decision network can be expressed as: A=π(S); Where, S=(P d ,Q d ,v d ,O,Risk(O)) is the current situation state vector of the UAV; P d Indicates the three-dimensional position coordinates of the drone; Q d Represents the attitude of the drone (for example, through quaternions or Euler angles); v d represents the velocity vector of the UAV in three-dimensional space; O represents the obstacle detection result generated by the data processing and fusion module (including the position, size, and category of each obstacle); Risk(O) represents the overall risk level of the current flight situation calculated by the risk assessment module.

[0045] A is the obstacle avoidance behavior vector, representing the obstacle avoidance instruction output by the decision network. A contains target speed adjustment amount Δv, flight height adjustment amount Δh, and sensor scanning mode adjustment amount M scan .

[0046] π(·) is the pre-trained policy network function, which maps the current situation state to the optimal obstacle avoidance behavior.

[0047] Each adjustment amount in the obstacle avoidance behavior vector A has a clear physical meaning. For example, when a high risk is detected, the target speed adjustment amount Δv may indicate that the UAV decelerates or performs a lateral maneuver; the flight height adjustment amount Δh may indicate that the UAV climbs or descends to avoid obstacles; the sensor scanning mode adjustment amount M scan is used to optimize the performance of the perception system. The adjustment of the sensor scanning mode includes increasing the scanning frequency of the laser radar, adjusting the light sensitivity or capture mode of the low-light vision perception device. For example, in the face of complex obstacles or extremely low light conditions, the system can instruct the laser radar to increase the scanning frequency to obtain denser point cloud data, or instruct the low-light vision perception device to increase the light sensitivity to enhance the image brightness, thereby providing more detailed or clearer perception input for subsequent decision-making.

[0048] In addition, in order to realize the generation of the obstacle avoidance behavior sequence, the decision network usually needs a reward function to guide its learning process. The reward function aims to quantify the pros and cons of each obstacle avoidance behavior, prompting the network to learn behaviors that can maximize safety and flight efficiency. A simplified reward function can be defined as: R(S,A) = -λ1·Risk(O) - λ2·||Δv|| - λ3·|Δh| + λ4·SafetyMargin; where R(S,A) represents the reward value obtained by executing action A in state S; λ1, λ2, λ3, λ4 are weight coefficients used to balance the importance of different terms; Risk(O) is the current situation risk. The negative value of this term indicates that the higher the risk, the lower the reward, encouraging risk-averse behavior; ||Δv|| is the norm of the speed adjustment amount, indicating the amplitude of the speed adjustment. The negative value indicates that excessive speed adjustment will reduce the reward, encouraging smooth flight; |Δh| is the absolute value of the flight height adjustment amount. The negative value indicates that excessive height adjustment will reduce the reward; SafetyMargin represents the safety margin of the UAV from the nearest obstacle, and the positive value indicates the reward, encouraging maintaining a safe distance.

[0049] The behavior generation module 40 can generate a complete, situation-adaptive UAV obstacle avoidance behavior sequence according to the real-time situation and risk level, providing clear execution instructions for the flight control module 50.

[0050] In the path planning and flight control stage, the flight control module 50 updates the UAV flight path in real-time based on the obstacle avoidance behavior sequence generated by the behavior generation module 40, and controls the UAV to fly along the updated flight path.

[0051] The path planning part is realized through an optimization algorithm. This optimization algorithm minimizes the length of the flight path while ensuring a safe distance between the UAV and obstacles. The real-time position, size, and motion trajectory of obstacles provided by the data processing and fusion module 20 are fully considered in the path planning process. In addition, the path planning also incorporates the kinematic and dynamic constraints of the UAV itself, such as maximum speed, maximum acceleration, maximum climb / descent rate, and maximum turning radius, etc., to ensure that the planned path is actually executable by the UAV.

[0052] When a new flight path is planned or updated, the flight control system will generate precise control instructions according to the path. These control instructions include the pitch angle, roll angle, yaw angle, and thrust of the UAV. The flight control system sends these instructions to the actuators of the UAV (such as motors and rudders) to achieve precise adjustment of the UAV's attitude and speed. By continuously monitoring the deviation between the actual flight state of the UAV and the planned path, the flight control system can correct the control instructions in real time to ensure that the UAV can fly along the planned safe path, thereby achieving autonomous obstacle avoidance. The control process is closed-loop, that is, the flight control system will continuously receive state feedback from the UAV and adjust the output according to the feedback to respond to disturbances and environmental changes during flight.

[0053] In the multi-modal collaborative self-calibration stage, the multi-modal self-calibration module 60 runs continuously during the UAV flight process, using the depth information provided by the weak light vision perception device in the perception module 10 and the point cloud data provided by the laser radar device, and combining with the inertial measurement unit (IMU) data of the UAV for data registration.

[0054] The purpose of this self-calibration process is to iteratively update the extrinsic parameters between the weak light vision perception device and the laser radar device online to ensure that all sensor data is always fused in the same accurate coordinate system. The accuracy of the sensor extrinsic parameters is the basis for the accuracy of multi-modal data fusion. Since the UAV may be affected by vibration, temperature change or other factors during flight, the relative pose between sensors may drift slightly, causing data registration error to increase. The invention can dynamically correct these drifts through the online self-calibration mechanism.

[0055] Specifically, self-calibration is achieved by minimizing the data bias or re-projection error between multiple sensors. For example, the point cloud data of a lidar can be projected onto the visual image plane, and then the geometric error between the projected points and the corresponding feature points in the image is calculated. This error can be defined as an optimization objective function, and the sensor extrinsic parameters that minimize this error are solved by an iterative optimization algorithm.

[0056] Consider a typical re-projection error minimization problem, which aims to find the optimal camera-to-lidar transformation matrix T CL (containing rotation R CL and translation t CL ) such that the positions of lidar points projected onto the image are as close as possible to the positions of feature points in the image. This can be expressed as minimizing the following error function: In the formula, E(R CL ,t CL ) represents the total re-projection error, which is the objective function to be minimized; N represents the number of corresponding point pairs used for calibration; P lidar,i represents the i-th three-dimensional point in the lidar coordinate system; p image,i represents the i-th two-dimensional feature point (pixel coordinates) in the camera image plane corresponding to P lidar,i ; Proj(·) represents the projection function from three-dimensional points to two-dimensional image planes, which includes the intrinsic matrix of the camera; R CL represents the rotation matrix from the lidar coordinate system to the camera coordinate system, which is the extrinsic parameter to be optimized; t CL represents the translation vector from the lidar coordinate system to the camera coordinate system, which is the extrinsic parameter to be optimized; ||·|| 2 represents the square of the Euclidean distance, which is used to quantify the projection error.

[0057] The multi-modal self-calibration module 60 can update and maintain the accuracy of the extrinsic parameters between the low-light visual perception device and the lidar device in real time. This ensures that the data processing and fusion module 20 can obtain highly registered sensor data under any flight conditions, thereby improving the accuracy of obstacle identification and positioning, and further improving the long-term stability and reliability of the entire obstacle avoidance system.

[0058] The embodiments of the specific implementation are the preferred embodiments of the present application, and are not intended to limit the protection scope of the present application, wherein the same parts are denoted by the same reference numerals. Therefore: any equivalent changes made according to the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. A drone obstacle avoidance method that integrates vision and laser in a low-light environment, characterized by: The following steps are involved: S1. Obtain environmental perception data including visible light images, depth information, sparse laser point clouds, and real-time light intensity through the low-light visual perception device, lidar device, and light intensity sensor carried by the drone; S2. Processing the visible light image data and the sparse laser point cloud data to extract visual features and laser features, and inputting the data into a multi-scale cross-modal fusion network for fusion to generate the position, size, and category of the obstacle; S3. Based on the generated obstacle locations, sizes, and categories, construct a comprehensive risk assessment model to assess the risk level of the current flight scenario; S4. Based on the risk level assessment result, generating a UAV obstacle avoidance behavior sequence through a situation-risk adaptive decision network, wherein the behavior sequence includes flight speed adjustment, flight altitude adjustment, and sensor scanning mode adjustment; S5. Based on the generated obstacle avoidance behavior sequence, updating the UAV flight path in real time, and controlling the UAV to fly along the updated flight path; S6. During the flight of the UAV, the depth information provided by the low-light visual perception device and the point cloud data provided by the lidar device are continuously used to perform online iterative self-calibration of multi-sensor external parameters.

2. The method for avoiding obstacles in a UAV by integrating vision and laser in a low-light environment according to claim 1 is characterized in that: In step S1, the acquisition of environmental perception data including visible light images, depth information, sparse laser point clouds, and real-time light intensity includes the following steps: The light intensity sensor is used to obtain real-time light intensity data, and obtains the visible light image and the depth information through the low-light visual perception device. The visible light image adaptively adjusts the exposure time and gain under different light intensities; the laser radar device is used to obtain a sparse laser point cloud, which includes the three-dimensional coordinates and echo intensity of each point.

3. The method for avoiding obstacles in a UAV by integrating vision and laser in a low-light environment according to claim 1 is characterized in that: In step S2, the processing of the visible light image data and the sparse laser point cloud data includes the following steps: Preprocess visible light image data, including deep learning-based image denoising, contrast enhancement, and HDR reconstruction, to generate enhanced image features; Processing sparse laser point cloud data, including point cloud clustering, normal estimation and key point extraction, to generate geometric features; The multi-scale cross-modal fusion network adopts a multi-branch pyramid structure and performs cross-attention mechanism fusion at different feature levels.

4. The method for avoiding obstacles in a UAV by integrating vision and laser in a low-light environment according to claim 1, characterized in that: In step S3, evaluating the risk level of the current flight situation includes the following steps: Build a comprehensive risk assessment model based on the location, size, and type of obstacles, as well as the drone's real-time position, speed, and battery level; The comprehensive risk assessment model considers the distance between the drone and static obstacles and the collision risk with dynamic obstacles; The comprehensive risk assessment model determines the risk level of the current flight situation by calculating the weighted sum of risk factors.

5. The method for avoiding obstacles in a UAV by integrating vision and laser in a low-light environment according to claim 1, characterized in that: In step S4, generating the drone obstacle avoidance behavior sequence through the context-risk adaptive decision network includes the following steps: The situation-risk adaptive decision network adopts a deep reinforcement learning framework based on the risk level assessment results and the drone situation status as input; A pre-trained context-risk adaptive decision network outputs the adjustment amount of the drone's target speed, flight altitude, and sensor scanning pattern to generate a behavior sequence; The adjustment of the sensor scanning mode includes increasing the laser radar scanning frequency, adjusting the sensitivity or capture mode of the low-light visual perception device.

6. The method for avoiding obstacles in a UAV by integrating vision and laser in a low-light environment according to claim 1, characterized in that: In step S5, controlling the UAV to fly along the updated flight path includes the following steps: Through the optimization algorithm, the flight path length is minimized while satisfying the safety distance constraint between the drone and obstacles; Update the flight path based on the real-time position, size, and motion trajectory of obstacles, as well as the UAV's own kinematic and dynamic constraints; Through the updated flight path, control instructions are generated, including the pitch angle, roll angle, yaw angle and thrust of the drone to avoid obstacles autonomously.

7. The method for avoiding obstacles in a UAV by integrating vision and laser in a low-light environment according to claim 1, characterized in that: In step S6, the online iterative self-calibration of the multi-sensor extrinsic parameters includes the following steps: Data registration is performed using the depth information provided by the low-light visual perception device and the point cloud data provided by the lidar device in combination with the UAV inertial measurement unit (IMU) data; By minimizing the data deviation or reprojection error between multiple sensors, the external parameters between the low-light visual perception device and the lidar device are updated iteratively online.

8. The method for avoiding obstacles in a UAV by integrating vision and laser in a low-light environment according to claim 4 is characterized in that: The risk level of the flight scenario is calculated by weighted summation of risk factors: Risk(o k )=w1·exp(-α1·d k )+w2·exp(-α2·σ k )+w3·f dyn (o k ,v d ); Where, d k It is a drone and an obstacle. k distance; σ k It is an obstacle k Perceived uncertainty; f dyn (o k ,v d ) is related to the dynamic obstacle o k and the drone speed v d Related dynamic risk items; w1, w2, w3, α1, α2 are preset weight coefficients.

9. The method for avoiding obstacles in a UAV by integrating vision and laser in a low-light environment according to claim 5, characterized in that: The state-action mapping of the context-risk adaptive decision network is expressed as: A=π(S); Where, S=(P d ,Q d ,v d ,O,Risk(O)) is the drone situation state, including the drone’s real-time pose P d and Q d 、UAV velocity vector v d , obstacle detection result O and evaluation risk Risk(O), A is the obstacle avoidance behavior vector, and π(·) is the pre-trained strategy network.

10. A drone obstacle avoidance system that integrates vision and laser in a low-light environment, applied to a drone obstacle avoidance method that integrates vision and laser in a low-light environment as claimed in any one of claims 1 to 9, characterized in that: include: A perception module is used to acquire environmental perception data including visible light images, depth information, sparse laser point clouds, and real-time light intensity through a low-light visual perception device, a lidar device, and a light intensity sensor; A data processing and fusion module is used to receive the environmental perception data output by the perception module and construct a multi-scale cross-modal fusion network for generating the position, size and category of obstacles; a risk assessment module for constructing a comprehensive risk assessment model based on the location, size, and category of obstacles output by the data processing and fusion module, as well as the real-time position, speed, and battery level of the UAV itself, to determine the risk level of the current flight scenario; A behavior generation module is used to construct a situation-risk adaptive decision network based on the risk level assessment result determined by the risk assessment module to generate a UAV obstacle avoidance behavior sequence; A flight control module, configured to update the UAV's flight path in real time based on the UAV's obstacle avoidance behavior sequence generated by the behavior generation module, and control the UAV to fly along the updated flight path; The multimodal self-calibration module is used to continuously utilize the depth information provided by the low-light visual perception device and the point cloud data provided by the lidar device to perform online iterative self-calibration of multi-sensor external parameters during the flight of the drone.

Images