Unmanned aerial vehicle autonomous obstacle avoidance system and method based on millimeter wave radar and multi-mode vision fusion

Abstract

The invention discloses an unmanned aerial vehicle autonomous obstacle avoidance system and method based on millimeter wave radar and multi-mode vision fusion, and the system comprises a millimeter wave radar module which is used for obtaining the distance, speed and point cloud data of an obstacle; the multispectral vision module comprises an RGB camera and an infrared camera and is used for extracting the texture, the category and the thermal characteristics of the obstacle; the embedded AI calculation unit is used for real-time data processing and fusion; the time-space synchronization module is used for aligning timestamps of radar and visual data through hardware trigger signals and unifying space coordinates based on joint calibration and an SLAM (Simultaneous Localization and Mapping) technology; and the feature level fusion module is used for fusing the radar point cloud features and the visual semantic features by adopting a lightweight network. Aiming at single sensor defects, complex environment challenges and hardware computing power limitation, a more efficient unmanned aerial vehicle autonomous obstacle avoidance system is constructed through multi-modal hardware fusion, lightweight feature processing, a dynamic adaptive mechanism and intelligent algorithm optimization, and all-weather and full-scene autonomous operation requirements are met.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicles (UAVs), and in particular to an autonomous obstacle avoidance system and method for UAVs based on the fusion of millimeter-wave radar and multimodal vision. Background Art

[0002] Traditional drone obstacle avoidance technology relies on a single sensor and has significant limitations. Visual sensors such as cameras are significantly affected by lighting. In low light, strong light, or inclement weather such as fog, haze, rain, and snow, image clarity degrades, and target recognition accuracy plummets, making missed detections and misjudgments more likely. While millimeter-wave radar can penetrate complex environments to obtain geometric information such as obstacle distance and speed, it lacks semantic attributes and cannot distinguish obstacle type and risk level, leading to crude decision-making. Other sensors, such as ultrasonic and lidar, are difficult to adapt to small drones due to their limited detection range or high cost.

[0003] As drone applications expand into areas such as urban inspections, logistics distribution, and disaster response, the environmental challenges they face become more complex. Dynamic obstacles, such as high-speed vehicles and flying birds, have unpredictable trajectories. Traditional algorithms rely solely on current state decisions, lacking foresight into future motion trends, and can easily lead to collisions due to delayed responses. Unstructured environments present a wide variety of obstacle types, making it difficult for a single sensor to fully perceive their geometric characteristics and risk levels. When multiple drones operate collaboratively, the limited sensing range of a single unit and its isolated information can easily create information silos, leading to group obstacle avoidance conflicts.

[0004] In addition, drones are limited by payload weight and endurance, and their embedded computing units have weak computing power. Traditional multimodal fusion algorithms are computationally complex and have high latency, making it difficult to meet the millisecond-level response requirements of real-time obstacle avoidance. Therefore, an autonomous obstacle avoidance system and method for drones based on millimeter-wave radar and multimodal vision fusion is proposed. Summary of the Invention

[0005] The present invention solves the problems existing in the prior art through the following technical solutions, and the present invention comprises: a millimeter wave radar module for obtaining the distance, speed and point cloud data of obstacles;

[0006] Multispectral vision module, including RGB camera and infrared camera, used to extract the texture, category and thermal characteristics of obstacles;

[0007] Embedded AI computing unit for real-time data processing and fusion;

[0008] The spatiotemporal synchronization module aligns the timestamps of radar and visual data through hardware trigger signals and unifies the spatial coordinates based on joint calibration and SLAM technology;

[0009] Feature-level fusion module, which uses a lightweight network to fuse radar point cloud features with visual semantic features;

[0010] The decision module generates an obstacle avoidance path based on an improved path planning algorithm and adjusts the sensor confidence through dynamic weight allocation;

[0011] Environmental adaptation module switches the sensor working mode according to the real-time environmental visibility.

[0012] Furthermore, the network structure of the feature-level fusion module includes:

[0013] Radar branch, used to extract global features of point clouds;

[0014] The visual branch is used to extract local features of RGB and infrared images;

[0015] Fusion layer, used to achieve cross-modal feature interaction through attention mechanism.

[0016] Furthermore, the dynamic weight allocation formula is:

[0017] W fusion =α·S radar +β·S vision +γ·E env

[0018] Among them, W fusion is the radar signal-to-noise ratio, S radar Score visual clarity, E env is the environmental visibility factor; α, β, γ are dynamic optimization coefficients.

[0019] Furthermore, the environment adaptation module includes:

[0020] Light sensor, used to trigger the infrared camera to work preferentially in low light conditions;

[0021] The haze detection unit is used to increase the weight of the millimeter-wave radar based on image contrast.

[0022] The autonomous obstacle avoidance method for UAV based on the fusion of millimeter-wave radar and multimodal vision includes the following steps:

[0023] Synchronously collecting environmental data through the millimeter-wave radar module and the multispectral vision module;

[0024] De-noise and cluster the radar point cloud to generate candidate obstacle areas;

[0025] Perform semantic segmentation using the visual module in the candidate area to identify the type of obstacle;

[0026] The feature-level fusion module fuses radar and visual data to predict the trajectory of dynamic obstacles;

[0027] Generate a three-dimensional obstacle avoidance path based on the improved path planning algorithm of the decision module;

[0028] The sensor weights are dynamically adjusted based on the real-time visibility feedback from the environment adaptation module.

[0029] Furthermore, the path planning algorithm introduces a risk cost function:

[0030] C = λ1·D+λ2·V, where D is the obstacle distance, v is the relative speed, and λ1 and λ2 are weight coefficients.

[0031] Furthermore, the dynamic obstacle trajectory prediction includes:

[0032] Use Kalman filtering for short-term state estimation;

[0033] Modeling long-term motion dependencies via recurrent neural networks.

[0034] Furthermore, the method also includes a multi-drone collaborative obstacle avoidance step, sharing the obstacle map generated by the system through a communication module, and using a distributed algorithm to coordinate the paths of multiple drones.

[0035] Compared with existing technologies, the present invention has the following advantages: This autonomous obstacle avoidance system and method for unmanned aerial vehicles based on the fusion of millimeter-wave radar and multimodal vision uses a millimeter-wave radar to provide distance, speed, and point cloud data of obstacles, while a multispectral vision module extracts texture, category, and thermal features. The combination of the two enables comprehensive perception of the geometric and semantic information of obstacles, solving the problem of blind spots in detection by a single sensor in complex environments.

[0036] By aligning timestamps through hardware trigger signals and combining SLAM technology to unify spatial coordinates, we ensure precise alignment of radar and visual data in time and space, avoid fusion errors caused by data asynchrony, and improve obstacle positioning accuracy.

[0037] The feature-level fusion module adopts a lightweight network architecture to efficiently integrate the global features of radar point clouds with the local features of visual semantics, and realizes cross-modal feature interaction through the attention mechanism. While reducing computational complexity, it enhances the ability to understand complex scenes and meets the computing power requirements of drones for real-time obstacle avoidance.

[0038] Dynamically adjust sensor confidence based on environmental visibility. For example, infrared cameras are prioritized in low-light conditions and radar weighting is increased in foggy and hazy environments. This ensures that the system can rely on highly reliable sensor data in different environments and avoids obstacle avoidance failures caused by performance degradation of a single sensor.

[0039] The improved path planning algorithm introduces a risk cost function, comprehensively considering the obstacle distance (D) and relative speed (v), to generate a three-dimensional obstacle avoidance path that balances safety and efficiency, reducing the conservative or aggressive decision-making problems caused by traditional algorithms that rely solely on distance.

[0040] Combining Kalman filtering for short-term state estimation and recurrent neural network modeling for long-term motion dependencies, it is possible to predict the trajectory of dynamic obstacles, plan obstacle avoidance strategies in advance, and improve adaptability to unstructured environments.

[0041] Through the light sensor and haze detection unit, the sensor working mode can be switched in real time to ensure stable operation in complex environments such as day and night transitions and severe weather, thereby broadening the application scenarios of drones.

[0042] By sharing obstacle maps through the communication module and using distributed algorithms to coordinate the paths of multiple drones, group collaborative obstacle avoidance is achieved. It is suitable for scenarios such as drone formation operations and large-scale area monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0044] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. However, the protection scope of the present invention is not limited to the following embodiment.

[0045] like Figure 1 As shown, this embodiment provides a technical solution: an autonomous obstacle avoidance system for a UAV based on the fusion of millimeter-wave radar and multimodal vision, comprising: a millimeter-wave radar module for obtaining the distance, speed and point cloud data of obstacles;

[0046] Multispectral vision module, including RGB camera and infrared camera, used to extract the texture, category and thermal characteristics of obstacles;

[0047] Embedded AI computing unit for real-time data processing and fusion;

[0048] The spatiotemporal synchronization module aligns the timestamps of radar and visual data through hardware trigger signals and unifies the spatial coordinates based on joint calibration and SLAM technology;

[0049] Feature-level fusion module, which uses a lightweight network to fuse radar point cloud features with visual semantic features;

[0050] The decision module generates an obstacle avoidance path based on an improved path planning algorithm and adjusts the sensor confidence through dynamic weight allocation;

[0051] Environmental adaptation module switches the sensor working mode according to the real-time environmental visibility.

[0052] The network structure of the feature-level fusion module includes:

[0053] Radar branch, used to extract global features of point clouds;

[0054] The visual branch is used to extract local features of RGB and infrared images;

[0055] Fusion layer, used to achieve cross-modal feature interaction through attention mechanism;

[0056] For the point cloud data output by the millimeter-wave radar, a dedicated network branch is used to extract the global geometric characteristics of the obstacle, such as spatial distribution, distance, speed, etc. (such as point cloud density and motion trend), to ensure the accurate perception of the obstacle's position accuracy and relative motion state, especially providing stable spatial positioning information in low-visibility environments (such as fog and haze, at night).

[0057] The system extracts local texture, color, thermal features, and semantic categories (such as obstacle types: trees, buildings, and moving objects) from image data from RGB and infrared cameras, supplementing the attribute information and detailed features of the obstacles. This addresses the lack of semantic information in radar point clouds and prevents inappropriate identification of harmless objects (such as fluttering plastic bags) as threats.

[0058] By combining the two types of features at the fusion layer, the system possesses dual cognitive capabilities. For example, it can locate fast-moving drones using the speed information in the radar point cloud, and also identify them as dynamic obstacles through visual semantics and prioritize them, significantly improving decision-making reliability in complex scenarios.

[0059] The fusion layer uses attention mechanisms (such as self-attention and cross-attention) to dynamically filter key features of radar and vision based on the current scene and suppress invalid noise. For example, in dense obstacle environments, the spatial position feature weights of the radar point cloud are enhanced to avoid blind spots caused by visual occlusion.

[0060] In scenarios with high target classification requirements (such as distinguishing between wires and tree branches), the focus on visual semantic features is enhanced to reduce misjudgments.

[0061] There are modal differences between radar point clouds (three-dimensional sparse data) and visual images (two-dimensional dense data). The attention mechanism learns the correlation between the two (such as the mapping relationship between point cloud coordinates and image pixels) through cross-modal interaction, achieving alignment and complementarity of feature spaces, avoiding information loss caused by modal mismatch in traditional fusion methods (such as simple splicing).

[0062] A lightweight network architecture (such as lightweight convolutional neural networks and point cloud processing optimization operators) is adopted to reduce the number of model parameters and calculation time while ensuring the accuracy of feature extraction. It adapts to the low power consumption and low computing power requirements of the drone's embedded AI computing unit, ensures that the fusion processing delay is controlled at the millisecond level, and meets the response speed requirements of real-time obstacle avoidance (such as the need for rapid decision-making during high-speed flight).

[0063] Through hierarchical feature extraction (the radar branch simplifies the point cloud processing process, and the visual branch shares the basic feature extraction layer) and on-demand focusing of the attention mechanism, redundant calculation of irrelevant features is avoided, and efficient operation can still be maintained in complex scenarios, thereby improving the overall energy efficiency of the system.

[0064] The dynamic weight allocation formula is:

[0065] W fusion =α·S radar +β·S vision +γ·E env

[0066] Among them, W fusion is the radar signal-to-noise ratio, S radar Score visual clarity, E env is the environmental visibility factor; α, β, γ are dynamic optimization coefficients;

[0067] The above process directly reflects the quality of the millimeter-wave radar signal. When the signal-to-noise ratio is high (such as when there is no strong electromagnetic interference or the obstacle has high reflectivity), the radar distance / speed measurement is more accurate, and the system automatically increases its weight, strengthening its reliance on radar point cloud data.

[0068] Evaluate the imaging quality of RGB / infrared cameras using metrics such as image contrast and edge sharpness. Reduce visual weighting when the image resolution is low (e.g., lens contamination, motion blur) to avoid semantic segmentation errors caused by image noise.

[0069] By combining real-time feedback from illumination sensors and haze detection units (e.g., low illumination, high haze concentrations), the system dynamically compensates for sensor performance degradation (e.g., infrared cameras have improved advantages in darkness, and radar has better penetration than vision in haze). Through weighted calculation of these three factors, the system can quantify the reliability of each sensor in the current environment in real time, avoiding the risk of misjudgment caused by traditional fixed-weight fusion when the environment suddenly changes (e.g., relying on vision in strong haze, leading to missed detections).

[0070] The dynamic optimization coefficient can be dynamically adjusted through offline training (such as calibration based on a large amount of environmental data) or online adaptive algorithms (such as reinforcement learning), so that the weight distribution strategy can adapt to different drone models (such as the difference in sensor performance between small drones and large drones) and mission scenarios (such as the different priorities of inspection and logistics distribution), thereby improving the system's generalization ability.

[0071] Low-light scenarios: The light sensor triggers the infrared camera to operate. As the clarity of the RGB image decreases, the formula automatically reduces the weight of the visual branch and relies instead on the fusion of infrared thermal features and radar point clouds to ensure effective detection of nighttime obstacles (such as heating equipment and animals).

[0072] When the haze detection unit recognizes that visibility is reduced, the radar signal-to-noise ratio may remain stable due to its penetration advantage. At this time, the formula increases the radar weight to avoid visual detection failure caused by a sudden drop in image contrast.

[0073] Collaborative enhancement of dynamic obstacle scenarios: When high-speed moving targets (such as drones and vehicles) are detected, the radar's speed measurement accuracy (which directly affects the calculation of relative speed v) is crucial for path planning. At this time, if the radar signal-to-noise ratio is good, the weighting formula will temporarily increase the radar weight to ensure the timeliness of dynamic trajectory prediction; and if the vision also identifies the target as a high-risk object (such as a metal structure obstacle), its semantic feature weight will also be indirectly enhanced through the clarity score, achieving a dual weighted fusion of geometric information and risk level.

[0074] To avoid sensor conflicts and information redundancy, traditional fixed-weight fusion may cause decision delays or errors due to sensor data conflicts (such as radar detecting an obstacle but visually misjudging it as background). Dynamic weight allocation automatically suppresses interference from low-reliability sensors through real-time trust calculation (such as reducing the M weight when the radar signal is affected by metal reflection clutter), ensuring that the fused feature vector is closer to the real environment and reducing the probability of misjudgment.

[0075] The lightweight computing formula adapted to the embedded platform adopts a linear weighted form with low computational complexity (only involving multiplication, addition and normalization). It can complete weight updates within microseconds, adapting to the low computing power limit of the drone's embedded AI computing unit and ensuring the response speed of real-time obstacle avoidance (such as hundreds of weight adjustments per second during high-speed flight).

[0076] The environment adaptation module includes:

[0077] Light sensor, used to trigger the infrared camera to work preferentially in low light conditions;

[0078] A haze detection unit, used to increase the weight of millimeter-wave radar based on image contrast;

[0079] The light sensor monitors ambient illumination in real time, automatically triggering the infrared camera to prioritize operation in low-light scenarios (such as at night and in tunnels). Based on thermal imaging, the infrared camera is unaffected by visible light intensity and can effectively detect heat-generating objects (such as humans and engines) or obstacles with significant temperature differences. This fills the detection blind spots of the RGB camera in dark environments, ensuring continuous day and night drone operations and preventing obstacle avoidance failures caused by sudden changes in light (such as the visual blind spot when entering a shadowy area from a brightly lit area).

[0080] The haze detection unit assesses visibility in real time through image contrast analysis. In low-visibility scenarios like haze, rain, and snow, it proactively increases the fusion weight of the millimeter-wave radar. Radar, with its strong penetration and minimal susceptibility to atmospheric interference, provides more reliable point cloud data than visual sensing, which suffers from image blur and reduced contrast, leading to a sharp drop in semantic segmentation accuracy.

[0081] Through dynamic weight adjustment, the system prioritizes the distance and speed information of the radar to locate obstacles, and combines it with the residual effective visual features (such as infrared heat signals) to form a fusion strategy of radar-dominated and vision-assisted, significantly reducing the missed detection rate in severe weather.

[0082] Hardware-level signal triggering (not software logic) is used to switch sensor modes, ensuring microsecond response delays to environmental changes such as low illumination and high haze, meeting the real-time requirements of high-speed drone flight. For example, when the light sensor detects a sudden drop in illumination, the hardware immediately triggers the infrared camera to operate at full power, while simultaneously reducing the sampling frequency of the RGB camera. This avoids the risk of delays caused by software algorithm judgment and improves the system's robustness in dynamic environmental changes.

[0083] The environmental adaptation module switches sensor states through threshold comparisons and simple logical operations, offering extremely low computational overhead and adapting to the drone's lightweight hardware architecture. For example, haze detection requires only image contrast mean analysis, eliminating the need for complex deep learning models. This reduces computing power while ensuring real-time weight adjustments, freeing up resources for the embedded AI computing unit, allowing it to focus on core tasks such as feature fusion and path planning.

[0084] The autonomous obstacle avoidance method for UAV based on the fusion of millimeter-wave radar and multimodal vision includes the following steps:

[0085] Synchronously collecting environmental data through the millimeter-wave radar module and the multispectral vision module;

[0086] De-noise and cluster the radar point cloud to generate candidate obstacle areas;

[0087] Perform semantic segmentation using the visual module in the candidate area to identify the type of obstacle;

[0088] The feature-level fusion module fuses radar and visual data to predict the trajectory of dynamic obstacles;

[0089] Generate a three-dimensional obstacle avoidance path based on the improved path planning algorithm of the decision module;

[0090] The sensor weights are dynamically adjusted based on the real-time visibility feedback from the environment adaptation module.

[0091] The path planning algorithm introduces the risk cost function as follows:

[0092] C = λ1·D+λ2·V, where D is the obstacle distance, v is the relative speed, and λ1 and λ2 are weight coefficients;

[0093] Traditional obstacle avoidance algorithms often plan paths based solely on obstacle distance, often overlooking the impact of dynamic objects' speed. Approaching objects at high speed, even at greater distances, can pose a high risk due to the drone's limited reaction time. However, approaching slow-moving or stationary objects, even at closer range, can be safely navigated through with more flexible path adjustments. The new algorithm incorporates both distance and relative speed as key factors, dynamically quantifying the threat level of obstacles. Approaching high-speed targets are prioritized as high-risk, leading to the system's pre-planned emergency obstacle avoidance paths. For slow-moving or stationary targets, the drone can flexibly choose a detour strategy tailored to the environment, avoiding under-reaction or over-avoidance caused by a single factor.

[0094] The algorithm adjusts the risk assessment focus based on the drone model, flight speed, and mission requirements: in high-speed flight scenarios (such as long-distance logistics and delivery), it focuses on the relative speed of objects, predicting potential collision risks in advance and reserving sufficient response space; in low-altitude, complex environments (such as inspections between urban buildings), it places greater emphasis on detecting close-range obstacles to avoid collisions caused by flying close to obstacles. This gradient risk response mechanism enables drones to quickly avoid danger in dangerous scenarios while maintaining efficient flight in safe ones, reducing unnecessary track deviations and improving mission execution efficiency.

[0095] The system directly utilizes the precise distance and speed of obstacles measured by millimeter-wave radar, combined with the visual module's ability to identify obstacle types (such as distinguishing between hard and soft objects, and dynamic and static targets), to further refine the risk level. For example, when a high-speed metal structure obstacle is identified, the system automatically increases its threat priority and plans a safer detour. For slow-moving vegetation-like targets, the system allows the vehicle to maintain its original course within a safe distance. This dual assessment ensures that obstacle avoidance decisions are based not only on the obstacle's position and motion state, but also on its potential risk attributes, significantly reducing misjudgments and missed detections.

[0096] The algorithm utilizes efficient computational logic to rapidly assess the threat level of single or multiple obstacles without complex mathematical calculations. This minimizes computational time and enables real-time processing even in densely populated obstacle scenarios. This feature perfectly matches the computing power limitations of drone embedded hardware, ensuring the system can dynamically adjust its path with millisecond-level response speeds even at high speeds, preventing obstacle avoidance failures caused by algorithmic delays. This provides key technical support for stable drone flight in dynamic and complex environments.

[0097] The dynamic obstacle trajectory prediction includes:

[0098] Use Kalman filtering for short-term state estimation;

[0099] Modeling long-term motion dependencies via recurrent neural networks;

[0100] By leveraging efficient recursive computing features, the high-frequency distance and speed data of millimeter-wave radar can be processed in real time to perform high-precision short-term state estimation (with error controlled at the centimeter level) of the current position and speed of dynamic obstacles (such as moving vehicles and flying birds). This ensures accurate capture of immediate motion trends and meets the real-time requirements of high-speed obstacle avoidance (such as millisecond-level response).

[0101] Recurrent neural networks model long-term dependencies: By learning the time series characteristics of historical trajectory data (such as changes in movement direction and acceleration patterns), they capture the long-term motion patterns of dynamic targets (such as vehicle turns and coordinated movement of drone formations) and predict trajectory changes within the next few seconds. For example, for vehicles about to slow down at intersections, their path turning points can be predicted in advance, avoiding the lag caused by obstacle avoidance paths relying solely on current speed.

[0102] The complementarity of long-term and short-term predictions improves the robustness of predictions: short-term estimation responds to sudden movements (such as sudden braking and sharp turns), and long-term modeling adapts to regular movements (such as circular trajectories and uniform linear motion). The combination of the two enables the system to deal with immediate dangers and plan forward-looking paths to reduce collision risks in dynamic scenarios (such as avoiding mistakenly entering the target's future trajectory due to only looking at the current position).

[0103] To address the limitations of traditional single-model prediction (e.g., Kalman filtering is only applicable to linear motion and RNN is sensitive to high-frequency noise), this method effectively handles complex motion in nonlinear and unstructured environments through heterogeneous model fusion:

[0104] For linear moving targets (such as drones flying at a constant speed), Kalman filtering is used to guide prediction and ensure computational efficiency.

[0105] For nonlinear moving targets, RNN improves prediction accuracy by learning the nonlinear laws of historical trajectories, avoiding missed detections due to model assumption bias (such as traditional methods misjudging changing directions as linear motion and thus missing targets).

[0106] The method also includes a multi-drone collaborative obstacle avoidance step, in which an obstacle map generated by the system is shared through a communication module, and a distributed algorithm is used to coordinate the paths of multiple drones.

[0107] Through communication modules (such as wireless ad hoc networks), each drone synchronizes its detected obstacle location, type, and motion status in real time, creating a global environmental map covering a wider area. For example, if one drone in a formation detects a mountain obstacle ahead, it can immediately share this information with all members, preventing subsequent drones from duplicating detections or missing detections due to blind spots.

[0108] Distributed Algorithm Collaborative Path Planning: Unlike centralized control (which relies on a central server), this model utilizes distributed algorithms (such as consensus protocols and local optimization strategies). Each drone independently plans its path based on a shared map, balancing its own mission with group collaboration (such as maintaining formation and avoiding collisions). This model does not require high-bandwidth communication support and is suitable for the low-latency collaboration needs of drone swarms in remote areas (such as disaster zones and the wild).

[0109] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0110] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0111] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. The autonomous obstacle avoidance system for UAVs based on the fusion of millimeter-wave radar and multimodal vision is characterized by: include: Millimeter-wave radar module, used to obtain distance, speed, and point cloud data of obstacles; Multispectral vision module, including RGB camera and infrared camera, used to extract the texture, category and thermal characteristics of obstacles; Embedded AI computing unit for real-time data processing and fusion; The spatiotemporal synchronization module aligns the timestamps of radar and visual data through hardware trigger signals and unifies the spatial coordinates based on joint calibration and SLAM technology; Feature-level fusion module, which uses a lightweight network to fuse radar point cloud features with visual semantic features; The decision module generates an obstacle avoidance path based on an improved path planning algorithm and adjusts the sensor confidence through dynamic weight allocation; Environmental adaptation module switches the sensor working mode according to the real-time environmental visibility.

2. The UAV autonomous obstacle avoidance system based on millimeter-wave radar and multimodal vision fusion according to claim 1 is characterized by: The network structure of the feature-level fusion module includes: Radar branch, used to extract global features of point clouds; The visual branch is used to extract local features of RGB and infrared images; Fusion layer, used to achieve cross-modal feature interaction through attention mechanism.

3. The UAV autonomous obstacle avoidance system based on millimeter-wave radar and multimodal vision fusion according to claim 1 is characterized by: The dynamic weight allocation formula is: W fusion =α·S radar +β·S vision +γ·E env Among them, W fusion is the radar signal-to-noise ratio, S radar Score visual clarity, E env is the environmental visibility factor; α, β, γ are dynamic optimization coefficients.

4. The UAV autonomous obstacle avoidance system based on millimeter-wave radar and multimodal vision fusion according to claim 1 is characterized by: The environment adaptation module includes: Light sensor, used to trigger the infrared camera to work preferentially in low light conditions; The haze detection unit is used to increase the weight of the millimeter-wave radar based on image contrast.

5. An autonomous obstacle avoidance method for unmanned aerial vehicles based on the fusion of millimeter-wave radar and multimodal vision, the method being applied to the obstacle avoidance system according to any one of claims 1 to 4, characterized in that: The method comprises the following steps: Synchronously collecting environmental data through the millimeter-wave radar module and the multispectral vision module; De-noise and cluster the radar point cloud to generate candidate obstacle areas; Perform semantic segmentation using the visual module in the candidate area to identify the type of obstacle; The feature-level fusion module fuses radar and visual data to predict the trajectory of dynamic obstacles; Generate a three-dimensional obstacle avoidance path based on the improved path planning algorithm of the decision module; The sensor weights are dynamically adjusted based on the real-time visibility feedback from the environment adaptation module.

6. The autonomous obstacle avoidance method for UAVs based on millimeter-wave radar and multimodal vision fusion according to claim 5 is characterized by: The path planning algorithm introduces the risk cost function as follows: C = λ1·D+λ2·V, where D is the obstacle distance, v is the relative speed, and λ1 and λ2 are weight coefficients.

7. The autonomous obstacle avoidance method for UAV based on millimeter wave radar and multimodal vision fusion according to claim 1 is characterized by: The dynamic obstacle trajectory prediction includes: Use Kalman filtering for short-term state estimation; Modeling long-term motion dependencies via recurrent neural networks.

8. The autonomous obstacle avoidance method for UAV based on millimeter-wave radar and multimodal vision fusion according to claim 1 is characterized by: The method also includes a multi-drone collaborative obstacle avoidance step, in which an obstacle map generated by the system is shared through a communication module, and a distributed algorithm is used to coordinate the paths of multiple drones.

Images