Flight obstacle avoidance system of low-altitude inspection unmanned aerial vehicle

Abstract

The invention relates to the technical field of unmanned aerial vehicle obstacle avoidance, and discloses a low-altitude inspection unmanned aerial vehicle flight obstacle avoidance system, which comprises a multi-mode sensing module used for collecting environment data and outputting an environment tensor; the hybrid driving prediction module is in communication connection with the multi-mode sensing module, receives the environment tensor, predicts a dynamic track of an obstacle based on a nonlinear kinetic equation and a data driving model, and outputs an obstacle prediction state; and the dynamic collaborative optimization module is in communication connection with the hybrid drive prediction module, receives the obstacle prediction state, optimizes the sensor weight through tensor decomposition and mixed integer programming, and outputs a reconstruction environment model. A hybrid drive prediction model is combined with a physical kinetic equation and LSTM data drive correction, the limitation of a traditional single model is broken through, the prediction error is smaller than or equal to 0.3 m in a random turning obstacle test, the target motion trend is accurately captured, and a reliable basis is provided for planning an obstacle avoidance path in advance.

Description

Technical Field

[0001] The present invention relates to the technical field of UAV obstacle avoidance, and particularly to a flight obstacle avoidance system for low-altitude inspection UAVs. Background Art

[0002] Low-altitude inspection UAVs have wide application requirements in fields such as power line inspection, urban logistics corridors, and forest monitoring, but they face safety obstacle avoidance challenges in complex dynamic environments. The traditional UAV obstacle avoidance system has the following core problems:

[0003] Insufficient dynamic obstacle prediction ability: Relying on a single sensor or a simple physical model, it is difficult to accurately capture the behaviors such as direction change, acceleration, and deceleration of dynamic targets such as pedestrians, vehicles, and birds, resulting in large prediction errors (the prediction error of traditional methods ≥ 1.2 m), and it cannot meet the real-time obstacle avoidance requirements.

[0004] Poor stability of multi-sensor fusion: In harsh environments such as rain, fog, and dust, a single sensor is easily interfered by noise, and the misjudgment rate of traditional weighted fusion algorithms is high (≥ 15%), and there is a lack of a dynamic weight optimization mechanism, making it difficult to adapt to the change of sensor data reliability in complex scenarios.

[0005] Low efficiency of multi-UAV collaboration: Traditional path planning algorithms do not effectively consider the collaboration and conflict avoidance among multiple UAVs, resulting in a high risk of path overlap or collision during the inspection process (for example, the number of conflicts in the traditional scheme of 10 UAVs ≥ 8 times / hour), and the task efficiency is low.

[0006] The contradiction between computing power and power consumption is prominent: Although the pure GPU computing scheme can handle complex algorithms, it has high power consumption and serious heat generation, making it difficult to meet the endurance requirements of UAVs; while lightweight models often result in insufficient accuracy and cannot balance real-time performance and computing efficiency.

[0007] In view of the above problems, the existing technology lacks a systematic solution, and there is an urgent need for an efficient obstacle avoidance system that integrates multi-modal perception, dynamic prediction, collaborative planning, and heterogeneous computing. Summary of the Invention

[0008] In view of the deficiencies of the existing technology, the present invention provides a flight obstacle avoidance system for low-altitude inspection UAVs to solve the deficiencies of the existing technology.

[0009] To achieve the above objectives, the present invention is realized through the following technical solutions: A flight obstacle avoidance system for low-altitude inspection UAVs, comprising:

[0010] A multi-modal perception module, used for collecting environmental data and outputting an environmental tensor;

[0011] A hybrid-driven prediction module, which is communicatively connected to the multi-modal perception module, receives the environmental tensor, predicts the dynamic trajectory of obstacles based on non-linear dynamics equations and data-driven models, and outputs the predicted state of obstacles;

[0012] A dynamic collaborative optimization module, which is communicatively connected to the hybrid-driven prediction module, receives the predicted state of obstacles, optimizes the sensor weights through tensor decomposition and mixed integer programming, and outputs a reconstructed environmental model;

[0013] A hierarchical game planning module, which is communicatively connected to the dynamic collaborative optimization module, receives the reconstructed environmental model, generates a global obstacle avoidance path based on distributed pheromone update and potential game, and outputs a path instruction;

[0014] A heterogeneous computing execution module, which is communicatively connected to the hierarchical game planning module, receives the path instruction, dynamically allocates hardware resources, generates a real-time control signal and transmits it to the UAV flight control system;

[0015] Among them, each module communicates in a publish-subscribe mode through a high-speed data bus to form a closed-loop control process. The data bus supports parallel transmission of environmental tensors, predicted obstacle states, reconstructed environmental models, and path instructions.

[0016] Preferably, the multi-modal perception module includes:

[0017] A visual sensor, a lidar, and a millimeter-wave radar, which are respectively used to collect images, point clouds, and velocity data;

[0018] A hardware trigger synchronization unit to achieve time synchronization between vision and lidar, and the millimeter-wave radar aligns timestamps through an interpolation algorithm;

[0019] An environmental tensor construction unit that encodes multi-source data into a 4D tensor T ∈ R H×W×D×C ;

[0020] An outlier filtering unit that uses a statistical outlier removal algorithm to filter noise points with a radius of 0.2 m and a minimum of 3 neighboring points;

[0021] A dynamic target detection unit that performs real-time target detection based on an optimized YOLOv5s model.

[0022] Preferably, the prediction equation of the hybrid-driven prediction module is:

[0023]

[0024] Among them, A and B are physical model matrices including air resistance. The LSTM network inputs the historical 5-frame 6D state and outputs a state correction amount Δx _t , and combines the EM algorithm and the Viterbi algorithm to predict the trajectory in the next 3 seconds.

[0025] Preferably, the dynamic collaborative optimization module includes:

[0026] An adaptive Tucker decomposition unit that dynamically adjusts the decomposition rank according to the obstacle density

[0027] A sensor weight optimization unit that solves for the optimal weight w through mixed-integer programming * , and the objective function is:

[0028]

[0029] Preferably, the hierarchical game planning module includes:

[0030] A local non-cooperative game unit whose utility function combines path probability and obstacle distance;

[0031] A global potential game unit that solves for the Nash equilibrium through a potential function and supports virtual game iteration;

[0032] A failure response unit that switches to the improved A * algorithm and the leader-follower mode when triggered.

[0033] Preferably, the heterogeneous computing execution module includes:

[0034] FPGA and NPU heterogeneous hardware that respectively process numerical calculations and neural network inferences;

[0035] A dynamic resource allocation strategy that adjusts the computing power allocation based on a delay threshold of 50 ms and migrates the LSTM task to the FPGA when the threshold is exceeded;

[0036] A PID controller whose gain parameters are tuned by the Ziegler-Nichols method and whose control frequency is 100 Hz.

[0037] Preferably, the failure response strategy of the multi-modal perception module includes:

[0038] Triggering condition: Visual / mmWave data not received for 3 consecutive frames;

[0039] Degraded mode: Single-channel lidar tensor, lightweight LSTM model, safety margin extended by 50%, detection range maintained at 80%, and computing power requirement reduced to 45%.

[0040] Preferably, the time synchronization mechanism of the multi-modal perception module includes:

[0041] The visual sensor and the lidar achieve time synchronization through a hardware trigger signal, and the synchronization error is less than 1 ms;

[0042] The millimeter wave radar adopts asynchronous acquisition mode and aligns the timestamps through the interpolation algorithm. The specific formula is:

[0043] t′ radar =t camera +N(t radar -t camera )·Δt

[0044] Where N is the number of interpolation points and Δt is the sampling interval.

[0045] Preferably, the obstacle behavior modeling method of the hybrid drive prediction module includes:

[0046] Define obstacle behavior state set S = {s1: uniform speed, s2: acceleration, s3: change of direction};

[0047] Based on the EM algorithm, the state transition probability is trained offline, and the mean μ of each state is obtained by clustering historical data. j and variance σ j ;

[0048] During real-time prediction, the Viterbi algorithm is used to solve the most likely state sequence and output the obstacle trajectory within the next 3 seconds.

[0049] Preferably, the sensor weight optimization process of the dynamic collaborative optimization module includes:

[0050] Receive the raw data confidence score from the multimodal perception module and the obstacle prediction state error from the hybrid drive prediction module;

[0051] The optimal weight w is solved by mixed integer programming * Feedback is sent to the multimodal perception module to dynamically adjust the sensor sampling frequency.

[0052] The present invention provides a low-altitude inspection UAV flight obstacle avoidance system, which significantly improves the obstacle avoidance performance of the UAV in complex dynamic environments through multidisciplinary technology integration and hardware-algorithm collaborative design, and has the following beneficial effects:

[0053] 1. The present invention combines the physical dynamics equation with the LSTM data-driven correction through a hybrid drive prediction model, breaking through the limitations of the traditional single model. In the random change of direction obstacle test, the prediction error is ≤0.3m (the traditional method is ≥1.2m), accurately capturing the target movement trend, and providing a reliable basis for planning the obstacle avoidance path in advance. At the same time, the behavior state modeling uses the EM algorithm and the Viterbi algorithm to effectively identify the uniform speed, acceleration, change of direction and other behavior patterns of obstacles, improve the prediction ability of complex motion trajectories, and is suitable for real-time tracking of low-altitude multi-type dynamic targets (such as birds and drones).

[0054] 2. In the present invention, multi-source data synchronization and tensor construction adopt hardware-triggered synchronization (error <1ms) and interpolation alignment algorithm to ensure the spatio-temporal consistency of visual, lidar, and millimeter-wave radar data; 4D environmental tensors are used to encode multi-modal information, combined with SOR outlier filtering (noise filtering rate ≥95%, effective point retention rate ≥98%) and YOLOv5s optimized detection (miss detection rate ≤5%) to construct a high-precision environmental model. At the same time, the dynamic cooperative optimization mechanism is based on Tucker tensor decomposition and mixed integer programming, adaptively adjusts the sensor weights according to the obstacle density, and the misjudgment rate ≤2% (traditional weighted fusion ≥15%) in bad weather such as rain and fog, significantly improving the perception reliability in complex environments.

[0055] 3. In the present invention, the hierarchical game planning algorithm combines local non-cooperative games and global potential games to achieve distributed optimization of the paths of multiple UAVs. The number of path conflicts for 10 UAVs during cooperative inspection is 0 (traditional solution ≥8 times / hour), greatly improving the task execution efficiency and safety. At the same time, when the planning times out or the equilibrium solution fails, the failure response strategy automatically switches to the improved A* algorithm and the "leader-follower" mode to ensure the system robustness and adapt to dynamic networking and emergency scenarios.

[0056] 4. In the present invention, the heterogeneous computing resource allocation utilizes the hardware heterogeneous characteristics of FPGA and NPU to dynamically adjust the computing power allocation (such as migrating the LSTM task to FPGA when the delay >50ms). The end-to-end delay ≤30ms, and the power consumption is reduced by 40% compared with the pure GPU solution, taking into account both the computing efficiency and the UAV endurance. The PID control parameter optimization tunes the controller gain through the Ziegler-Nichols method to achieve 100Hz high-frequency control, ensuring the precise execution of the UAV's path instructions and improving the flight stability.

[0057] 5. Each module in the present invention adopts the publish-subscribe communication mode, supports flexible replacement and extension of sensor configurations and algorithm models, is applicable to multiple scenarios such as power inspection, urban logistics, and forest monitoring, and has strong implementability and engineering application value.

[0058] In summary, the present invention systematically solves the core technical problems of obstacle avoidance for low-altitude inspection UAVs, achieves significant breakthroughs in prediction accuracy, fusion robustness, cooperative efficiency, and energy efficiency ratio, and has important technical innovation and practical application significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the system flow of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0061] Please refer to the attached Figure 1 , the embodiment of the present invention provides a low-altitude inspection UAV flight obstacle avoidance system, which is based on multi-modal sensor fusion, dynamic obstacle trajectory prediction and multi-aircraft collaborative planning, and is applicable to UAV safety obstacle avoidance in complex dynamic environments in scenarios such as power line inspection, urban logistics corridors, and forest monitoring.

[0062] The low-altitude inspection UAV flight obstacle avoidance system includes a multi-modal perception module, a hybrid drive prediction module, a dynamic collaborative optimization module, a hierarchical game planning module and a heterogeneous computing execution module. The above modules are communicatively connected through a high-speed data bus and interconnected through a real-time communication protocol to form a closed-loop control process. Among them, the data bus supports parallel transmission of environmental tensors, obstacle prediction states, reconstructed environmental models and path instructions.

[0063] The communication architecture between each module adopts a publish-subscribe mode, where:

[0064] The multi-modal perception module is the publisher of the environmental tensor;

[0065] The hybrid drive prediction module, the dynamic collaborative optimization module, and the hierarchical game planning module are successively subscribers and publishers;

[0066] The heterogeneous computing execution module is the final subscriber of the path instruction.

[0067] The following is a detailed description of each module in the system of the present invention, and a comprehensive elaboration is carried out on the specific implementation principles, technical details and processes of each module.

[0068] Multi-modal perception module

[0069] The multi-modal perception module of the present invention collects environmental data through a vision sensor, a lidar and a millimeter-wave radar, and outputs an environmental tensor.

[0070] During the process of collecting environmental data, the data synchronization mechanism between each sensor is as follows:

[0071] The time synchronization of the vision sensor and the lidar is achieved through a hardware trigger signal (PPS pulse), and the error is <1ms;

[0072] The millimeter-wave radar adopts asynchronous acquisition, and the timestamps are aligned through an interpolation algorithm:

[0073] t′ radar = t camera + N(t radar - t camera )·Δt

[0074] where N is the number of interpolation points and Δt is the sampling interval.

[0075] In the multi-modal perception module, the recommended configurations for each sensor are as follows:

[0076] The visual sensor uses an Intel RealSense D455 RGB-D camera with a resolution of 1280×720, a frame rate of 30fps, and a depth measurement accuracy of ±2cm@2m;

[0077] The lidar uses a Hesai Technology PandarXT-16 line lidar with a detection range of 150m, a horizontal angular resolution of 0.1°, and a vertical field of view of 30° (-15° to +15°);

[0078] The millimeter-wave radar uses a TI AWR1843 77GHz frequency-modulated continuous-wave radar with a maximum detection range of 200m and a speed resolution of 0.1m / s.

[0079] The process for constructing the environmental tensor is as follows:

[0080] Encode the multi-source data into a 4D tensor T ∈ R H×W×D×C :

[0081] H = 1080, W = 1920: Visual image resolution;

[0082] D = 16: Number of LiDAR (lidar) vertical beams;

[0083] C = 3: Modal channels (visual sensor = 1, LiDAR = 2, millimeter-wave radar = 3)

[0084] Data alignment:

[0085] Map the LiDAR point cloud and the RGB-D (visual sensor) image to the same coordinate system through calibration parameters:

[0086]

[0087] where is a 3×3 rotation matrix, is a 3×1 translation vector, pre-calibrated by the checkerboard calibration method, and P cam is the point cloud coordinate in the camera coordinate system.

[0088] Tensor filling rule:

[0089] Channel 1 (C = 1): Depth information of RGB-D image (normalized to 0 - 255);

[0090] Channel 2 (C = 2): Reflection intensity of LiDAR point cloud (normalized to 0 - 1);

[0091] Channel 3 (C = 3): Velocity information of millimeter-wave radar (linearly mapped from -5 m / s to +5 m / s to 0 - 1).

[0092] Specifically, it is also necessary to filter out LiDAR outliers from the point cloud data collected by the lidar:

[0093] Due to environmental noise (such as rain, fog, dust) or sensor anomalies, there are invalid point clouds in the LiDAR point cloud. The Statistical Outlier Removal (SOR) algorithm is used to filter out the above invalid point clouds. This algorithm is based on the statistical characteristics of local point cloud density and removes outliers that do not conform to the spatial distribution law. Specifically as follows:

[0094] For any point p ∈ R 3 in the point cloud, calculate the spatial distribution of points in its neighborhood:

[0095] Neighborhood point set N(p) = {p1, p2, …, p k}, where k is the total number of points in the neighborhood,

[0096] Outlier determination condition: The retained point p is

[0097]

[0098] where the neighborhood is defined as a spherical space centered at p with a radius r = 0.2 m; is the indicator function, taking 1 when the condition is met and 0 otherwise.

[0099] And for the selection basis of the above parameters:

[0100] Radius r = 0.2 m: Determined through experimental verification, which can effectively filter out isolated noise points while retaining the effective point clouds of fine structures such as power lines and tree branches;

[0101] Minimum neighborhood point number threshold 3: Ensure that there are at least 3 neighboring points to form a local plane or surface to avoid misdeleting real obstacle points.

[0102] Specific implementation steps of the algorithm:

[0103] Construct a KD-Tree: Establish a spatial index for the input point cloud to accelerate neighborhood search;

[0104] Traverse all points: For each point p, search for the neighborhood point set N(p) within its radius r;

[0105] Statistics and filtering: Calculate the number of neighborhood points. If it is less than the threshold, mark it as an outlier.

[0106] Output the filtered point cloud: Remove all marked points and retain the valid point cloud.

[0107] According to the above scheme, more than 95% of the isolated noise points (such as rain, fog, and dust interference) are filtered out through radius filtering (SOR algorithm), while the valid point cloud of fine structures (power lines, tree branches, etc.) is retained, and the retention rate is ≥98%, ensuring the integrity of environmental modeling.

[0108] In addition, for visual dynamic target detection, the YOLOv5s (You Only Look Once version5 small) model is used for real-time target detection. Its lightweight design is suitable for the drone embedded platform. The model is optimized for the low-altitude inspection scenario as follows:

[0109] Input resolution: Adjusted to 640×640 pixels to balance detection accuracy and computational efficiency;

[0110] Training dataset: Jointly trained using the COCO dataset and a custom low-altitude obstacle dataset (including 10 types of targets such as pedestrians, vehicles, and birds). The data augmentation strategies include Mosaic and MixUp; Joint training with the COCO and custom datasets improves the detection ability for low-altitude small targets (birds, drones), and the missed detection rate is ≤5% (test set).

[0111] Model quantization: Using INT8 quantization to compress the model size to 14.4MB, and the inference speed is increased to 120FPS (NVIDIA Jetson Xavier NX platform).

[0112] The specific model structure is as follows:

[0113] Backbone: CSPDarknet53, extracting multi-scale features;

[0114] Neck: PANet (Path Aggregation Network), enhancing feature fusion;

[0115] Head: Output detection results at 3 scales (80×80, 40×40, 20×20);

[0116] The relevant loss function is:

[0117]

[0118] Among them, is the classification loss (Focal Loss); is the target existence loss (BCEWithLogitsLoss); is the bounding box regression loss (CIoU Loss);

[0119] Weight coefficient: λ cls = 0.5, λ obj = 1.0, λ box = 0.05.

[0120] Post - processing of detection results:

[0121] Non - maximum suppression (NMS): Overlap threshold IoU = 0.5, keep the detection box with the highest confidence;

[0122] Coordinate transformation: Transform the bounding box (x min , y min , x max , y max ) in the image coordinate system to the UAV body coordinate system:

[0123]

[0124] where K is the camera intrinsic matrix and d is the depth value (from the RGB - D camera).

[0125] Multi - source data synchronization and tensor construction use hardware - triggered synchronization (error < 1ms) and interpolation alignment algorithms to ensure the spatio - temporal consistency of visual, lidar, and millimeter - wave radar data; use 4D environmental tensors to encode multi - modal information, combine SOR outlier filtering (noise filtering rate ≥ 95%, effective point retention rate ≥ 98%) and YOLOv5s optimized detection (miss detection rate ≤ 5%) to build a high - precision environmental model. At the same time, the dynamic collaborative optimization mechanism is based on Tucker tensor decomposition and mixed - integer programming, adaptively adjusts the sensor weights according to the obstacle density, and the misjudgment rate is ≤ 2% in bad weather such as rain and fog (traditional weighted fusion ≥ 15%), significantly improving the perception reliability in complex environments.

[0126] In addition, when the above - mentioned multi - modal perception module fails, the following countermeasures are taken:

[0127] Trigger condition: No visual / millimeter - wave radar data is received for 3 consecutive frames, and the LiDAR heartbeat signal is normal,

[0128] Implementation of the degradation mode:

[0129] Data input reconstruction: Simplify the environmental tensor channel C = 3 to a single - channel mode:

[0130] T LiDAR ∈R H×W×D×1 (Channel 1: Intensity value normalization)

[0131] Prediction model switching: Enable lightweight LSTM model (parameter quantity reduced to 30% of the original)

[0132] Safety margin expansion: Obstacle inflation radius increased by 50% (original radius r → 1.5r)

[0133] Performance metrics:

[0134] Detection range: Maintain 80% of the original system's ability (200m → 160m)

[0135] Computing power requirement: Reduced to 45% of the original.

[0136] Multi-modal degradation verification data

[0137]

[0138] Hybrid drive prediction module

[0139] In the present invention, the hybrid drive prediction module is communicatively connected to the multi-modal perception module, receives the environmental tensor, predicts the dynamic trajectory of the obstacle based on the non-linear dynamics equation and the data-driven model, and outputs the predicted state of the obstacle.

[0140] Among them, the obstacle state vector x t =[x, y, z, v x , v y , v z T's prediction equation is:

[0141]

[0142] Ax t + Bu t is the physical model, and LSTM(x t-5:t-1 , h t-1 ) is the data-driven correction.

[0143] Physical model parameters:

[0144]

[0145] Among them, c = 0.25 is the air resistance coefficient, and m is the mass of the obstacle (default 1kg).

[0146] LSTM network structure:

[0147] Input layer: 6-dimensional state sequence (last 5 frames);

[0148] Hidden layer: 2 layers, with 64 neurons in each layer;

[0149] Output layer: 6-dimensional state correction amount Δx t .

[0150] Define the obstacle behavior state set S = {s1: uniform speed, s2: acceleration, s3: change of direction}, and use the EM algorithm to train the parameters offline:

[0151] State transition probability:

[0152]

[0153] Among them, μ j and σ j Obtained through clustering of historical data.

[0154] Real-time prediction: Based on the Viterbi algorithm, the most likely state sequence is solved and the obstacle trajectory within the next 3 seconds is output.

[0155] By combining the hybrid drive prediction model with the physical dynamics equation and LSTM data-driven correction, the limitations of the traditional single model are broken through. In the random change of direction obstacle test, the prediction error is ≤0.3m (the traditional method is ≥1.2m), accurately capturing the target movement trend, and providing a reliable basis for planning the obstacle avoidance path in advance. At the same time, the behavior state modeling uses the EM algorithm and the Viterbi algorithm to effectively identify the uniform speed, acceleration, change of direction and other behavior patterns of obstacles, improve the prediction ability of complex motion trajectories, and is suitable for real-time tracking of low-altitude multi-type dynamic targets (such as birds and drones).

[0156] Dynamic collaborative optimization module

[0157] In the present invention, the dynamic collaborative optimization module is communicatively connected with the hybrid drive prediction module, receives the obstacle prediction state, optimizes the sensor weights through tensor decomposition and mixed integer programming, and outputs a reconstructed environment model.

[0158] Tensor decomposition adopts an adaptive method, that is, the tensor decomposition rank is dynamically adjusted according to the obstacle density:

[0159]

[0160] The decomposition algorithm uses Tucker decomposition, and the objective function is:

[0161]

[0162] Among them, G is the core tensor, U (i) is a factor matrix.

[0163] The sensor weight optimization process includes:

[0164] Receive raw data confidence scores from the multimodal perception module;

[0165] receiving an obstacle prediction state error from a hybrid drive prediction module;

[0166] Dynamically adjust the weights according to the scoring and error, and feedback the optimization result to the multi-modal perception module to adjust the sensor sampling frequency.

[0167] Mixed-integer programming optimization

[0168] Optimize the sensor weights w = [w1, w2, w3] T :

[0169]

[0170] Solution method: Use the Gurobi solver for branch and bound to calculate the optimal weights w * ;

[0171] Dynamic feedback: Feed w * back to the multi-modal perception module to adjust the sensor sampling frequency (e.g., reduce the visual frame rate from 30fps to 15fps to reduce the computational load).

[0172] Hierarchical game planning module

[0173] The hierarchical game planning module of the present invention is communicatively connected to the dynamic collaborative optimization module, receives the reconstructed environmental model, generates a global obstacle avoidance path based on distributed pheromone update and potential game, and outputs a path instruction.

[0174] Local non-cooperative game utility function:

[0175]

[0176] d k : The Euclidean distance between the path point k and the nearest obstacle;

[0177] Pheromone update:

[0178]

[0179] Global potential game coordination:

[0180] Define the potential function and solve for the Nash equilibrium:

[0181]

[0182] Among them, Overlap(p i , p j ) is the path overlap degree, and the calculation method is:

[0183]

[0184] Solution algorithm: Virtual game iteration, the maximum number of iterations is 100, and the convergence threshold ∈ = 0.01.

[0185] The hierarchical game planning algorithm combines local non - cooperative games with global potential games to achieve distributed optimization of the paths of multiple UAVs. The number of path conflicts during the collaborative inspection of 10 UAVs is 0 (≥8 times per hour in the traditional scheme), significantly improving the task execution efficiency and safety. At the same time, when the planning times out or the equilibrium solution fails, the failure response strategy automatically switches to the improved A * algorithm and the "leader - follower" mode to ensure the robustness of the system and adapt to dynamic networking and emergency scenarios.

[0186] In addition, when the hierarchical game planning module fails, the following response strategies are taken:

[0187] Trigger conditions: Path planning timeout (>100ms) or Nash equilibrium solution failure,

[0188] A * Algorithm improvement:

[0189] Dynamic cost function:

[0190] f(n) = g(n)+h(n)+λ·risk(n)

[0191] risk(n) = The probability of dynamic obstacles provided by the prediction module (0 - 1)

[0192] Adaptive weight λ = 0.3 - 1.0 (adjusted according to flight speed)

[0193] Collaborative strategy degradation:

[0194] Switch to the "leader - follower" mode, and only the leading aircraft executes the complete A * planning

[0195] Dynamic adjustment of following distance: d = 3v + 2 (v is the flight speed in m / s).

[0196] Performance comparison of planning degradation

[0197]

[0198] Heterogeneous computing execution module

[0199] In the present invention, the heterogeneous computing execution module is communicatively connected to the hierarchical game planning module, receives the path instructions, dynamically allocates FPGA and NPU hardware resources, generates real - time control signals and transmits them to the UAV flight control system.

[0200] The heterogeneous computing execution module is connected to the FPGA and NPU through a PCIe interface, where:

[0201] FPGA (Xilinx Zynq UltraScale+ MPSoC): Responsible for solving differential equations and performing tensor decomposition calculations of the hybrid drive prediction module;

[0202] NPU (NVIDIA Jetson Xavier NX): Responsible for performing neural network inference and game equilibrium solving of the hierarchical game planning module.

[0203] Dynamic resource allocation strategy:

[0204] Adjust the computing power allocation ratio according to the real-time delay:

[0205]

[0206] Threshold T threshold = 50ms, adjust the slope k = 0.5;

[0207] Task migration mechanism: When T delay > 50ms, migrate the LSTM computing task from the NPU to the FPGA.

[0208] PID controller parameters:

[0209]

[0210] The gain parameters are tuned by the Ziegler-Nichols method, and the control frequency is 100Hz.

[0211] Technical effect verification data

[0212] Dynamic obstacle prediction: In the random direction-changing obstacle test, the prediction error ≤ 0.3m (traditional method ≥ 1.2m);

[0213] Multi-sensor fusion: In rainy and foggy weather, the misjudgment rate ≤ 2% (traditional weighted fusion ≥ 15%);

[0214] Multi-aircraft cooperation efficiency: The number of path conflicts for 10 drones in cooperative inspection is 0 (traditional scheme ≥ 8 times / hour);

[0215] Real-time performance and power consumption: End-to-end delay ≤ 30ms, power consumption reduced by 40% (compared with the pure GPU scheme).

[0216] The heterogeneous computing resource allocation makes use of the hardware heterogeneous characteristics of FPGA and NPU to dynamically adjust the computing power allocation (e.g., migrating the LSTM task to FPGA when the latency > 50 ms). The end-to-end latency ≤ 30 ms, and the power consumption is reduced by 40% compared with the pure GPU solution, taking into account both the computing efficiency and the drone endurance. The PID control parameter optimization tunes the controller gain by the Ziegler-Nichols method to achieve 100Hz high-frequency control, ensuring the precise execution of the path commands by the drone and improving the flight stability.

[0217] In view of the core problems in the low-altitude inspection scenario, such as inaccurate prediction of dynamic obstacles, poor stability of multi-sensor fusion, low efficiency of multi-aircraft cooperation, and the contradiction between computing power and power consumption, the present invention constructs a complete drone obstacle avoidance system through the integration of multi-disciplinary theories and the collaborative design of hardware and algorithms. The technical solutions cover innovation points such as sensor synchronization, non-linear prediction models, dynamic optimization algorithms, hierarchical game planning, and heterogeneous computing architectures. The parameters and formulas of each module are clearly defined, with strong implementability, and have significant technological breakthroughs and application values.

[0218] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A flight obstacle avoidance system for low-altitude inspection unmanned aerial vehicles, characterized in that, Including: A multi-modal perception module, which is used to collect environmental data and output an environmental tensor; A hybrid-driven prediction module, which is communicatively connected to the multi-modal perception module, receives the environmental tensor, predicts the dynamic trajectory of an obstacle based on a non-linear dynamics equation and a data-driven model, and outputs the predicted state of the obstacle; A dynamic collaborative optimization module, which is communicatively connected to the hybrid-driven prediction module, receives the predicted state of the obstacle, optimizes the sensor weights through tensor decomposition and mixed integer programming, and outputs a reconstructed environmental model; A hierarchical game planning module, which is communicatively connected to the dynamic collaborative optimization module, receives the reconstructed environmental model, generates a global obstacle avoidance path based on distributed pheromone update and potential game, and outputs a path instruction; A heterogeneous computing execution module, which is communicatively connected to the hierarchical game planning module, receives the path instruction, dynamically allocates hardware resources, generates a real-time control signal and transmits it to the UAV flight control system; Among them, each module communicates in a publish-subscribe mode through a high-speed data bus to form a closed-loop control process, and the data bus supports parallel transmission of the environmental tensor, the predicted state of the obstacle, the reconstructed environmental model and the path instruction.

2. The low-altitude inspection UAV flight obstacle avoidance system according to claim 1, wherein The multi-modal perception module includes: A vision sensor, a lidar, and a millimeter-wave radar, which are respectively used to collect images, point clouds, and velocity data; A hardware trigger synchronization unit to achieve time synchronization between vision and lidar, and the millimeter-wave radar aligns timestamps through an interpolation algorithm; An environmental tensor construction unit that encodes multi-source data into a 4D tensor T ∈ R H×W×D×C ; An outlier filtering unit, which uses a statistical outlier removal algorithm to filter noise points with a radius of 0.2m and a minimum of 3 neighboring points; A dynamic target detection unit, which performs real-time target detection based on an optimized YOLOv5s model.

3. The flight obstacle avoidance system for a low-altitude inspection UAV according to claim 1, wherein, The prediction equation of the hybrid-driven prediction module is: Among them, A and B are physical model matrices including air resistance. The LSTM network inputs the historical 5-frame 6-dimensional state and outputs the state correction amount Δx _t , and combines the EM algorithm and the Viterbi algorithm to predict the trajectory in the next 3 seconds.

4. The flight obstacle avoidance system for a low-altitude inspection UAV according to claim 1, characterized in that, The dynamic collaborative optimization module includes: Adaptive Tucker decomposition unit, dynamically adjusting the decomposition rank according to the obstacle density Sensor weight optimization unit, which solves for the optimal weight w through mixed-integer programming * , and the objective function is:

5. The flight obstacle avoidance system for a low-altitude inspection UAV according to claim 1, characterized in that, The hierarchical game planning module includes: A local non-cooperative game unit, where the utility function combines path probability and obstacle distance; Global potential game unit, through the potential function Solve the Nash equilibrium and support the virtual game iteration; Failure response unit, switch to improvement A when triggered * Algorithm and leader-follower mode.

6. The low-altitude inspection UAV flight obstacle avoidance system according to claim 1, characterized in that, The heterogeneous computing execution module includes: FPGA and NPU heterogeneous hardware, which respectively process numerical calculations and neural network inferences; A dynamic resource allocation strategy, which adjusts the computing power allocation based on a delay threshold of 50ms, and migrates the LSTM task to the FPGA when it exceeds the limit; A PID controller, whose gain parameters are tuned by the Ziegler-Nichols method, and the control frequency is 100Hz.

7. The flight obstacle avoidance system for a low-altitude inspection UAV according to claim 1, characterized in that, The failure coping strategy of the multi-modal perception module includes: Trigger condition: No vision / millimeter-wave data is received for 3 consecutive frames; Degraded mode: Single-channel lidar tensor, lightweight LSTM model, safety margin extended by 50%, detection range maintained at 80%, and computing power requirement reduced to 45%.

8. The flight obstacle avoidance system for a low-altitude inspection UAV according to claim 2, wherein, The time synchronization mechanism of the multi-modal perception module includes: The vision sensor and the lidar achieve time synchronization through a hardware trigger signal, and the synchronization error is less than 1ms; The millimeter-wave radar adopts an asynchronous acquisition mode and aligns timestamps through an interpolation algorithm. The specific formula is t′ radar = t camera + N(t radar - t camera )·Δt Where N is the number of interpolation points and Δt is the sampling interval.

9. The flight obstacle avoidance system for a low-altitude inspection unmanned aerial vehicle according to claim 3, characterized in that, The obstacle behavior modeling method of the hybrid-driven prediction module includes: Define the obstacle behavior state set S = {s1: uniform speed, s2: acceleration, s3: direction change}; Offline training of state transition probabilities based on the EM algorithm, and obtaining the mean μ of each state through clustering of historical data j and variance σ j ; When making real-time predictions, the Viterbi algorithm is used to solve the most likely state sequence, and the obstacle trajectory within the next 3 seconds is output.

10. The flight obstacle avoidance system for a low-altitude inspection UAV according to claim 4, characterized in that, The sensor weight optimization process of the dynamic collaborative optimization module includes: Receiving the original data confidence score of the multimodal perception module and the obstacle prediction state error of the hybrid drive prediction module; Optimal weight w solved by mixed-integer programming * Feed back to the multi-modal perception module to dynamically adjust the sensor sampling frequency.

Images