Unmanned aerial vehicle multi-source perception fusion ai real-time intelligent guidance and adaptive obstacle avoidance method

Abstract

The application relates to an unmanned aerial vehicle multi-source perception fusion AI real-time intelligent guidance and self-adaptive obstacle avoidance method, flight data is collected through a vision sensor and an infrared sensor, timestamp and spatial coordinate information are acquired, and a sampling frequency difference is calculated, a time-space aligned fusion data set is generated by using an adaptive interpolation algorithm, image and heat source features in the fusion data set are extracted, a multi-dimensional feature set is generated, whether an obstacle exists is judged, and related information is extracted, environment data is analyzed, an environment complexity index is calculated, a risk level is generated in combination with the obstacle information, a sensor dynamic weight is predicted, and the working mode is adjusted, based on the obstacle situation, a comprehensive risk value is calculated through a path risk model, a motion trend is predicted, an optimal obstacle avoidance path and flight instruction are finally generated, efficient sensor data processing, accurate risk assessment and real-time obstacle avoidance planning are realized, and the unmanned aerial vehicle is stably and safely flown in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to an AI real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion. Background Technology

[0002] In recent years, drone technology has made rapid progress, and its application areas have continued to expand, covering many industries such as logistics and delivery, geographic surveying and mapping, agricultural and forestry plant protection, emergency rescue, and film and television shooting. However, obstacle avoidance and navigation remain key bottlenecks limiting the widespread application of drones when flying in complex environments. Traditional obstacle avoidance methods mainly rely on a single sensor, such as an ultrasonic sensor or a simple visual sensor. Ultrasonic sensors are limited by their short detection range and are easily affected by environmental noise, resulting in poor performance when detecting obstacles at long distances. Simple visual sensors suffer significant reductions in obstacle recognition and detection capabilities in low light, inclement weather (such as heavy rain and sandstorms), and complex texture environments, making it difficult to meet the needs of drones for safe flight in complex environments.

[0003] The rise of multi-source perception fusion technology has brought new opportunities to solve the above problems. By fusing data from multiple types of sensors, such as visual sensors and infrared sensors, it is theoretically possible to perceive the surrounding environment more comprehensively. However, current multi-source perception fusion methods still have many shortcomings in practical applications. In the sensor data processing stage, differences in sampling frequency and data format between different sensors make data fusion difficult, making it hard to achieve accurate time synchronization and spatial alignment, thus affecting the quality of fused data. In terms of risk assessment, existing methods often fail to fully consider environmental factors and the dynamic changes of obstacles, resulting in inaccurate and incomplete risk assessment results, which cannot provide a reliable basis for UAV decision-making. In obstacle avoidance path planning, the planning algorithms are usually computationally complex and have poor real-time performance, making it difficult to meet the timeliness requirements of obstacle avoidance decisions during rapid flight of UAVs. Summary of the Invention

[0004] The main objective of this invention is to provide an AI real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion, so as to achieve efficient processing of sensor data, accurate risk assessment and real-time obstacle avoidance path planning, and achieve the goal of stable and safe flight of UAVs in complex environments.

[0005] To achieve the above objectives, this invention provides an AI real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion, comprising the following steps:

[0006] Flight data is collected by the drone's visual and infrared sensors to obtain timestamps and spatial coordinate information, and the sampling frequency difference between the two types of sensors is calculated.

[0007] Based on the aforementioned sampling frequency difference, an adaptive interpolation algorithm is designed to align the raw data from the two types of sensors, generating a time-synchronized and spatially aligned fusion dataset.

[0008] Extract image features and heat source features from the fused dataset and fuse them to generate a multi-dimensional feature set;

[0009] Based on the multidimensional feature set, determine whether there are obstacles in the original flight path. If there are obstacles, extract obstacle information, including position and motion state.

[0010] Analyze real-time environmental data, calculate the environmental complexity index, and generate obstacle risk levels by combining obstacle information;

[0011] Based on the dynamic risk level and environmental complexity index, predict the dynamic weight values ​​of the visual sensor and the infrared sensor, and adjust the sampling frequency and working mode of the sensor accordingly.

[0012] Based on the location, motion state, and dynamic weight value of obstacles, the comprehensive risk value of each path is calculated through the path risk model, and the movement trend of obstacles is predicted.

[0013] Based on the prediction results of the path risk model and the obstacle movement trend, the optimal obstacle avoidance path is generated, including path length, path curvature and path safety, and the corresponding flight command is generated.

[0014] Furthermore, the step of collecting flight data using the UAV's visual and infrared sensors, obtaining timestamps and spatial coordinate information, and calculating the sampling frequency difference between the two types of sensors includes:

[0015] The raw data from the visual sensor and the infrared sensor are acquired separately, and the corresponding timestamps and spatial coordinate information are extracted.

[0016] Kalman filtering is applied to spatial coordinates to eliminate noise errors;

[0017] The difference in sampling frequency between the visual sensor and the infrared sensor is calculated based on the timestamp.

[0018] Furthermore, the step of designing an adaptive interpolation algorithm to align the raw data of the two types of sensors based on the sampling frequency difference, and generating a time-synchronized and spatially aligned fusion dataset, includes:

[0019] If the sampling frequency difference exceeds a preset threshold, the data collected by the high-frequency sensor will be downsampled.

[0020] Based on the collected raw data, the time series of the two types of sensors are aligned using a convolutional neural network, and the interpolation step size is adjusted according to the rate of change of spatial coordinates.

[0021] Spatial alignment of image data from the two types of sensors is achieved through feature matching;

[0022] The spatiotemporally aligned data is input into the mapping model to generate a fusion dataset that includes the correlation between visual and infrared data.

[0023] Further, the step of extracting image features and heat source features from the fused dataset and fusing them to generate a multi-dimensional feature set includes:

[0024] Image features of edge, texture and shape are extracted from the visual data of the fused dataset, and heat source features of heat source distribution and intensity are extracted from the infrared data;

[0025] An attention mechanism is used to fuse extracted image features and heat source features to generate a multidimensional feature set containing joint spatial-heat source information.

[0026] Further, the step of determining whether there are obstacles on the original flight path based on the multi-dimensional feature set, and extracting obstacle information if they exist, includes:

[0027] The multidimensional feature set is input into a pre-trained convolutional neural network to determine the existence of obstacles;

[0028] If obstacles exist, the position information of all obstacles is calculated through coordinate transformation, including the distance and azimuth of each obstacle relative to the UAV. The trajectory of all obstacles is tracked based on the continuous frame feature set, and the motion information of each obstacle is calculated, including the speed and direction of each obstacle.

[0029] Furthermore, the steps of analyzing real-time environmental data, calculating the environmental complexity index, and generating obstacle risk levels by combining obstacle information include:

[0030] Environmental data within a defined range is acquired through multiple sensors, including light intensity, temperature distribution, and wind speed. The environmental data is then normalized, and the environmental complexity index is calculated using the entropy method.

[0031] Based on the obstacle speed threshold, obstacles are classified to distinguish between static and dynamic obstacles;

[0032] Based on the environmental complexity index and obstacle classification results, the risk level of all obstacles is calculated using a pre-established risk level model.

[0033] Furthermore, the step of predicting the dynamic weight values ​​of the visual sensor and the infrared sensor based on the dynamic risk level and environmental complexity index, and adjusting the sampling frequency and operating mode of the sensors accordingly, includes:

[0034] Based on obstacle risk level and environmental complexity index, the weights of visual and infrared sensors are predicted using an LSTM model.

[0035] The sampling frequency of the sensors is adjusted according to the predicted weight ratio of the visual sensor and the infrared sensor.

[0036] When the environmental index exceeds the set threshold, the visual sensor is switched to high-resolution mode; when the risk level exceeds the set threshold, the sensitivity of the infrared sensor is increased.

[0037] Furthermore, the step of calculating the comprehensive risk value of each path and predicting the obstacle movement trend based on the obstacle's position, motion state, and dynamic weight value using a path risk model includes:

[0038] Static risk is calculated based on the distance between the obstacle and the drone and the density of the obstacle distribution. Dynamic risk is calculated based on the probability of a collision between a moving obstacle and the drone.

[0039] Based on the weights of visual and infrared sensors, the static and dynamic risks corresponding to visual and infrared sensors are weighted and combined to obtain the comprehensive risk value of each possible path.

[0040] Kalman filtering is used to predict the future positions of each obstacle. If the predicted future position of an obstacle overlaps with the original flight path, dynamic risk weights are added, and reinforcement learning is used to optimize and adjust the weight parameters of each factor in the path risk assessment model.

[0041] Furthermore, the step of generating the optimal obstacle avoidance path and the corresponding flight command based on the prediction results of the path risk model and the obstacle movement trend includes:

[0042] Based on the comprehensive risk value, a series of candidate paths are generated;

[0043] Based on the risk level, path length, and curvature of each candidate path, the A* algorithm is used to select the path with the best overall performance, thus obtaining the obstacle avoidance path.

[0044] Based on the drone's own power performance and flight control constraints, control commands corresponding to the obstacle avoidance path are generated, including the drone's turning angle, acceleration, and the time range for executing the control operation.

[0045] Further, after the step of generating the optimal obstacle avoidance path, the following steps are included:

[0046] Real-time monitoring of the drone's flight status, including remaining battery power, flight altitude, and real-time environmental data updates;

[0047] If the environmental complexity index or obstacle risk level changes beyond the preset threshold, the sensor dynamic weight value will be recalculated and the path risk model parameters will be updated.

[0048] Based on the updated comprehensive risk value, the candidate path set is dynamically adjusted, and the path safety factor is reassessed.

[0049] If the risk difference between the current optimal path and the new candidate path exceeds a threshold, path replanning is triggered to generate an updated obstacle avoidance path. During path execution, the UAV attitude is corrected in real time through a PID controller.

[0050] The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion provided by this invention has the following beneficial effects: In data processing, this invention designs an adaptive interpolation algorithm to solve the differences in sensor sampling frequency and format, achieving accurate spatiotemporal alignment, generating a high-quality fusion dataset, and improving data availability and processing accuracy. During risk assessment, the dynamic changes in the environment and obstacles are considered. The entropy method is used to calculate the environmental complexity index, and the risk level model and LSTM model are combined to predict sensor weights, making the assessment more realistic and ensuring flight safety. Obstacle avoidance path planning adopts innovative models and algorithms, comprehensively calculating risk values ​​from multiple factors, selecting the optimal path, reducing computational complexity, meeting real-time requirements, and enabling real-time monitoring and on-demand path replanning. In terms of system protection, a comprehensive mechanism is constructed to monitor sensors in real time and adopt emergency strategies for different anomalies. After extreme weather testing, it also exhibits good adaptability and reliability in complex and harsh environments. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of unmanned aerial vehicles in one embodiment of the present invention.

[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] Reference Figure 1 This is a flowchart illustrating an AI real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion proposed in this invention, including the following steps:

[0055] S1 collects flight data through the drone's visual and infrared sensors, obtains timestamps and spatial coordinate information, and calculates the difference in sampling frequencies between the two types of sensors;

[0056] S2, Based on the sampling frequency difference, an adaptive interpolation algorithm is designed to align the raw data of the two types of sensors and generate a time-synchronized and spatially aligned fusion dataset;

[0057] S3, extract image features and heat source features from the fused dataset, and fuse them to generate a multi-dimensional feature set;

[0058] S4. Based on the multi-dimensional feature set, determine whether there are obstacles in the original flight path. If there are obstacles, extract obstacle information, including position and motion state.

[0059] S5 analyzes real-time environmental data, calculates the environmental complexity index, and generates obstacle risk levels by combining obstacle information;

[0060] S6. Based on the dynamic risk level and environmental complexity index, predict the dynamic weight values ​​of the visual sensor and the infrared sensor, and adjust the sampling frequency and working mode of the sensor accordingly.

[0061] S7, based on the location, motion state, and dynamic weight value of obstacles, calculates the comprehensive risk value of each path through the path risk model and predicts the movement trend of obstacles;

[0062] S8 generates the optimal obstacle avoidance path based on the prediction results of the path risk model and the obstacle movement trend, including path length, path curvature and path safety, and generates the corresponding flight command.

[0063] As described in step S1 above, raw data collected by the UAV's visual and infrared sensors are acquired respectively. This raw data contains various information from the UAV's flight process, from which timestamps and spatial coordinate information are extracted. The timestamp records the moment of data acquisition, and the spatial coordinate information reflects the UAV's position at the time of data acquisition. Since sensors inevitably introduce noise errors during data acquisition, affecting the accuracy of the spatial coordinate information, a Kalman filter is used to correct the spatial coordinates to eliminate these noise errors. Kalman filtering is an algorithm that uses the state equations of a linear system to optimally estimate the system state using system input and output observation data. In this step, the specific implementation of the Kalman filter error compensation factor is as follows: In Kalman filtering, the calculation of the error compensation factor K is crucial. Let the predicted state covariance be P. k|k-1 The measurement error covariance is R. k The observation matrix is ​​H k Then the error compensation factor By continuously updating K kKalman filtering can effectively compensate for noise in sensor data, improving data accuracy. Based on the previously extracted timestamp information, the sampling frequency difference between the visual sensor and the infrared sensor is calculated. The sampling frequency determines the density of data acquisition by the sensor; different sensors may have different sampling frequencies. By calculating the difference in their sampling frequencies, the time interval differences in data acquisition between the two sensors can be understood, providing an important basis for subsequent data alignment and fusion.

[0064] As described in step S2 above, when the difference in sampling frequencies between the visual sensor and the infrared sensor calculated in step S1 exceeds a preset threshold, it indicates a significant difference in the frequency of data collected by the two types of sensors. To facilitate subsequent data alignment and fusion, it is necessary to downsample the data collected by the high-frequency sensor. By reducing the amount of data from the high-frequency sensor, its data collection frequency is brought closer to that of the low-frequency sensor to a certain extent. For example, if the sampling frequency of the visual sensor is much higher than that of the infrared sensor, and the difference exceeds the threshold, downsampling is performed on the visual sensor data. Based on the collected raw data, a spatiotemporal convolutional neural network is used to align the time series of the two types of sensors. The spatiotemporal convolutional neural network (ST-CNN) of this invention learns the spatiotemporal correlation between visual and infrared sensor data through training, adaptively generates interpolation weights, and solves the alignment error problem of traditional interpolation algorithms in scenarios with violent motion. For example, when a drone is flying at high speed, traditional methods may fail to align due to motion blur, while ST-CNN can predict a more accurate interpolation position by learning motion patterns in historical data. At the same time, the interpolation step size is adjusted according to the rate of change of spatial coordinates. The rate of change of spatial coordinates reflects the changes in the speed and direction of the UAV's movement in space. By adjusting the interpolation step size according to this rate of change, the data can be interpolated more accurately during the time-series alignment process, making the aligned data more consistent with the actual motion. For image data from both types of sensors, a feature matching method is used for spatial alignment. Feature points, such as corners and edges, are extracted from the images, and then matching feature point pairs are found between images from different sensors. Based on these matching feature point pairs, spatial transformation relationships between images are calculated, such as translation, rotation, and scaling, thereby aligning the images from different sensors in space. This ensures that image information acquired from different angles accurately corresponds, and the spatiotemporally aligned data is input into the mapping model. The mapping model establishes a correlation between visual and infrared data based on the characteristics of the input data and pre-defined rules, ultimately generating a fusion dataset containing the visual-infrared data correlation. This fusion dataset integrates the advantages of visual and infrared sensors, providing a comprehensive and accurate data foundation for subsequent feature extraction, obstacle detection, and path planning.

[0065] As described in step S3 above, edge, texture, and shape features are extracted from the visual data portion of the fused dataset using specific algorithms and techniques. These image features reflect information such as the contours, surface details, and geometry of objects in the environment surrounding the drone, helping to identify different objects and scenes. For example, edge detection algorithms can determine the boundaries of objects, and texture analysis can distinguish objects of different materials. Simultaneously, heat source distribution and intensity features are extracted from the infrared data. Heat source distribution shows the location and range of heat sources in the environment, while intensity features reflect the energy level of the heat source, enabling the detection of objects generating heat, such as moving machines or animals. A Transformer architecture with multimodal feature fusion is used to fuse the extracted image features and heat source features. The attention mechanism is upgraded to a multimodal Transformer, dynamically allocating weights for visual and infrared features through a self-attention mechanism, and introducing a cross-modal attention layer to capture complementary information from the two sensor data. For example, in low-light environments, the weights of heat source features in infrared data are automatically enhanced, while the weights of texture features in visual data are reduced, thereby improving the robustness of obstacle detection. In this invention, the importance of image features and thermal source features is determined based on the environment and mission of the UAV, and then they are organically fused together to generate a multi-dimensional feature set containing joint spatial-thermal source information. This multi-dimensional feature set integrates the advantages of both visual and infrared sensor data, providing a more comprehensive description of the environmental information surrounding the UAV, enabling the UAV to perceive its surroundings more accurately in complex environments.

[0066] As described in step S4 above, obstacles are identified and relevant information is extracted based on a multi-dimensional feature set. The generated multi-dimensional feature set is input into a pre-trained convolutional neural network (CNN). The CNN is used to determine the existence of obstacles. It has been trained using a large number of image samples containing and without obstacles, enabling the network to learn the feature representation of obstacles. During training, the cross-entropy loss function and stochastic gradient descent optimization algorithm are used to continuously adjust the network parameters and improve the accuracy of the model. The CNN calculates whether there are obstacles on the original flight path. If obstacles exist, coordinate transformation is performed, and mathematical methods such as trigonometric functions are used to calculate the distance and azimuth angle of each obstacle relative to the UAV, thereby determining the position information of the obstacles. At the same time, based on the continuous frame feature set, a target tracking algorithm is used to track the trajectory of all obstacles. By calculating the changes in the position of obstacles in continuous frames, the movement speed and direction of each obstacle are obtained, comprehensively acquiring the movement information of the obstacles.

[0067] As described in step S5 above, environmental data such as light intensity, temperature distribution, and wind speed within a defined range are collected using multiple sensors. This data is normalized to the [0,1] interval to eliminate the influence of data dimensions. Next, the entropy method is used to calculate the environmental complexity index. The entropy method determines the weight of each environmental factor based on the dispersion of the data, thus obtaining the environmental complexity index. Based on a preset obstacle speed threshold, obstacles with speeds below the threshold are classified as static obstacles, while those above the threshold are classified as dynamic obstacles. Finally, based on the environmental complexity index and the obstacle classification results, the risk level of all obstacles is calculated using a pre-established risk level model and weighted calculation methods.

[0068] As described in step S6 above, the obstacle risk level and environmental complexity index are used as inputs, and a Long Short-Term Memory (LSTM) network model is used to predict the weights of the visual sensor and the infrared sensor. The LSTM model can process time-series data and capture data change trends, thereby predicting appropriate weights. Based on the predicted weight ratios, the sampling frequencies of the visual sensor and the infrared sensor are adjusted; the sampling frequency of the sensor with a higher weight is increased, and vice versa. When the environmental index exceeds a set threshold, such as when the light intensity is extremely low, the visual sensor is switched to high-resolution mode to obtain clearer images; when the risk level exceeds a set threshold, i.e., when the obstacle is highly dangerous, the sensitivity of the infrared sensor is increased to enhance the obstacle detection capability. Let the visual sensor data be V, the infrared sensor data be I, the environmental complexity index be E, and the obstacle risk level be R. First, a joint feature vector F = [V, I, E, R] is defined. The weights are predicted by the LSTM model, assuming the output of the LSTM model is W = [ω]. v ,ω i ], where ω v For the visual sensor weights, ω i The weights are for the infrared sensors. Through training on a large amount of sample data, the model learns the optimal weight allocation under different environmental and obstacle conditions. The specific loss function can be defined as follows: Where y n For the true weight allocation, The weights are assigned to the model for prediction, where N is the number of samples. The model parameters are adjusted by minimizing the loss function to obtain the optimal weight prediction model.

[0069] As described in step S7 above, the risk value is calculated and the obstacle movement trend is predicted using the path risk model. First, static risk is calculated based on the distance and density of obstacles to the drone; the closer the obstacle and the denser the distribution, the higher the static risk. Simultaneously, dynamic risk is calculated based on the probability of a collision between the obstacle and the drone during movement; the higher the probability of a collision, the higher the dynamic risk. Then, based on the weights of the visual and infrared sensors, the static and dynamic risks corresponding to the two types of sensors are weighted and combined to obtain the comprehensive risk value for each possible path. The Kalman filter algorithm is used to predict the future position of each obstacle. If the predicted future position of an obstacle coincides with the original flight path, the dynamic risk weight is increased to highlight the degree of danger. In the risk assessment stage, a deep reinforcement learning (DRL) module is added to optimize the weight parameters of the risk level model in real time. The model uses the environmental complexity index and obstacle dynamic information as the state space, and sensor weight adjustment and path planning actions as the action space. It iterates the strategy through reward functions (such as obstacle avoidance success rate and path efficiency) to achieve adaptive decision-making in dynamic environments.

[0070] As described in step S8 above, based on the calculated comprehensive risk value, a series of candidate paths are generated according to certain rules. For example, a hybrid path planning strategy is generated using a Deep Q-Network (DQN) to generate multiple possible paths starting from the current position and according to different angles and distances. The DQN learns obstacle avoidance experience in complex environments through offline training, quickly generating candidate paths during online planning, and then optimizing path details using the A algorithm. Based on the risk level, path length, and curvature of each candidate path, the A algorithm is used for searching. The A algorithm combines the advantages of breadth-first search and best-first search of Dijkstra's algorithm, selecting the path with the best overall performance—that is, a path with low risk, short length, and minimal curvature—as the obstacle avoidance path through an evaluation function. Finally, based on the UAV's own power performance and flight control constraints, such as maximum steering angle and maximum acceleration, control commands corresponding to the obstacle avoidance path are generated, including the UAV's steering angle, acceleration, and the time range for executing control operations, ensuring that the UAV can fly according to the planned path. After generating the optimal obstacle avoidance path, the UAV's remaining battery power, flight altitude, and real-time environmental data updates are continuously monitored in real time. If the environmental complexity index or obstacle risk level changes beyond a preset threshold, it indicates a significant change in the environment or obstacle situation. In this case, the sensor dynamic weight values ​​are recalculated, and the path risk model parameters are updated to adapt to the new situation. Based on the updated comprehensive risk value, the candidate path set is dynamically adjusted, and the path safety factor is reassessed. If the risk difference between the current optimal path and the new candidate path exceeds a threshold, it means the current path is no longer the best choice, triggering path replanning and generating an updated obstacle avoidance path. During path execution, the UAV attitude is corrected in real time by a PID controller. The PID controller adjusts the control input through proportional, integral, and derivative operations based on the deviation between the setpoint and the actual value to ensure stable flight of the UAV.

[0071] In one embodiment, drones are used for urban logistics delivery. During flight, the drone simultaneously acquires data using visual and infrared sensors. The visual sensor captures high-resolution images, while the infrared sensor detects heat source distribution in the environment. Kalman filtering is used to correct the timestamps of the visual and infrared sensors, eliminating time errors. ST-CNN (Spatiotemporal Convolutional Neural Network) is used to perform spatiotemporal alignment of the visual and infrared data. ST-CNN learns motion patterns from historical data to predict interpolation positions. Experimental results show that the alignment error is reduced from ±15cm in traditional methods to ±5cm. Edge, texture, and shape features are extracted from the visual data, and heat source distribution and intensity features are extracted from the infrared data. A self-attention mechanism dynamically assigns weights to the visual and infrared features. In the nighttime scene of this embodiment, the weight of the infrared data is automatically increased to 0.8, while the weight of the visual data is decreased to 0.2. The fused multidimensional feature set significantly improves the accuracy of obstacle detection. Experimental data shows that the obstacle detection accuracy reaches 98.5% (compared to 85% in traditional methods).

[0072] The environmental complexity index was calculated using the entropy method after acquiring light intensity, temperature distribution, and wind speed values ​​from multiple sensors. During the day, the environmental complexity index was 0.3; at night, due to insufficient light, the index rose to 0.7. The weights of the visual and infrared sensors were predicted based on an LSTM model. At night, the weight of the infrared sensor increased to 0.8, while the weight of the visual sensor decreased to 0.2. Candidate paths were generated using DQN (Deep Q-Network), and then the path details were optimized using the A* algorithm. Experimental results showed that the average planning time was 120ms, and the path length was reduced by 12% compared to the traditional A* algorithm. In 100 tests, the UAV successfully avoided obstacles 99 times, achieving a success rate of 99.2% (compared to 89% for the traditional method). In a simulated rainstorm environment (visibility <10m), the infrared sensor dominated detection, maintaining a 92% obstacle avoidance success rate.

[0073] In another embodiment, a drone is used for emergency rescue in a sandstorm environment. Sensor configuration: a visual sensor with 720p resolution and a 20Hz sampling frequency, switched to infrared-assisted mode to reduce noise; and an infrared sensor with 640×512 resolution and a 25Hz sampling frequency, with sensitivity increased to 0.1℃, used to detect heat sources (trapped personnel) in the sandstorm. The simulated sandstorm environment includes a wind speed of 15m / s, visibility <5m, and high sand particle density.

[0074] In dust storm environments, visual sensors are limited by low visibility, relying primarily on infrared sensors to detect heat sources. Kalman filtering is used to correct the timestamps of both visual and infrared sensors, eliminating time errors. ST-CNN is used for spatiotemporal alignment of visual and infrared data. Due to the intense motion in dust storm environments, traditional methods have large alignment errors (±2m), while ST-CNN learns motion patterns from historical data, reducing the error to ±0.5m. The Kalman filter error compensation factor is calculated using the following formula: in, It is the predicted state covariance, H k It is the observation matrix, R k It is the measurement error covariance. By continuously updating K... k Kalman filtering can effectively compensate for noise errors in sensor data. ST-CNN alignment error formula: Where, x 视觉 and y 视觉 These are the coordinates of the vision sensor, x 红外 and y 红外 These are the coordinates of the infrared sensor. ST-CNN learns motion patterns from historical data to predict interpolation positions, reducing alignment errors to ±0.5m. Multimodal feature fusion extracts edge and texture features from visual data (capturing some information despite low visibility) and heat source distribution and intensity features from infrared data. In dust storm environments, the weight of infrared data is automatically increased to 0.8, while the weight of visual data is decreased to 0.2. The fused multidimensional feature set significantly improves the accuracy of heat source detection. Experimental data shows that the heat source detection accuracy reaches 98.7%. The calculation formula for the multimodal Transformer weight allocation is as follows: Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k It is a dimension. Through the self-attention mechanism, the Transformer dynamically assigns weights to visual and infrared features.

[0075] Wind speed, temperature distribution, and dust concentration were acquired using multiple sensors, and the environmental complexity index was calculated using the entropy method. In a dust storm environment, the environmental complexity index was 0.8. The calculation formula is as follows: Where, p i This represents the normalized values ​​of various environmental factors (such as wind speed, temperature, and dust concentration). The environmental complexity index is calculated using the entropy method. The weights of visual and infrared sensors are predicted based on an LSTM model. In a dust storm environment, the weight of the infrared sensor is increased to 0.8, while the weight of the visual sensor is decreased to 0.2. LSTM weight prediction formula: LSTM 输出 =σ(W f *[h t-1 ,x t ]+bf , where w f It is the weight matrix, h t-1 It is the hidden state from the previous moment, x t This is the current input, b f This is the bias term. The LSTM model predicts the weights of the visual and infrared sensors using time-series data. Candidate paths are generated using DQN, and then the path details are optimized using the A algorithm. Experimental results show that the average planning time is 200ms, and the path length is reduced by 15% compared to the traditional A algorithm.

[0076] In 50 tests, the drone successfully avoided obstacles 45 times, achieving a success rate of 90% (compared to 65% for traditional methods). When the dust concentration suddenly increased, the environmental complexity index rose from 0.6 to 0.9, triggering path replanning. DQN generated a path around the dust, and the PID controller adjusted the drone's attitude in real time, with a pitch angle error of <1°.

[0077] In this embodiment, the computational speed of the UAV is optimized. Multi-threaded parallel computing is employed, distributing data acquisition, feature extraction, risk assessment, and path planning tasks across multiple threads for parallel execution. This multi-threading optimization improves the overall computational speed by 30%. Furthermore, deep learning models (such as ST-CNN, Transformer, and LSTM) are quantized from 32-bit floating-point to 8-bit integers, reducing computational load and memory usage. After quantization, model inference speed is increased by 2 times, and memory usage is reduced by 50%. Simultaneously, pruning techniques are used to remove redundant neurons and connections, reducing model complexity. After pruning, model computation is reduced by 40%, and inference speed is increased by 1.5 times. Commonly used environmental and sensor data are cached in high-speed memory, reducing data retrieval time. Through data caching, data retrieval time is reduced from 20ms to 5ms. During UAV flight, multiple candidate paths are pre-calculated, and the results are cached. When the environment changes, the pre-calculated paths are directly retrieved from the cache, reducing real-time planning time. Experimental data shows that path pre-calculation reduces planning time from 200ms to 100ms.

[0078] In summary, this method collects flight data using the UAV's visual and infrared sensors, obtaining timestamps and spatial coordinates, and calculates the sampling frequency difference between the two types of sensors. Based on this sampling frequency difference, an adaptive interpolation algorithm is designed to align the raw data from both sensors, generating a time-synchronized and spatially aligned fusion dataset. Image features and heat source features are extracted from the fusion dataset and fused to generate a multi-dimensional feature set. Based on this multi-dimensional feature set, it is determined whether there are obstacles on the original flight path; if so, obstacle information is extracted. Real-time environmental data is analyzed, and an environmental complexity index is calculated. An obstacle risk level is generated by combining the obstacle information. Based on the dynamic risk level and environmental complexity index, the dynamic weight values ​​of the visual and infrared sensors are predicted, and the sampling frequency and operating mode of the sensors are adjusted accordingly. Based on the obstacle's position, motion state, and dynamic weight values, a path risk model is used to calculate the comprehensive risk value of each path and predict the obstacle's movement trend. Based on the prediction results of the path risk model and the obstacle's movement trend, the optimal obstacle avoidance path is generated, and corresponding flight commands are generated. This achieves efficient processing of sensor data, accurate risk assessment, and real-time obstacle avoidance path planning, enabling stable and safe flight of the UAV in complex environments.

[0079] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the present invention and embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM, etc.

[0080] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0081] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for real-time AI-based intelligent guidance and adaptive obstacle avoidance using multi-source perception fusion in unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: Flight data is collected by the drone's visual and infrared sensors to obtain timestamps and spatial coordinate information, and the difference in sampling frequency between the two types of sensors is calculated. Based on the sampling frequency difference, an adaptive interpolation algorithm is designed to align the raw data of the two types of sensors and generate a time-synchronized and spatially aligned fusion dataset. Extract image features and heat source features from the fused dataset and fuse them to generate a multi-dimensional feature set; Based on the multidimensional feature set, determine whether there are obstacles in the original flight path. If there are obstacles, extract obstacle information, including position and motion state. Analyze real-time environmental data, calculate the environmental complexity index, and generate obstacle risk levels by combining obstacle information; Based on the obstacle risk level and environmental complexity index, predict the dynamic weight values ​​of the visual sensor and the infrared sensor, and adjust the sampling frequency and working mode of the sensor accordingly. Based on the location, motion state, and dynamic weight value of obstacles, the comprehensive risk value of each path is calculated through the path risk model, and the movement trend of obstacles is predicted. Based on the prediction results of the path risk model and the obstacle movement trend, the optimal obstacle avoidance path is generated, including path length, path curvature and path safety, and the corresponding flight command is generated.

2. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion according to claim 1, characterized in that, The steps of collecting flight data using the UAV's visual and infrared sensors, obtaining timestamps and spatial coordinate information, and calculating the sampling frequency difference between the two types of sensors include: The raw data from the visual sensor and the infrared sensor are acquired separately, and the corresponding timestamps and spatial coordinate information are extracted. Kalman filtering is applied to spatial coordinates to eliminate noise errors; The difference in sampling frequency between the visual sensor and the infrared sensor is calculated based on the timestamp.

3. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion according to claim 1, characterized in that, The step of designing an adaptive interpolation algorithm to align the raw data of the two types of sensors based on the sampling frequency difference value, and generating a time-synchronized and spatially aligned fusion dataset, includes: If the sampling frequency difference exceeds a preset threshold, the data collected by the high-frequency sensor will be downsampled. Based on the collected raw data, the time series of the two types of sensors are aligned using a convolutional neural network, and the interpolation step size is adjusted according to the rate of change of spatial coordinates. Spatial alignment of image data from the two types of sensors is achieved through feature matching; The spatiotemporally aligned data is input into the mapping model to generate a fusion dataset containing the correlation between visual and infrared data.

4. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion according to claim 1, characterized in that, The step of extracting image features and heat source features from the fused dataset and fusing them to generate a multidimensional feature set includes: Image features of edge, texture and shape are extracted from the visual data of the fused dataset, and heat source features of heat source distribution and intensity are extracted from the infrared data; An attention mechanism is used to fuse extracted image features and heat source features to generate a multidimensional feature set containing joint spatial-heat source information.

5. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion according to claim 1, characterized in that, The step of determining whether there are obstacles on the original flight path based on the multi-dimensional feature set, and extracting obstacle information if they exist, includes: The multidimensional feature set is input into a pre-trained convolutional neural network to determine the existence of obstacles; If obstacles exist, the position information of all obstacles is calculated through coordinate transformation, including the distance and azimuth of each obstacle relative to the UAV. The trajectory of all obstacles is tracked based on the continuous frame feature set, and the motion information of each obstacle is calculated, including the speed and direction of each obstacle.

6. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion according to claim 1, characterized in that, The steps of analyzing real-time environmental data, calculating the environmental complexity index, and generating obstacle risk levels by combining obstacle information include: Environmental data within a defined range is acquired through multiple sensors, including light intensity, temperature distribution, and wind speed. The environmental data is then normalized, and the environmental complexity index is calculated using the entropy method. Based on the obstacle speed threshold, obstacles are classified to distinguish between static and dynamic obstacles; Based on the environmental complexity index and obstacle classification results, the risk level of all obstacles is calculated using a pre-established risk level model.

7. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion according to claim 1, characterized in that, The step of predicting the dynamic weight values ​​of the visual sensor and the infrared sensor based on the obstacle risk level and environmental complexity index, and adjusting the sampling frequency and working mode of the sensors accordingly, includes: Based on obstacle risk level and environmental complexity index, the weights of visual and infrared sensors are predicted using an LSTM model. The sampling frequency of the sensors is adjusted according to the predicted weight ratio of the visual sensor and the infrared sensor. When the environmental index exceeds the set threshold, the visual sensor is switched to high-resolution mode; when the obstacle risk level exceeds the set threshold, the sensitivity of the infrared sensor is increased.

8. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion according to claim 1, characterized in that, The step of calculating the comprehensive risk value of each path and predicting the obstacle movement trend based on the obstacle's position, movement state, and dynamic weight value using a path risk model includes: Static risk is calculated based on the distance between the obstacle and the drone and the density of the obstacle distribution. Dynamic risk is calculated based on the probability of a collision between a moving obstacle and the drone. Based on the weights of visual and infrared sensors, the static and dynamic risks corresponding to visual and infrared sensors are weighted and combined to obtain the comprehensive risk value of each possible path. Kalman filtering is used to predict the future positions of each obstacle. If the predicted future position of an obstacle overlaps with the original flight path, dynamic risk weights are added, and reinforcement learning is used to optimize and adjust the weight parameters of each factor in the path risk assessment model.

9. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for UAVs based on multi-source perception fusion according to claim 1, characterized in that, The step of generating the optimal obstacle avoidance path and corresponding flight commands based on the prediction results of the path risk model and the obstacle movement trend includes: Based on the comprehensive risk value, a series of candidate paths are generated; Based on the risk level, path length, and curvature of each candidate path, the A* algorithm is used to select the path with the best overall performance, thus obtaining the obstacle avoidance path. Based on the drone's own power performance and flight control constraints, control commands corresponding to the obstacle avoidance path are generated, including the drone's turning angle, acceleration, and the time range for executing the control operation.

10. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for UAV multi-source perception fusion according to claim 9, characterized in that, After the step of generating the optimal obstacle avoidance path, the following steps are included: Real-time monitoring of the drone's flight status, including remaining battery power, flight altitude, and real-time environmental data updates; If the environmental complexity index or obstacle risk level changes beyond the preset threshold, the sensor dynamic weight value will be recalculated and the path risk model parameters will be updated. Based on the updated comprehensive risk value, the candidate path set is dynamically adjusted, and the path safety factor is reassessed. If the risk difference between the current optimal path and the new candidate path exceeds a threshold, path replanning is triggered to generate an updated obstacle avoidance path. During path execution, the UAV attitude is corrected in real time through a PID controller.

Images