Unmanned aerial vehicle multi-source sensing fusion AI real-time intelligent guidance and adaptive obstacle avoidance method

Abstract

The invention relates to an AI real-time intelligent guidance and self-adaptive obstacle avoidance method for multi-source sensing fusion of an unmanned aerial vehicle, and the method comprises the steps: collecting flight data through a visual sensor and an infrared sensor, obtaining a timestamp and space coordinate information, calculating a sampling frequency difference, and generating a fusion data set of time-space alignment through a self-adaptive interpolation algorithm; image and heat source features in the fusion data set are extracted, and a multi-dimensional feature set is generated, so that whether an obstacle exists or not is judged, and related information is extracted; and analyzing environment data, calculating an environment complexity index, generating a risk level in combination with obstacle information, predicting a dynamic weight of the sensor, and adjusting a working mode of the sensor. And then based on the obstacle condition, calculating a comprehensive risk value and predicting a motion trend through a path risk model, and finally generating an optimal obstacle avoidance path and a flight instruction, thereby realizing high-efficiency processing of sensor data, accurate risk assessment and real-time obstacle avoidance planning, and ensuring stable and safe flight of the unmanned aerial vehicle in a complex environment.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and particularly to an AI real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of unmanned aerial vehicles. Background Art

[0002] In recent years, unmanned aerial vehicle technology has made rapid progress, and its application fields have been continuously expanded, covering many industries such as logistics distribution, geographical mapping, agricultural and forestry plant protection, emergency rescue, and film shooting. However, when an unmanned aerial vehicle flies in a complex environment, the problems of obstacle avoidance and navigation are still the key bottlenecks restricting its wide application. Traditional obstacle avoidance methods mainly rely on a single sensor, such as an ultrasonic sensor or a simple vision sensor. The ultrasonic sensor is limited by a short detection distance and is easily interfered by environmental noise, and performs poorly in detecting obstacles at a long distance; a simple vision sensor has greatly reduced capabilities in identifying and detecting obstacles under low light, bad weather (such as heavy rain, sand and dust, etc.) and complex texture environments, and it is difficult to meet the requirements for the safe flight of an unmanned aerial vehicle in a complex environment.

[0003] The rise of multi-source perception fusion technology has brought new opportunities to solve the above problems. By fusing various types of sensor data, such as vision sensors and infrared sensors, it is theoretically possible to more comprehensively perceive the surrounding environmental information. However, there are still many deficiencies in the current multi-source perception fusion methods in practical applications. In the sensor data processing link, there are differences in the sampling frequencies and data formats between different sensors, resulting in greater difficulty in data fusion, and it is difficult to achieve accurate time synchronization and spatial alignment, thus affecting the quality of the fused data. In terms of risk assessment, existing methods often fail to fully consider environmental factors and the dynamic changes of obstacles, and the risk assessment results are not accurate and comprehensive enough to provide a reliable basis for the decision-making of an unmanned aerial vehicle. In terms of obstacle avoidance path planning, the planning algorithms usually have too high computational complexity and poor real-time performance, and it is difficult to meet the timeliness requirements for obstacle avoidance decisions during the rapid flight of an unmanned aerial vehicle. Summary of the Invention

[0004] The main object of the present invention is to provide an AI real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of unmanned aerial vehicles, so as to achieve efficient processing of sensor data, accurate risk assessment, and real-time obstacle avoidance path planning, and achieve the purpose of stable and safe flight of an unmanned aerial vehicle in a complex environment.

[0005] To achieve the above object, the present invention provides an AI real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of unmanned aerial vehicles, including the following steps:

[0006] Collect flight data through the vision sensor and infrared sensor of the unmanned aerial vehicle, obtain the timestamp and spatial coordinate information, and calculate the sampling frequency difference between the two types of sensors;

[0007] Based on the sampling frequency difference, design an adaptive interpolation algorithm to align the original data of the two types of sensors, and generate a fused dataset with time synchronization and spatial alignment;

[0008] Extract the image features and heat source features in the fused dataset, and fuse them to generate a multi-dimensional feature set;

[0009] Based on the multi-dimensional feature set, determine whether there are obstacles in the original flight path. If there are obstacles, extract obstacle information, including position and motion state;

[0010] Analyze the real-time environmental data, calculate the environmental complexity index, and generate an obstacle risk level in combination with the obstacle information;

[0011] According to the dynamic risk level and the environmental complexity index, predict the dynamic weight values of the visual sensor and the infrared sensor, and correspondingly adjust the sampling frequency and working mode of the sensors;

[0012] Based on the position, motion state of the obstacle and the dynamic weight value, calculate the comprehensive risk value of each path through a path risk model, and predict the motion trend of the obstacle;

[0013] According to the prediction results of the path risk model and the motion trend of the obstacle, generate an optimal obstacle avoidance path, including path length, path curvature and path safety, and generate corresponding flight instructions.

[0014] Further, the step of collecting flight data through the visual sensor and the infrared sensor of the unmanned aerial vehicle, obtaining the timestamp and spatial coordinate information, and calculating the sampling frequency difference between the two types of sensors includes:

[0015] Respectively obtain the original data of the visual sensor and the infrared sensor, and extract the corresponding timestamp and spatial coordinate information;

[0016] Perform Kalman filter correction on the spatial coordinates to eliminate noise errors;

[0017] Based on the timestamp, calculate the sampling frequency difference value between the visual sensor and the infrared sensor.

[0018] Further, the step of designing an adaptive interpolation algorithm based on the sampling frequency difference to align the original data of the two types of sensors and generate a fused dataset with time synchronization and spatial alignment includes:

[0019] If the sampling frequency difference value exceeds a preset threshold, downsample the data collected by the high-frequency sensor;

[0020] Based on the collected original data, align the time series of the two types of sensors through a convolutional neural network, and adjust the interpolation step according to the spatial coordinate change rate;

[0021] Spatially align the image data of two types of sensors through feature matching;

[0022] Input the data after spatio-temporal alignment into a mapping model to generate a fusion data set containing the correlation between visual-infrared data.

[0023] Further, the step of extracting image features and heat source features from the fusion data set and fusing them to generate a multi-dimensional feature set includes:

[0024] Extract image features such as edges, textures, and shape features from the visual data of the fusion data set, and extract heat source features such as heat source distribution and intensity features from the infrared data;

[0025] Adopt an attention mechanism to fuse the extracted image features and heat source features to generate a multi-dimensional feature set containing spatial-heat source joint information.

[0026] Further, the step of judging whether there are obstacles in the original flight path based on the multi-dimensional feature set and extracting obstacle information if there are obstacles includes:

[0027] Input the multi-dimensional feature set into a pre-trained convolutional neural network to judge the existence of obstacles;

[0028] If there are obstacles, calculate the position information of all obstacles through coordinate transformation, including the distance and azimuth angle of each obstacle relative to the drone, track the trajectories of all obstacles based on the continuous frame feature set, and calculate the motion information of each obstacle, including the motion speed and direction of each obstacle.

[0029] Further, the step of analyzing real-time environmental data, calculating the environmental complexity index, and generating an obstacle risk level in combination with obstacle information includes:

[0030] Obtain environmental data within a limited range through multi-sensors, including light intensity, temperature distribution, and wind speed value, normalize the environmental data, and calculate the environmental complexity index through the entropy method;

[0031] Classify obstacles according to the obstacle speed threshold to distinguish static obstacles and dynamic obstacles;

[0032] According to the environmental complexity index and the obstacle classification result, calculate the risk level of all obstacles through a pre-established risk level model.

[0033] Further, the step of predicting the dynamic weight values of the visual sensor and the infrared sensor according to the dynamic risk level and the environmental complexity index and correspondingly adjusting the sampling frequency and working mode of the sensors includes:

[0034] Based on the obstacle risk level and the environmental complexity index, use the LSTM model to predict the weights of the visual sensor and the infrared sensor;

[0035] According to the predicted weight ratios of the visual sensor and the infrared sensor, adjust the sampling frequencies of the sensors respectively;

[0036] When the environmental index exceeds the set threshold, switch the visual sensor to the high-resolution mode; when the risk level exceeds the set threshold, increase the sensitivity of the infrared sensor.

[0037] Further, the step of calculating the comprehensive risk value of each path and predicting the movement trend of the obstacle through the path risk model based on the position, movement state and dynamic weight value of the obstacle includes:

[0038] Calculate the static risk according to the distance between the obstacle and the UAV and the density of the obstacle distribution, and calculate the dynamic risk according to the probability of collision between the moving obstacle and the UAV;

[0039] Based on the weights of the visual sensor and the infrared sensor, perform weighted synthesis on the static risk and dynamic risk corresponding to the visual sensor and the infrared sensor to obtain the comprehensive risk value of each possible path;

[0040] Use the Kalman filter to predict the future positions of the obstacles. If it is predicted that the future positions of the obstacles coincide with the original flight path, increase the dynamic risk weight, and optimize and adjust the weight parameters of each factor in the path risk assessment model through reinforcement learning.

[0041] Further, the step of generating the optimal obstacle avoidance path according to the prediction result of the path risk model and the movement trend of the obstacle and generating the corresponding flight instruction includes:

[0042] Generate a series of candidate paths according to the comprehensive risk value;

[0043] Based on the risk degree, path length and bending degree of each candidate path, select the path with the best overall performance through the A* algorithm to obtain the obstacle avoidance path;

[0044] Generate the control instructions corresponding to the obstacle avoidance path according to the power performance and flight control limit conditions of the UAV itself, including the steering angle, acceleration of the UAV and the time range for executing the control operation.

[0045] Further, after the step of generating the optimal obstacle avoidance path, it includes:

[0046] Real-time monitor the flight state of the UAV, including the remaining battery power, flight altitude and real-time environmental data update;

[0047] If the change in the environmental complexity index or the obstacle risk level exceeds the preset threshold, recalculate the dynamic weight value of the sensor and update the parameters of the path risk model;

[0048] According to the updated comprehensive risk value, dynamically adjust the candidate path set and re-evaluate the path safety factor;

[0049] If the risk difference between the current optimal path and the new candidate path exceeds the threshold, trigger path replanning, generate an updated obstacle avoidance path, and during the path execution, use a PID controller to correct the UAV attitude in real time.

[0050] The AI real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of UAVs provided by the present invention has the following beneficial effects: In data processing, the present invention designs an adaptive interpolation algorithm to solve the differences in sensor sampling frequencies and formats, achieve accurate spatio-temporal alignment, generate a high-quality fusion data set, and improve the data availability and processing accuracy. When evaluating risks, considering the dynamic changes of the environment and obstacles, using the entropy method to calculate the environmental complexity index, and combining the risk level model and the LSTM model to predict the sensor weights, making the evaluation more in line with the actual situation and ensuring flight safety. The obstacle avoidance path planning adopts innovative models and algorithms, calculates the risk value by integrating multiple factors, selects the optimal path, reduces the computational complexity, meets the real-time requirements, and can also monitor and replan the path as needed in real time. In terms of system guarantee, a perfect mechanism is constructed to monitor the sensors in real time, adopt emergency strategies for different anomalies, and after extreme weather tests, it also has good adaptability and reliability in complex and harsh environments. Description of the Drawings

[0051] Figure 1 is a schematic flowchart of the AI real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of UAVs in an embodiment of the present invention;

[0052] The implementation, functional features, and advantages of the object of the present invention will be further described in conjunction with the embodiments with reference to the drawings. Detailed Embodiments

[0053] In order to make the object, technical solution, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] Refer to Figure 1 , which is a schematic flowchart of an AI real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of UAVs proposed by the present invention, including the following steps:

[0055] S1, collect flight data through the visual sensor and infrared sensor of the UAV, obtain the timestamp and spatial coordinate information, and calculate the sampling frequency difference between the two types of sensors;

[0056] S2. Based on the sampling frequency difference, design an adaptive interpolation algorithm to align the original data of the two types of sensors, and generate a fused dataset with time synchronization and spatial alignment;

[0057] S3. Extract the image features and heat source features in the fused dataset, and fuse them to generate a multi-dimensional feature set;

[0058] S4. Based on the multi-dimensional feature set, determine whether there are obstacles in the original flight path. If there are, extract the obstacle information, including the position and motion state;

[0059] S5. Analyze the real-time environmental data, calculate the environmental complexity index, and generate the obstacle risk level in combination with the obstacle information;

[0060] S6. According to the dynamic risk level and the environmental complexity index, predict the dynamic weight values of the visual sensor and the infrared sensor, and correspondingly adjust the sampling frequency and working mode of the sensors;

[0061] S7. Based on the position, motion state and dynamic weight value of the obstacle, calculate the comprehensive risk value of each path through the path risk model, and predict the motion trend of the obstacle;

[0062] S8. According to the prediction results of the path risk model and the motion trend of the obstacle, generate the optimal obstacle avoidance path, including the path length, path curvature and path safety, and generate the corresponding flight instructions.

[0063] As described in step S1 above, obtain the original data collected by the visual sensor and the infrared sensor of the unmanned aerial vehicle (UAV) respectively. These original data contain various information during the UAV flight. Extract the timestamp and spatial coordinate information among them. The timestamp records the moment of data collection, and the spatial coordinate information reflects the position of the UAV when collecting data. Since noise errors are inevitably introduced during data collection by the sensors, affecting the accuracy of the spatial coordinate information. To eliminate these noise errors, the Kalman filtering method is used to correct the spatial coordinates. Kalman filtering is an algorithm that uses the linear system state equation and, through the system input-output observation data, optimally estimates the system state. In this step, the specific implementation method of the Kalman filtering error compensation factor: In Kalman filtering, the calculation of the error compensation factor K is crucial. Let the predicted state covariance be P k|k-1 , the measurement error covariance be R k , and the observation matrix be H k . Then the error compensation factor By continuously updating K k, it can enable the Kalman filter to effectively compensate for the noise in the sensor data and improve the accuracy of the data. Based on the timestamp information extracted previously, calculate the sampling frequency difference value between the vision sensor and the infrared sensor. The sampling frequency determines the density of the data collected by the sensor, and different sensors may have different sampling frequencies. By calculating the sampling frequency difference value between the two, it is possible to understand the difference in the time intervals of data collection by different sensors, providing an important basis for subsequent data alignment and fusion.

[0064] As described in step S2 above, when the sampling frequency difference value between the vision sensor and the infrared sensor calculated in step S1 exceeds the preset threshold, it indicates that the frequency gap between the data collected by the two types of sensors is relatively large. To facilitate subsequent data alignment and fusion, it is necessary to downsample the data collected by the high-frequency sensor. By reducing the amount of data of the high-frequency sensor, its data collection frequency is made to approach that of the low-frequency sensor to a certain extent. For example, if the sampling frequency of the vision sensor is much higher than that of the infrared sensor and the difference exceeds the threshold, a downsampling operation is performed on the vision sensor data. Based on the collected original data, use a spatio-temporal convolutional neural network to align the time series of the two types of sensors. The spatio-temporal convolutional neural network (ST-CNN) of the present invention learns the spatio-temporal correlation of the vision and infrared sensor data through training, and adaptively generates interpolation weights to solve the alignment error problem of traditional interpolation algorithms in scenes with intense motion. For example, when a drone is flying at high speed, traditional methods may fail to align due to motion blur, while ST-CNN can predict more accurate interpolation positions by learning the motion patterns in historical data. At the same time, adjust the interpolation step size according to the spatial coordinate change rate. The spatial coordinate change rate reflects the motion speed and direction change of the drone in space. By adjusting the interpolation step size according to this change rate, it is possible to more accurately interpolate the data during the time series alignment process, making the aligned data more in line with the actual motion situation. For the image data of the two types of sensors, use the method of feature matching for spatial alignment. By extracting feature points in the image, such as corner points, edges, etc., and then finding pairs of mutually matching feature points between the images of different sensors. According to these matching feature point pairs, calculate the spatial transformation relationship between the images, such as translation, rotation, scaling, etc., so as to align the images of different sensors in space, ensuring that the image information obtained from different angles corresponds accurately. Input the data after spatio-temporal alignment into the mapping model. The mapping model will establish the correlation relationship between the vision data and the infrared data according to the characteristics of the input data and the preset rules, and finally generate a fusion data set containing the vision-infrared data correlation relationship. This fusion data set integrates the advantageous information of the vision sensor and the infrared sensor, providing a comprehensive and accurate data basis for subsequent feature extraction, obstacle judgment, and path planning, etc.

[0065] As described in the above step S3, from the visual data part of the fusion dataset, specific algorithms and technologies are used to extract edge, texture, and shape features. These image features can reflect information such as the contours, surface details, and geometric shapes of objects in the environment around the UAV, helping to identify different objects and scenes. For example, the boundary of an object can be determined through an edge detection algorithm, and texture analysis can distinguish objects of different materials. At the same time, heat source distribution and intensity features are extracted from the infrared data. The heat source distribution can show the position and range of heat sources in the environment, and the intensity feature reflects the energy magnitude of the heat source, enabling the detection of heat-emitting objects such as running machines and animals. A Transformer architecture for multi-modal feature fusion is used to fuse the extracted image features and heat source features. The attention mechanism is upgraded to a multi-modal Transformer, which dynamically assigns weights to visual and infrared features through the self-attention mechanism and introduces a cross-modal attention layer to capture complementary information from the two sensor data. For example, in a low-light environment, the weight of the heat source feature of the infrared data is automatically enhanced, while the weight of the texture feature of the visual data is reduced, thereby improving the robustness of obstacle detection. In the present invention, according to the environment where the UAV is located and the tasks it faces, the importance levels of the image features and heat source features are determined, and then they are organically fused together to generate a multi-dimensional feature set containing spatial-heat source joint information. This multi-dimensional feature set integrates the advantages of the visual and infrared sensor data, more comprehensively describes the environmental information around the UAV, and enables the UAV to more accurately perceive the surrounding situation in a complex environment.

[0066] As described in the above step S4, obstacles are judged based on the multi-dimensional feature set and relevant information is extracted. The generated multi-dimensional feature set is input into a pre-trained convolutional neural network (CNN). The convolutional neural network (CNN) is used to judge the existence of obstacles. It has been trained with a large number of image samples containing obstacles and non-obstacles, enabling the network to learn the feature representations of obstacles. During the training process, a cross-entropy loss function and a stochastic gradient descent optimization algorithm are used to continuously adjust the network parameters and improve the accuracy of the model. Through the calculation of the CNN, it is judged whether there are obstacles in the original flight path. If there are obstacles, coordinate transformation is performed, and mathematical methods such as trigonometric functions are used to calculate the distance and azimuth angle of each obstacle relative to the UAV, thereby determining the position information of the obstacles. At the same time, based on the continuous frame feature set, a target tracking algorithm is used to track the trajectories of all obstacles. By calculating the changes in the positions of obstacles in consecutive frames, the movement speed and direction of each obstacle are obtained, and the movement information of the obstacles is comprehensively acquired.

[0067] As described in step S5 above, environmental data such as light intensity, temperature distribution, and wind speed values within a defined range are collected using multiple sensors. These data are normalized to the [0,1] interval to eliminate the influence of data dimensions. Then, the entropy method is used to calculate the environmental complexity index. The entropy method determines the weights of each environmental factor based on the degree of data dispersion, and thus obtains the environmental complexity index. According to the preset obstacle speed threshold, obstacles with a speed lower than the threshold are determined as static obstacles, and those higher than the threshold are determined as dynamic obstacles. Finally, based on the environmental complexity index and the obstacle classification results, through a pre-established risk level model, methods such as weighted calculation are used to calculate the risk levels of all obstacles.

[0068] As described in step S6 above, the obstacle risk level and the environmental complexity index are used as inputs, and a long short-term memory network (LSTM) model is used to predict the weights of the visual sensor and the infrared sensor. The LSTM model can process time series data and capture the data change trend, so as to predict appropriate weights. According to the predicted weight ratio, the sampling frequencies of the visual sensor and the infrared sensor are adjusted. The sensor with a higher weight correspondingly increases the sampling frequency, and vice versa. When the environmental index exceeds the set threshold, such as when the light intensity is extremely low, the visual sensor is switched to the high-resolution mode to obtain a clearer image; when the risk level exceeds the set threshold, that is, when the obstacle is highly dangerous, the sensitivity of the infrared sensor is increased to enhance the detection ability of the obstacle. Let the visual sensor data be V, the infrared sensor data be I, the environmental complexity index be E, and the obstacle risk level be R. First, a joint feature vector F = [V, I, E, R] is defined. The weights are predicted through the LSTM model. Assume the output of the LSTM model is W = [ω v , ω i , where ω v is the weight of the visual sensor, and ω i is the weight of the infrared sensor. Through the training of a large number of sample data, the model learns the optimal weight allocation in different environmental and obstacle situations. The specific loss function can be defined as where y n is the true weight allocation, is the weight allocation predicted by the model, and N is the number of samples. The model parameters are adjusted by minimizing the loss function to obtain the optimal weight prediction model.

[0069] As described in step S7 above, the risk value is calculated through the path risk model and the movement trend of the obstacle is predicted. First, the static risk is calculated according to the distance between the obstacle and the UAV and the density of the distribution. The closer the distance and the denser the distribution, the higher the static risk. At the same time, the dynamic risk is calculated according to the likelihood of collision between the moving obstacle and the UAV. The greater the likelihood of collision, the higher the dynamic risk. Then, based on the weights of the visual sensor and the infrared sensor, the static risk and the dynamic risk corresponding to the two types of sensors are weighted and synthesized to obtain the comprehensive risk value of each possible path. The Kalman filtering algorithm is used to predict the future positions of the obstacles. If it is predicted that the future positions of the obstacles coincide with the original flight path, the weight of the dynamic risk is increased to highlight the degree of danger. In the risk assessment stage, a new deep reinforcement learning (DRL) module is added to optimize the weight parameters of the risk level model in real time. The model takes the environmental complexity index and the dynamic information of the obstacles as the state space, and the sensor weight adjustment and the path planning action as the action space, and performs policy iteration through the reward function (such as the obstacle avoidance success rate and the path efficiency) to achieve adaptive decision-making in a dynamic environment.

[0070] As described in step S8 above, based on the calculated comprehensive risk value, a series of candidate paths are generated according to certain rules. For example, through a Deep Q-Network (DQN), a hybrid path planning strategy is generated that takes the current position as the starting point and generates multiple possible paths at different angles and distances. The DQN learns obstacle avoidance experience in complex environments through offline training, quickly generates candidate paths during online planning, and then optimizes the path details through the A* algorithm. Based on the risk level, path length, and curvature of each candidate path, the A* algorithm is used for searching. The A* algorithm combines the advantages of the breadth-first search and best-first search of the Dijkstra algorithm, and selects the path with the best overall performance through an evaluation function, that is, a path with low risk, short path, and small curvature as the obstacle avoidance path. Finally, according to the dynamic performance and flight control limit conditions of the UAV itself, such as the maximum steering angle, maximum acceleration, etc., control instructions corresponding to the obstacle avoidance path are generated, including the steering angle, acceleration of the UAV, and the time range for executing control operations, to ensure that the UAV can fly according to the planned path. After generating the optimal obstacle avoidance path, continuously monitor the remaining battery power, flight altitude, and real-time environmental data update of the UAV in real time. If the change in the environmental complexity index or the obstacle risk level exceeds the preset threshold, it indicates that the environment or the obstacle situation has changed significantly. At this time, recalculate the dynamic weight value of the sensor and update the path risk model parameters to adapt to the new situation. According to the updated comprehensive risk value, dynamically adjust the candidate path set and re-evaluate the path safety factor. If the risk difference between the current optimal path and the new candidate path exceeds the threshold, it means that the current path is no longer the best choice, trigger path replanning, and generate an updated obstacle avoidance path. During the path execution process, the UAV attitude is corrected in real time through a PID controller. The PID controller adjusts the control quantity through proportional, integral, and differential operations according to the deviation between the set value and the actual value to ensure the stable flight of the UAV.

[0071] In one embodiment, drones are used for urban logistics distribution. During flight, visual sensors and infrared sensors collect data simultaneously. The visual sensors capture high-resolution images, and the infrared sensors detect the heat source distribution in the environment. The timestamps of the visual and infrared sensors are corrected through Kalman filtering to eliminate time errors. The ST-CNN (Spatio-Temporal Convolutional Neural Network) is used to perform spatio-temporal alignment on the visual and infrared data. The ST-CNN learns motion patterns from historical data and predicts interpolation positions. Experimental results show that the alignment error is reduced from ±15 cm of traditional methods to ±5 cm. Edge, texture, and shape features are extracted from the visual data, and heat source distribution and intensity features are extracted from the infrared data. The weights of visual and infrared features are dynamically allocated through the self-attention mechanism. In the night scene of this embodiment, the weight of the infrared data is automatically increased to 0.8, and the weight of the visual data is decreased to 0.2. The fused multi-dimensional feature set significantly improves the accuracy of obstacle detection. Experimental data shows that the obstacle detection accuracy rate reaches 98.5% (85% for traditional methods).

[0072] The light intensity, temperature distribution, and wind speed values are obtained through multi-sensors, and the entropy method is used to calculate the environmental complexity index. During the day, the environmental complexity index is 0.3; at night, due to insufficient lighting, the environmental complexity index rises to 0.7. The weights of the visual and infrared sensors are predicted based on the LSTM model. At night, the weight of the infrared sensor is increased to 0.8, and the weight of the visual sensor is decreased to 0.2. The DQN (Deep Q-Network) is used to generate candidate paths, and then the path details are optimized through the A* algorithm. Experimental results show that the average planning time is 120 ms, and the path length is shortened by 12% compared with the traditional A* algorithm. In 100 tests, the drone successfully avoided obstacles 99 times, and the obstacle avoidance success rate was 99.2% (89% for traditional methods). In a simulated heavy rain environment (visibility < 10 m), the infrared sensor dominates the detection, and the obstacle avoidance success rate remains at 92%.

[0073] In another embodiment, drones are used for emergency rescue in a sandstorm environment. Sensor configuration: Visual sensor, 720p resolution, 20 Hz sampling frequency, switched to infrared auxiliary mode to reduce noise. Infrared sensor, 640×512 resolution, 25 Hz sampling frequency, sensitivity increased to 0.1 °C, used to detect heat sources (trapped people) in the sandstorm. Simulate a sandstorm environment with a wind speed of 15 m / s, visibility < 5 m, and a high density of sand particles.

[0074] In a sandstorm environment, visual sensors are limited by low visibility and rely mainly on infrared sensors to detect heat sources. The timestamps of the visual and infrared sensors are corrected through Kalman filtering to eliminate time errors. ST-CNN is used to align the visual and infrared data in time and space. Due to the violent movement in the sandstorm environment, the traditional method has a large alignment error (±2m), while ST-CNN learns the movement pattern through historical data and reduces the error to ±0.5m. The error compensation factor is calculated by the Kalman filter formula: in, is the predicted state covariance, H k is the observation matrix, R k is the measurement error covariance. By continuously updating K k , Kalman filtering can effectively compensate for the noise error of sensor data. ST-CNN alignment error formula: Among them, x 视觉 and 视觉 is the coordinate of the visual sensor, x 红外 and 红外 are the coordinates of the infrared sensor. ST-CNN learns motion patterns from historical data and predicts interpolated positions, reducing the alignment error to ±0.5m. Multimodal feature fusion extracts edge and texture features from visual data (although some information can still be captured in low visibility), and extracts heat source distribution and intensity features from infrared data. In a sandstorm environment, the weight of infrared data is automatically increased to 0.8, and the weight of visual data is reduced to 0.2. The fused multidimensional feature set significantly improves the accuracy of heat source detection. Experimental data show that the accuracy of heat source detection reaches 98.7%. Among them, the calculation formula for multimodal Transformer weight allocation is: Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension. Through the self-attention mechanism, Transformer dynamically assigns the weights of visual and infrared features.

[0075] Wind speed, temperature distribution and dust concentration are obtained through multiple sensors, and the entropy method is used to calculate the environmental complexity index. In a sandstorm environment, the environmental complexity index is 0.8. Its calculation formula is: Among them, p i It is the normalized value of various environmental factors (such as wind speed, temperature, and dust concentration). The environmental complexity index is calculated by the entropy method. The weights of visual and infrared sensors are predicted based on the LSTM model. In a sandstorm environment, the weight of the infrared sensor is increased to 0.8, and the weight of the visual sensor is reduced to 0.2. LSTM weight prediction formula: LSTM 输出 =σ(W f *[h t-1 ,x t ]+bf , where w f is the weight matrix, h t-1 is the hidden state at the previous moment, x t is the current input, and b f is the bias term. The LSTM model predicts the weights of visual and infrared sensors through time-series data. The DQN is used to generate candidate paths, and the path details are optimized by the A* algorithm. The experimental results show that the average planning time is 200 ms, and the path length is shortened by 15% compared with the traditional A* algorithm.

[0076] In 50 tests, the drone successfully avoided obstacles 45 times, and the obstacle avoidance success rate was 90% (65% for the traditional method). When the dust concentration suddenly increased, the environmental complexity index rose from 0.6 to 0.9, triggering path replanning. The DQN generated a path around the dust, and the PID controller adjusted the attitude of the drone in real time, with the pitch angle error < 1°.

[0077] In this embodiment, the computing rate of the drone is optimized. Multi-threaded parallel computing is used to allocate data acquisition, feature extraction, risk assessment, and path planning tasks to multiple threads for parallel execution. Through multi-threaded optimization, the overall computing rate is increased by 30%. In addition, deep learning models (such as ST-CNN, Transformer, LSTM) are quantized from 32-bit floating-point numbers to 8-bit integers, reducing the amount of computation and memory occupancy. After quantization, the model inference speed is doubled, and the memory occupancy is reduced by 50%. At the same time, redundant neurons and connections in the model are removed through pruning technology to reduce the model complexity. After pruning, the model computation amount is reduced by 40%, and the inference speed is increased by 1.5 times. Commonly used environmental data and sensor data are cached in high-speed memory to reduce data reading time. Through data caching, the data reading time is reduced from 20 ms to 5 ms. During the flight of the drone, multiple candidate paths are pre-computed and the results are cached. When the environment changes, the pre-computed paths are directly called from the cache to reduce the real-time planning time. The experimental data shows that path pre-computation reduces the planning time from 200 ms to 100 ms.

[0078] In summary, flight data is collected through the visual sensor and infrared sensor of the UAV to obtain timestamp and spatial coordinate information, and the sampling frequency difference between the two types of sensors is calculated; based on the sampling frequency difference, an adaptive interpolation algorithm is designed to align the original data of the two types of sensors, generating a fused dataset with time synchronization and spatial alignment; image features and heat source features in the fused dataset are extracted and fused to generate a multi-dimensional feature set; based on the multi-dimensional feature set, it is determined whether there are obstacles in the original flight path, and if so, obstacle information is extracted; real-time environmental data is analyzed, the environmental complexity index is calculated, and the obstacle risk level is generated in combination with the obstacle information; according to the dynamic risk level and the environmental complexity index, the dynamic weight values of the visual sensor and the infrared sensor are predicted, and the sampling frequency and working mode of the sensors are adjusted accordingly; based on the position, motion state, and dynamic weight value of the obstacle, the comprehensive risk value of each path is calculated through a path risk model, and the motion trend of the obstacle is predicted; according to the prediction result of the path risk model and the motion trend of the obstacle, an optimal obstacle avoidance path is generated, and the corresponding flight instruction is generated. To achieve efficient processing of sensor data, accurate risk assessment, and real-time obstacle avoidance path planning, and to achieve the purpose of stable and safe flight of the UAV in a complex environment.

[0079] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database, or other media provided by the present invention and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM, etc.

[0080] It should be noted that in this text, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent in such a process, device, article or method. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, device, article or method comprising that element.

[0081] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. An AI real-time intelligent guidance and adaptive obstacle avoidance method for drones with multi-source perception fusion, characterized in that: The following steps are involved: The flight data is collected through the UAV's visual sensor and infrared sensor, the timestamp and spatial coordinate information are obtained, and the sampling frequency difference between the two types of sensors is calculated; Based on the sampling frequency difference, an adaptive interpolation algorithm is designed to align the raw data of the two types of sensors to generate a fused data set that is time synchronized and spatially aligned; Extracting image features and heat source features from the fused data set, and fusing them to generate a multidimensional feature set; Determine whether there is an obstacle in the original flight path based on the multi-dimensional feature set, and if so, extract obstacle information, including position and motion state; Analyze real-time environmental data, calculate the environmental complexity index, and generate obstacle risk levels based on obstacle information; According to the dynamic risk level and the environmental complexity index, the dynamic weight values ​​of the visual sensor and the infrared sensor are predicted, and the sampling frequency and the working mode of the sensor are adjusted accordingly; Based on the location, movement status and dynamic weight value of obstacles, the comprehensive risk value of each path is calculated through the path risk model, and the movement trend of obstacles is predicted; According to the prediction results of the path risk model and the movement trend of obstacles, the optimal obstacle avoidance path is generated, including path length, path curvature and path safety, and the corresponding flight instructions are generated.

2. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of unmanned aerial vehicles according to claim 1 is characterized in that: The steps of collecting flight data through the visual sensor and infrared sensor of the drone, obtaining timestamp and spatial coordinate information, and calculating the sampling frequency difference between the two types of sensors include: Obtain the raw data of the visual sensor and infrared sensor respectively, and extract the corresponding timestamp and spatial coordinate information; Perform Kalman filtering correction on the spatial coordinates to eliminate noise errors; The sampling frequency difference between the visual sensor and the infrared sensor is calculated based on the timestamp.

3. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of unmanned aerial vehicles according to claim 1 is characterized in that: The step of designing an adaptive interpolation algorithm based on the sampling frequency difference to align the raw data of the two types of sensors and generate a fused data set that is time synchronized and space aligned includes: If the sampling frequency difference value exceeds a preset threshold, downsampling the data collected by the high-frequency sensor; Based on the collected raw data, the time series of the two types of sensors are aligned through a convolutional neural network, and the interpolation step size is adjusted according to the rate of change of the spatial coordinates; The image data of the two types of sensors are spatially aligned through feature matching; The spatiotemporally aligned data are input into the mapping model to generate a fused dataset containing the association relationship between visual and infrared data.

4. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of unmanned aerial vehicles according to claim 1 is characterized in that: The step of extracting image features and heat source features from the fused data set and fusing them to generate a multidimensional feature set includes: Extracting image features of edge, texture and shape features from the visual data of the fused data set, and extracting heat source features of heat source distribution and intensity features from the infrared data; The attention mechanism is used to fuse the extracted image features and heat source features to generate a multidimensional feature set containing joint space-heat source information.

5. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of unmanned aerial vehicles according to claim 1 is characterized in that: The step of judging whether there is an obstacle in the original flight path based on the multidimensional feature set, and extracting obstacle information if there is an obstacle, comprises: Input the multi-dimensional feature set into the pre-trained convolutional neural network to determine the existence of obstacles; If there are obstacles, the position information of all obstacles, including the distance and azimuth of each obstacle relative to the drone, is calculated through coordinate transformation. The trajectories of all obstacles are tracked based on the continuous frame feature set, and the motion information of each obstacle is calculated, including the speed and direction of each obstacle.

6. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for unmanned aerial vehicle multi-source perception fusion according to claim 1 is characterized in that: The steps of analyzing real-time environmental data, calculating the environmental complexity index, and generating the obstacle risk level in combination with the obstacle information include: Use multiple sensors to obtain environmental data within a limited range, including light intensity, temperature distribution, and wind speed values, normalize the environmental data, and calculate the environmental complexity index using the entropy method; Obstacles are classified according to the obstacle speed threshold to distinguish between static obstacles and dynamic obstacles; According to the environmental complexity index and obstacle classification results, the risk levels of all obstacles are calculated through the pre-established risk level model.

7. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for unmanned aerial vehicle multi-source perception fusion according to claim 1 is characterized in that: The step of predicting the dynamic weight values ​​of the visual sensor and the infrared sensor according to the dynamic risk level and the environmental complexity index, and adjusting the sampling frequency and the working mode of the sensor accordingly, comprises: Based on the obstacle risk level and environment complexity index, the LSTM model is used to predict the weights of the visual sensor and the infrared sensor; According to the predicted weight ratio of the visual sensor and the infrared sensor, the sampling frequency of the sensor is adjusted respectively; When the environmental index exceeds the set threshold, the visual sensor is switched to high-resolution mode; when the risk level exceeds the set threshold, the sensitivity of the infrared sensor is increased.

8. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of unmanned aerial vehicles according to claim 1 is characterized in that: The step of calculating the comprehensive risk value of each path through the path risk model based on the position, movement state and dynamic weight value of the obstacle and predicting the movement trend of the obstacle includes: The static risk is calculated based on the distance between the obstacle and the drone and the density of the obstacle distribution. The dynamic risk is calculated based on the probability of collision between the moving obstacle and the drone. Based on the weights of the visual sensor and the infrared sensor, the static risk and dynamic risk corresponding to the visual sensor and the infrared sensor are weighted and integrated to obtain the comprehensive risk value of each possible path; Kalman filtering is used to predict the future position of each obstacle. If the predicted future position of the obstacle overlaps with the original flight path, the dynamic risk weight is increased, and the weight parameters of each factor in the path risk assessment model are optimized and adjusted through reinforcement learning.

9. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for multi-source perception fusion of unmanned aerial vehicles according to claim 1 is characterized in that: The step of generating an optimal obstacle avoidance path according to the prediction result of the path risk model and the obstacle movement trend, and generating corresponding flight instructions includes: generating a series of candidate paths according to the comprehensive risk value; Based on the risk level, path length, and curvature of each candidate path, the A* algorithm is used to select the path with the best overall performance to obtain the obstacle avoidance path. According to the UAV's own power performance and flight control constraints, control instructions corresponding to the obstacle avoidance path are generated, including the UAV's steering angle, acceleration, and the time range for executing control operations.

10. The AI ​​real-time intelligent guidance and adaptive obstacle avoidance method for unmanned aerial vehicle multi-source perception fusion according to claim 9 is characterized in that: After the step of generating the optimal obstacle avoidance path, the method further comprises: Real-time monitoring of the drone’s flight status, including remaining battery power, flight altitude, and real-time environmental data updates; If the environmental complexity index or obstacle risk level changes beyond the preset threshold, the sensor dynamic weight value is recalculated and the path risk model parameters are updated; According to the updated comprehensive risk value, the candidate path set is dynamically adjusted and the path safety factor is re-evaluated; If the risk difference between the current optimal path and the new candidate path exceeds the threshold, path replanning is triggered to generate an updated obstacle avoidance path, and during the path execution, the drone attitude is corrected in real time through the PID controller.

Images