Unmanned aerial vehicle real-time path planning and obstacle avoidance method and device, medium and equipment

Abstract

The invention discloses an unmanned aerial vehicle real-time path planning and obstacle avoidance method and device, a medium and equipment. The method comprises the following steps: acquiring environment perception information, terrain and weather information and self state information in a flight process of an unmanned aerial vehicle; preprocessing the environmental perception information, the terrain and weather information and the self state information; constructing an unmanned aerial vehicle path planning and obstacle avoidance model, and training the model; and inputting the preprocessed environmental perception information, terrain and weather information and self-state information into a trained unmanned aerial vehicle path planning and obstacle avoidance model so as to plan an optimal flight path of the unmanned aerial vehicle and give an optimal obstacle avoidance mode of the unmanned aerial vehicle. The autonomous navigation capability of the unmanned aerial vehicle in a complex and dynamic environment can be remarkably improved.

Description

Technical Field

[0001] The present application belongs to the field of UAV technology, and specifically relates to a method, device, medium and equipment for real-time path planning and obstacle avoidance of UAV. Background Art

[0002] With the development of drone technology, its application scope is becoming more and more extensive, including but not limited to logistics distribution, agricultural monitoring, film and television shooting and other fields. However, how to ensure the safe flight of drones in complex environments, avoid collisions and complete tasks efficiently is a problem that needs to be solved urgently. Traditional path planning algorithms such as A* and Dijkstra often fail to meet the requirements of real-time and flexibility when facing dynamic environments or obstacles. Therefore, developing a real-time path planning and obstacle avoidance method based on machine learning has important research value and practical significance. Summary of the invention

[0003] The main purpose of this application is to provide a method, device, medium and equipment for real-time path planning and obstacle avoidance of a UAV. This application aims to improve the autonomous navigation capability of the UAV when facing dynamic environments or obstacles.

[0004] To achieve the above objectives, this application provides the following technical solutions: A method for real-time path planning and obstacle avoidance of an unmanned aerial vehicle (UAV), the method comprising: obtaining environmental perception information, terrain and meteorological information, and self-state information of the UAV during flight; preprocessing the environmental perception information, terrain and meteorological information, and self-state information; constructing a UAV path planning and obstacle avoidance model, and training the model; inputting the preprocessed environmental perception information, terrain and meteorological information, and self-state information into the trained UAV path planning and obstacle avoidance model, so as to plan the optimal flight path of the UAV and provide the optimal obstacle avoidance mode of the UAV.

[0005] Optionally, the environmental perception information, terrain and meteorological information and self-status information are preprocessed, including: cleaning and denoising the environmental perception information, terrain and meteorological information and self-status information; performing data synchronization and timestamp alignment on the cleaned and denoised environmental perception information, terrain and meteorological information and self-status information; and fusing the environmental perception information, terrain and meteorological information and self-status information after data synchronization and timestamp alignment.

[0006] Optionally, the UAV path planning and obstacle avoidance model includes: a multimodal spatiotemporal data fusion module, a path generation layer and an obstacle avoidance decision layer, wherein the multimodal spatiotemporal data fusion module is used to integrate the pre-processed environmental perception information, terrain and meteorological information, and its own status information and perform multi-scale feature extraction; the path generation layer is used to generate an optimal flight path based on multi-scale features; the obstacle avoidance decision layer is used to process obstacles encountered by the UAV in real time during its flight along the optimal flight path, and give reasonable obstacle avoidance decisions.

[0007] Optionally, the multimodal spatiotemporal data fusion module includes: an input layer and a spatiotemporal encoder, wherein the input layer is used to receive the preprocessed environmental perception information, terrain and meteorological information and its own status information; the spatiotemporal encoder is used to extract multi-scale features from the preprocessed environmental perception information, terrain and meteorological information and its own status information, and plan the UAV flight path based on the multi-scale features.

[0008] Optionally, the spatiotemporal encoder includes: an adaptive multi-scale feature extraction module, a lightweight spatiotemporal attention mechanism, a hierarchical dynamic graph neural network, and a feedback enhancement module connected in sequence.

[0009] Optionally, the UAV path planning and obstacle avoidance model is trained by the following steps: collecting different sensor data and preprocessing to obtain a model training data set; expanding the data set and dividing the expanded data set into a training set and a validation set; setting training parameters and training the model with the training set, and during the training process, when the cross entropy loss function converges, the model training ends; The trained model is verified through the validation set. If the mean absolute error, mean square error, and root mean square error, which are the model performance evaluation indicators, are all less than the threshold, the model verification is passed; otherwise, the training parameters are adjusted or the training set samples are expanded to retrain the model until the model verification is passed.

[0010] The present application also provides a real-time path planning and obstacle avoidance device for a UAV, the device comprising: an acquisition module, used to acquire environmental perception information, terrain and meteorological information, and its own status information during the flight of the UAV based on different sensors; a preprocessing module, used to preprocess the environmental perception information, terrain and meteorological information, and its own status information; a model construction and training module, used to construct a UAV path planning and obstacle avoidance model, and train the model; a path planning module, used to input the preprocessed environmental perception information, terrain and meteorological information, and its own status information into the trained UAV path planning and obstacle avoidance model, so as to plan the optimal flight path of the UAV and provide the optimal obstacle avoidance mode of the UAV.

[0011] Optionally, the preprocessing module includes: a cleaning and denoising submodule, used to clean and denoise the environmental perception information, terrain and meteorological information, and its own status information; a synchronization and alignment submodule, used to perform data synchronization and timestamp alignment on the cleaned and denoised environmental perception information, terrain and meteorological information, and its own status information; and a fusion submodule, used to fuse the environmental perception information, terrain and meteorological information, and its own status information after data synchronization and timestamp alignment.

[0012] The present application also provides a storage medium, which includes instructions, and when the instructions are executed on a computer, the computer executes the method as described in any of the preceding items.

[0013] The present application also provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the program.

[0014] Compared with the existing technology, this application can bring the following beneficial effects: This application can significantly improve the autonomous navigation capability of drones in complex and dynamic environments through the constructed drone path planning and obstacle avoidance model. This application can not only maximize the comprehensive utilization efficiency of multiple sensor data, but also ensure that drones can complete path planning and obstacle avoidance tasks more safely and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of a method for real-time path planning and obstacle avoidance of a drone provided by an embodiment of the present application; Figure 2 It is a structural diagram of a drone real-time path planning and obstacle avoidance model provided by another embodiment of the present application; Figure 3 is a structural schematic diagram of a real-time path planning and obstacle avoidance device for a drone provided by another embodiment of the present application; Figure 4 is a schematic diagram of the structure of a storage medium provided by another embodiment of the present application; Figure 5 It is a schematic diagram of the structure of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0018] In this application, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0019] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing in the full text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or a scheme that satisfies both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0020] Figure 1 FIG. 1 is a flow chart of a method for real-time path planning and obstacle avoidance of a drone according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps: S100: Acquire environmental perception information, terrain and weather information, and self-state information of the drone during flight based on different sensors, wherein the environmental perception information includes static obstacle information (such as buildings, electric poles, trees, etc.) and dynamic obstacle information (such as birds, floating objects in the air, and other moving drones); the terrain and weather information includes terrain information (such as changes in the height of the ground) and weather information (such as wind speed, wind direction, temperature, humidity, and other factors); and the self-state information includes the attitude angle, position coordinates, and velocity vector of the drone.

[0021] S200: Preprocessing the environmental perception information, terrain and meteorological information, and self-state information; S300: Build a UAV path planning and obstacle avoidance model and train the model; S400: Input the pre-processed environmental perception information, terrain and weather information, and self-state information into the trained UAV path planning and obstacle avoidance model to plan the optimal flight path of the UAV and provide the optimal obstacle avoidance mode of the UAV.

[0022] By building a UAV path planning and obstacle avoidance model, this technology can not only enhance the comprehensive utilization efficiency of various sensor data, but also calmly deal with unknown obstacles and environmental changes, thereby ensuring that the UAV can complete the path planning and obstacle avoidance tasks safely and efficiently.

[0023] In another exemplary embodiment, in step S200, preprocessing the environmental perception information, terrain and meteorological information, and self-state information includes the following steps: S201: Cleaning and denoising the environmental perception information, terrain and meteorological information, and self-state information; In this step, a digital filter (such as a low-pass filter) is used to remove high-frequency noise in each piece of information, such as position and velocity data from an IMU or GPS. In addition, statistical methods (such as the 3σ principle) are required to identify and remove data points that are significantly deviated from the normal range. Furthermore, interpolation methods (linear interpolation, spline interpolation, etc.) can be used to fill in the missing data points in each piece of information.

[0024] S202: performing data synchronization and timestamp alignment on the cleaned and denoised environmental perception information, terrain and meteorological information, and self-state information; In this step, the timestamps of each sensor need to be corrected so that they are based on the same time reference to avoid errors caused by clock inconsistencies. In addition, data streams with different sampling frequencies need to be matched to ensure that key information is correctly updated at the same time.

[0025] S203: Fusing the environmental perception information, terrain and meteorological information, and self-state information after data synchronization and timestamp alignment.

[0026] In this step, the information obtained from different sensors can be fused through Kalman filtering, particle filtering and other technologies to improve the accuracy of UAV position estimation and the reliability of the environmental model.

[0027] In another exemplary embodiment, in step S300, Figure 2As shown, the UAV path planning and obstacle avoidance model includes: a multimodal spatiotemporal data fusion module, a path generation layer and an obstacle avoidance decision layer, wherein the multimodal spatiotemporal data fusion module is used to integrate the pre-processed environmental perception information, terrain and meteorological information and its own state information and perform multi-scale feature extraction; the path generation layer is used to generate an optimal flight path based on the extracted multi-scale features; the obstacle avoidance decision layer is used to process obstacles encountered by the UAV in the process of flying along the optimal flight path in real time, and give reasonable obstacle avoidance decisions.

[0028] In this embodiment, the multimodal spatiotemporal data fusion module includes an input layer and a spatiotemporal encoder, wherein the input layer is used to receive the pre-processed environmental perception information, terrain and meteorological information, and self-state information. The spatiotemporal encoder includes an adaptive multi-scale feature extraction module, a lightweight spatiotemporal attention mechanism, a hierarchical dynamic graph neural network, and a feedback enhancement module connected in sequence, wherein the adaptive multi-scale feature extraction module introduces a gating mechanism, which can dynamically select the scale of the convolution kernel according to the complexity of the real-time environment. For example, when the drone is in a dense obstacle area, small-scale convolutions (such as 3×3) are preferentially activated to capture details; while in open areas, large-scale convolutions (such as 7×7) are switched to improve computational efficiency. In addition, the adaptive multi-scale feature extraction module also includes a feature pyramid network to cross-layer fuse the multi-scale features extracted based on the gating mechanism to enhance the semantic understanding of complex terrain (such as hills and canyons) while reducing redundant features.

[0029] The lightweight spatiotemporal attention mechanism adopts a hierarchical attention design, including a local attention layer and a global attention layer. The local attention layer is based on a sliding window mechanism and only focuses on obstacles within the current field of view of the drone (such as a 50-meter radius) to reduce the computational complexity of the model. The global attention layer uses a sparse Transformer to perform long-range dependency modeling only on key dynamic obstacles (such as fast-moving birds), balancing accuracy and real-time performance. The lightweight spatiotemporal attention mechanism also includes an embedded lightweight LSTM module to record historical obstacle trajectories, predict movement trends within the next 3 seconds, and provide forward-looking decision support for dynamic obstacle avoidance.

[0030] The hierarchical dynamic graph neural network adopts a two-layer graph structure, including a global layer and a local layer, wherein the global layer uses terrain elevation and meteorological data as nodes to generate a global optimal path skeleton. The local layer uses real-time obstacles (static + dynamic) as nodes, dynamically updates the connection weights, and realizes refined obstacle avoidance. In addition, the hierarchical dynamic graph neural network also introduces a graph pruning strategy, which can prune irrelevant nodes (such as obstacles beyond the sensor range) according to the real-time position and speed of the drone, thereby improving the model calculation efficiency.

[0031] The feedback enhancement module feeds back the output of the hierarchical dynamic graph neural network (such as obstacle avoidance decision confidence or importance score) to the multi-scale feature extraction module through a feedback mechanism to guide the feature extraction focus of the next frame of data. Specifically, the feedback enhancement module includes an input layer, a filter and an integration layer connected in sequence, wherein the input layer is used to receive output information from the hierarchical dynamic graph neural network, including but not limited to obstacle position, type, movement direction prediction and obstacle avoidance decision confidence. The filter filters the received information according to predefined criteria (such as confidence threshold) to determine which important information needs to be fed back to the multi-scale feature extraction module, which can reduce unnecessary computational burden of the model and improve the operation efficiency of the model. The integration layer is used to integrate the output information from the hierarchical dynamic graph neural network with the original input data (i.e., environmental perception information, terrain and meteorological information and drone status information), and feed back the integrated information to the multi-scale feature extraction module to help the multi-scale feature extraction module dynamically adjust the feature extraction focus. For example, when encountering high-confidence obstacles or complex terrain, the feature extraction weights of these areas can be increased to analyze the information in these key areas more carefully, which helps to improve the speed and accuracy of the drone's response to environmental changes.

[0032] The path generation layer uses the following algorithm to generate the UAV flight path. The algorithm adjusts the path cost by introducing terrain elevation and static obstacle distribution to generate a UAV flight path that better meets actual needs. The specific algorithm is shown as follows:

[0033] in, Indicates that through the node The total cost is used to determine which path is the best; Represents the path length, that is, the actual distance from the current node to the next node; Indicates terrain risk; represents the energy consumption of the drone; represents the wind impact index; Indicates load sensitivity; Indicates time sensitivity; , , , , and They represent weight parameters respectively and can be adjusted according to specific task requirements.

[0034] Below, this application specifically explains how to generate a drone flight path based on the above algorithm: First, set the starting point as the current node and mark all other nodes as unexplored. Create a priority queue to store the nodes to be explored and their corresponding estimated total costs.

[0035] Secondly, the node with the lowest estimated total cost is taken from the priority queue as the current node. For each neighbor node of the current node, the information such as the impassable area, static obstacle distribution, and dynamic obstacle prediction is comprehensively considered. The actual cost of reaching each neighbor node through the current node is calculated, and the total cost of the neighbor node is updated using the more complex cost function mentioned above. According to real-time data (such as wind speed and wind direction changes), the relevant parameters in the cost function (such as and ) to reflect the latest environmental conditions.

[0036] Finally, repeat step 2 until the target node is found or the open list is empty. Once the target node is found, the complete optimal path can be obtained by tracing the predecessor nodes of each node back to the starting point.

[0037] The above algorithms can provide more accurate and flexible path planning solutions by integrating multiple environmental factors and dynamic adjustment mechanisms.

[0038] The obstacle avoidance decision layer adopts a spatiotemporal attention reinforcement learning network, which specifically includes a state space layer, an action space layer, a reward function layer and an attention mechanism.

[0039] The state space layer defines the set of states that the drone is in at any time, which usually includes a series of variables that can describe the current situation of the drone. Specifically, in the local obstacle avoidance scenario of the drone, the state space can contain the following information: Location information: The three-dimensional coordinates of the drone's current location ( x, y, z ).

[0040] Attitude information: including pitch angle ( pitch )、Roll angle( roll ) and yaw angle ( yaw ), these angles determine the direction of the drone.

[0041] Velocity vector: forward velocity ( vx ), lateral speed (vy ) and the vertical velocity ( v ).

[0042] Sensor data: Information from different sensors, such as LiDAR ( LiDAR ) or visual sensors provide information such as the distance and direction of surrounding obstacles.

[0043] Environmental characteristics: factors that affect flight stability, such as wind speed (windSpeed), wind direction (windDirection), etc.

[0044] Task-related information: factors such as remaining battery level, load status, etc. that may affect the decision.

[0045] The state space layer can be expressed as:

[0046] in, represents the state space, and They represent the distance to the nearest obstacle and its direction to the drone, respectively.

[0047] The action space layer defines all possible actions that the drone can perform, aiming to help the drone avoid obstacles and continue moving towards the target. Depending on the capabilities of the drone and the application scenario, the action space may include the following types of actions: 1. Translational motion: moving forward ( ), move backward ( ), move left and right ( ), rising or falling ( ); 2. Rotational motion: rotation around the X axis (change pitch angle), rotation around the Y axis (change roll angle), Rotate around the Z axis (change the yaw angle) or keep the current position ( ).

[0048] 3. Compound movements: combining translation and rotation to achieve more complex maneuvers, such as rising while turning.

[0049] The action space layer can be expressed as:

[0050] in, represents the action space. When the drone needs to change direction quickly to avoid collision, it can do so by turning left ( ) and right turn ( ), select the appropriate steering angle (d according to the location of the obstacle yaw ).

[0051] The reward function layer defines the feedback that the drone receives after taking an action, thereby guiding the drone to learn how to make the best decision to achieve its goal. The feedback specifically includes: Positive rewards are given when the drone successfully avoids obstacles; the reward value increases as the drone gradually approaches the final target position; negative rewards are given if the drone collides with an obstacle; and the use of action sequences with lower energy consumption is encouraged to extend the flight time.

[0052] The reward function layer is expressed as follows:

[0053] in, Rewards for approaching the goal, represents the collision penalty, represents the energy consumption optimization reward, Represents the posture stability reward.

[0054] By setting a reward function, this application can not only effectively guide the drone to avoid obstacles, but also ensure that it completes the scheduled task efficiently and economically. In this way, the drone can autonomously learn and optimize its flight path in a complex real-world environment.

[0055] The attention mechanism includes a spatiotemporal feature extraction module, an adaptive fusion layer, and a reinforcement learning decision layer. The spatiotemporal feature extraction module includes a spatial attention branch and a temporal attention branch. The spatial attention branch uses a convolutional neural network (CNN) to extract spatial features from sensor data, such as the distance and direction of obstacles, and generates a spatial attention map to highlight areas that require special attention. The temporal attention branch uses a recurrent neural network (RNN) to process historical data, predict the future movement trajectory of obstacles, and form temporal attention weights to emphasize obstacles that may have a significant impact on the flight path in the future.

[0056] The adaptive fusion layer is used to combine the outputs of the spatial attention branch and the temporal attention branch to form a comprehensive attention vector (this process can be completed, for example, by weighted summation, in order to ensure that the final attention distribution can reflect both the current spatial layout and predict future trends). In addition, the adaptive fusion layer also introduces an adaptive adjustment factor to allow the attention mechanism to adjust the importance ratio of each component according to real-time environmental feedback, for example, increasing the proportion of temporal attention in a rapidly changing environment, while focusing more on spatial feature analysis in a relatively stable environment.

[0057] For example, assume that the outputs of the spatial attention branch and the temporal attention branch are and , this application uses a weighted summation method and introduces an adaptive adjustment factor To adjust the weight distribution, as follows:

[0058] in, Represents the comprehensive attention vector, adaptive adjustment factor It is a regulation factor dynamically adjusted according to the Environmental Dynamics Index (EDI), expressed as:

[0059] when When it is close to 0, the environment is very unstable. A value close to 1 means more reliance on the temporal attention branch to predict future obstacle movement trends.

[0060] when When it is close to 1, the environment is very stable. Close to 0, at this time it mainly relies on the real-time information provided by the spatial attention branch to make decisions.

[0061] In summary, this application introduces an adaptive adjustment factor , which enables the UAV to flexibly adjust its focus when facing different environments, thereby improving the quality and efficiency of obstacle avoidance decisions.

[0062] The reinforcement learning decision layer uses the comprehensive attention vector generated by the adaptive fusion layer as input to help the model better understand which factors are the most critical, so that the model can develop the most effective obstacle avoidance strategy.

[0063] Below, this application introduces the mechanism of action of the reinforcement learning decision layer as follows: First, the state space With the comprehensive attention vector Combine to form an extended state representation ,Right now ; Next, define the function , used to estimate the extended state Take action The expected cumulative rewards that can be obtained after Represents the weight parameter.

[0064] Finally, the objective function is defined using the TD error to adjust the weight parameters To update, the objective function It is expressed as follows:

[0065] in,

[0066] Indicates immediate reward; Indicates the status ,action , Instant Rewards and the next state The distribution of is averaged; Represents the target Q value, which is composed of the immediate reward and estimates of future rewards composition; Represents the discount factor, between 0 and 1; For a given next state , choose the action that maximizes the Q value ; Represents the parameters of the target network, which are updated to the current network parameters after a certain number of steps a copy of; Indicates the Q value estimate of the current network, indicating that it is in the extended state The expected cumulative reward after taking action A; Represents the square of the TD error, which is used to measure the gap between the target Q value and the Q value predicted by the current network.

[0067] Through the above introduction, the reinforcement learning decision layer not only depends on the current state and action, but also needs to be combined with future reward predictions and environmental feedback for learning and optimization, so that the model can help drones make more intelligent and effective decisions in complex environments.

[0068] The attention mechanism, combined with the reward function shown above, can further optimize the attention allocation strategy, ensuring that the drone can not only avoid obstacles in front of it, but also predict potential risks and take action in advance.

[0069] This application introduces an attention mechanism in the obstacle avoidance decision layer, so that the model can focus on the most important parts when processing data. For example, in a complex environment, not all detected objects need to be given equal attention. By introducing the attention mechanism, the most urgent obstacles to be avoided can be highlighted according to the current flight status. In addition, by learning from historical data, the attention mechanism can help the model predict the future movement direction of obstacles, so as to make avoidance decisions in advance. Furthermore, based on changes in the surrounding environment, the attention mechanism allows the drone to adjust its obstacle avoidance strategy in real time to ensure that the optimal path is always selected.

[0070] In another exemplary embodiment, in step S300, the drone path planning and obstacle avoidance model is trained by the following steps: S301: Collect data from different sensors and perform preprocessing to obtain a model training data set; S302: Expanding the data set, for example, by adding noise, changing lighting conditions, simulating different weather conditions, etc., and dividing the expanded data set into a training set and a validation set, and the division ratio may be, for example, 7:3; S303: Setting training parameters. For example, booster selects the tree-based model gbtree by default, learning_rate is set to 0.01, max_depth is set to 3, and the model is trained using the training set. During the training process, when the cross entropy loss function converges, the model training ends. S304: Verify the trained model through the validation set. If the mean absolute error (MAE), mean square error (MSE) and root mean square error (RMSE) as the model performance evaluation indicators are all less than the threshold (the threshold of MAE is set to 0.05, and the threshold of MSE is set to 0.001), the model verification is passed; otherwise, adjust the training parameters (for example, the learning_rate can be adjusted to 0.005, and the max_depth can be adjusted to 5) or expand the training set samples (for example, adjust the division ratio to 8:2) to re-train the model until the model verification is passed.

[0071] In this embodiment, the present application proposes an original loss function combining dynamic weight adjustment and uncertainty estimation, which is expressed as follows:

[0072] in, represents the total number of samples; represents the true label; represents the corresponding value in the probability distribution predicted by the model; Represents a dynamic weight function, based on the current spatial characteristics and time characteristics Adaptively adjust the weight of each sample; represents an uncertainty estimate that measures the confidence in the model's predictions given the input; Represents the balance coefficient, which is used to adjust the impact of uncertainty estimation on the total loss.

[0073] In addition, in order to further improve the robustness of the model, this application further introduces an adversarial sample generator in the training process to simulate the worst-case environmental changes. The adversarial sample generator is expressed as follows:

[0074] in, represents the perturbation generated by the adversarial sample generator, and the goal is to find the perturbation that maximizes the loss function.

[0075] The final cross entropy loss function is then expressed as the weighted sum of the original loss and the adversarial loss:

[0076] in, represents the weight of the adversarial loss.

[0077] The cross-entropy loss function shown in this application can significantly improve the adaptability and robustness of the UAV path planning and obstacle avoidance model by introducing mechanisms such as dynamic weight adjustment, uncertainty estimation and adversarial learning. It can not only process complex spatiotemporal data more accurately, but also effectively deal with uncertainties and potential threats in the environment, thereby ensuring that the UAV can perform tasks safely and efficiently under various conditions.

[0078] Below, the present application compares and explains the method described in the present application with the traditional method by defining specific scenarios.

[0079] The scenario is defined as: Environmental perception information: The drone is flying over a forest. Its sensors detect a 30-meter-high hill 50 meters ahead and a flock of birds moving northeast 200 meters to its right.

[0080] Terrain information: The ground is undulating, with many hills and valleys, the highest of which is 100 meters above sea level.

[0081] Weather information: The current wind speed is 5m / s from the southwest, the temperature is moderate, and the humidity is high but will not affect flight.

[0082] UAV status information: The current position coordinates of the UAV are (0, 0, 50) m (relative to the take-off point), the target position is (500, 500, 50) m, the current speed is 10 m / s, and the optimal cruising altitude without crosswind is 50 m.

[0083] According to the above information, this application processes environmental perception information, terrain and meteorological information, and its own status information through a multimodal spatiotemporal data fusion module, and extracts features to select a path that is slightly deviated from the straight line for the drone, that is, first rise to a height of 70 meters to cross the hill in front, and then adjust the course according to the predicted movement trend of the bird to ensure a safe distance. As for the A* algorithm, if the three-dimensional space is simplified into a two-dimensional plane and the position of the static obstacles is known, the algorithm can find a shortest path from the starting point to the end point. However, for dynamic obstacles (such as birds), the algorithm needs to frequently update the node weights, which will cause a sharp increase in the amount of calculation and make it difficult to achieve real-time response. The Dijkstra algorithm does not use heuristic functions to guide the search process, which means that it is less efficient in large and complex environments. For situations with a large number of obstacles and dynamic changes, Dijkstra cannot quickly find a feasible solution.

[0084] In addition, this application uses a spatiotemporal attention reinforcement learning network to analyze the surrounding environment in real time and give the best obstacle avoidance strategy based on the current state. For example, when a bird is detected approaching, the drone's heading or altitude is changed in advance to avoid a collision. For emergencies (such as a bird suddenly approaching), the A* algorithm needs to rely on an external system to detect obstacles and then manually replan the path. The reaction time is long and it is impossible to avoid dynamic obstacles in time. Similarly, the Dijkstra algorithm lacks the ability to respond immediately when facing dynamic obstacles, and usually requires additional mechanisms to detect and handle obstacles.

[0085] In summary, since this method takes dynamic obstacle avoidance into account and can react in a short time, the overall flight time is short (e.g., 51 seconds). In contrast, the A* and Dijkstra algorithms may have a slightly longer flight time (e.g., 55 seconds) due to the delay caused by the need to recalculate the path. In addition, this method improves safety and reduces the risk of collision by predicting the behavior pattern of obstacles; traditional algorithms are difficult to achieve such fine adjustments without external intervention, and there is a higher risk of collision.

[0086] In another exemplary embodiment, the present application also provides a real-time path planning and obstacle avoidance device for a drone, such as Figure 3As shown, the device includes: an acquisition module 100, which is used to acquire environmental perception information, terrain and meteorological information, and self-state information of the UAV during flight based on different sensors; a preprocessing module 200, which is used to preprocess the environmental perception information, terrain and meteorological information, and self-state information; a model building and training module 300, which is used to build a UAV path planning and obstacle avoidance model and train the model; a path planning module 400, which is used to input the preprocessed environmental perception information, terrain and meteorological information, and self-state information into the trained UAV path planning and obstacle avoidance model, so as to plan the optimal flight path of the UAV and provide the optimal obstacle avoidance mode of the UAV.

[0087] Optionally, the preprocessing module 200 includes: a cleaning and denoising submodule, which is used to clean and denoise the environmental perception information, terrain and meteorological information, and its own status information; a synchronization and alignment submodule, which is used to perform data synchronization and timestamp alignment on the cleaned and denoised environmental perception information, terrain and meteorological information, and its own status information; and a fusion submodule, which is used to fuse the environmental perception information, terrain and meteorological information, and its own status information after data synchronization and timestamp alignment.

[0088] Based on the above embodiments, Figure 4 , for an explanation of the computer-readable storage medium of the exemplary embodiment of the present application, please refer to Figure 4 , the computer-readable storage medium shown is a CD 40, on which a computer program (i.e., a program product) is stored. When the computer program is executed by the processor, each step recorded in the above method implementation will be implemented, for example, obtaining environmental perception information, terrain and meteorological information, and self-state information of the UAV during flight based on different sensors; preprocessing the environmental perception information, terrain and meteorological information, and self-state information; constructing a UAV path planning and obstacle avoidance model, and training the model; inputting the preprocessed environmental perception information, terrain and meteorological information, and self-state information into the trained UAV path planning and obstacle avoidance model to plan the optimal flight path of the UAV and provide the optimal obstacle avoidance mode of the UAV. The specific implementation method of each step will not be repeated here.

[0089] It should be noted that the computer-readable storage medium includes but is not limited to phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0090] Based on the above embodiments, the present application also provides an electronic device, as shown below: Figure 5 An electronic device for downloading a file according to an exemplary embodiment of the present application is described.

[0091] Figure 5 A block diagram of an exemplary electronic device 50 suitable for implementing the embodiments of the present application is shown, and the electronic device 50 may be a computer system or a cloud server. Figure 5 The electronic device 50 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0092] like Figure 5 As shown, the electronic device 50 includes but is not limited to: one or more processors or processing units 501, a system memory 502, and a bus 503 connecting different system components (including the system memory 502 and the processing unit 501).

[0093] The electronic device 50 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 50, including volatile and non-volatile media, removable and non-removable media.

[0094] The system memory 502 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 5021 and / or cache memory 5022. The electronic device 50 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the ROM 5023 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 is not shown in the Figure 5 As shown in the figure, a disk drive for reading and writing a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media) can be provided. In these cases, each drive can be connected to the bus 503 through one or more data medium interfaces. The system memory 502 may include at least one program product, which has a set (such as at least one) of program modules, which are configured to perform the functions of each embodiment of the present application.

[0095] A program / utility 5025 having a set (at least one) of program modules 5024 may be stored, for example, in system memory 502, and such program modules 5024 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 5024 generally perform the functions and / or methods of the embodiments described herein.

[0096] The electronic device 50 may also communicate with one or more external devices 504 (e.g., a keyboard, a pointing device, a display, etc.). Such communication may be performed via an input / output (I / O) interface 505. Furthermore, the electronic device 50 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 506. Figure 5 As shown, the network adapter 506 communicates with other modules (such as the processing unit 501, etc.) of the electronic device 50 via the bus 503. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 50 .

[0097] The processing unit 501 executes various functional applications and data processing by running the program stored in the system memory 502, for example, obtaining environmental perception information, terrain and meteorological information and self-state information of the drone during flight based on different sensors; preprocessing the environmental perception information, terrain and meteorological information and self-state information; building a drone path planning and obstacle avoidance model, and training the model; inputting the preprocessed environmental perception information, terrain and meteorological information and self-state information into the trained drone path planning and obstacle avoidance model to plan the optimal flight path of the drone and give the optimal obstacle avoidance mode of the drone. The specific implementation of each step is not repeated here. It should be noted that although several units / modules or sub-units / sub-modules of the file concurrent downloading device are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above can be concretized in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be concretized.

[0098] In the description of the present application, it should be noted that the terms "first", "second" and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0099] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0100] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0103] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that can be executed by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a cloud server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0104] The above embodiments are only for illustrating the technical concept and features of the present application, and their purpose is to enable people familiar with the technology to understand the content of the present application and implement it accordingly, and they cannot be used to limit the protection scope of the present application. Any equivalent changes or modifications made according to the spirit of the present application should be included in the protection scope of the present application.

Claims

1. A real-time path planning and obstacle avoidance method for an unmanned aerial vehicle, characterized in that: The method comprises: Obtain environmental perception information, terrain and weather information, and the drone's own status information during flight; Preprocessing the environmental perception information, terrain and meteorological information, and self-state information; Build a UAV path planning and obstacle avoidance model and train the model; The pre-processed environmental perception information, terrain and weather information, and self-state information are input into the trained UAV path planning and obstacle avoidance model to plan the optimal flight path of the UAV and provide the optimal obstacle avoidance mode of the UAV.

2. A method for real-time path planning and obstacle avoidance of an unmanned aerial vehicle according to claim 1, characterized in that: Preprocessing the environmental perception information, terrain and meteorological information, and self-state information includes: Cleaning and denoising the environmental perception information, terrain and meteorological information, and self-state information; Performing data synchronization and timestamp alignment on the cleaned and denoised environmental perception information, terrain and meteorological information, and self-state information; The environmental perception information, terrain and meteorological information, and self-state information after data synchronization and timestamp alignment are integrated.

3. The method for real-time path planning and obstacle avoidance of an unmanned aerial vehicle according to claim 1, characterized in that: The UAV path planning and obstacle avoidance model includes: Multimodal spatiotemporal data fusion module, path generation layer and obstacle avoidance decision layer, among which, The multimodal spatiotemporal data fusion module is used to integrate the pre-processed environmental perception information, terrain and meteorological information, and self-state information and perform multi-scale feature extraction; The path generation layer is used to generate an optimal flight path based on multi-scale features; The obstacle avoidance decision layer is used to process obstacles encountered by the UAV in real time while flying along the optimal flight path, and to make reasonable obstacle avoidance decisions.

4. A method for real-time path planning and obstacle avoidance of a UAV according to claim 3, characterized in that: The multimodal spatiotemporal data fusion module includes: Input layer and spatiotemporal encoder, where The input layer is used to receive the pre-processed environmental perception information, terrain and meteorological information and self-state information; The spatiotemporal encoder is used to extract multi-scale features from the pre-processed environmental perception information, terrain and meteorological information, and self-state information, and plan the flight path of the UAV based on the multi-scale features.

5. A method for real-time path planning and obstacle avoidance of an unmanned aerial vehicle according to claim 4, characterized in that: The space-time encoder comprises: The adaptive multi-scale feature extraction module, lightweight spatiotemporal attention mechanism, hierarchical dynamic graph neural network and feedback enhancement module are connected in sequence.

6. The method for real-time path planning and obstacle avoidance of an unmanned aerial vehicle according to claim 1, characterized in that: The UAV path planning and obstacle avoidance model is trained through the following steps: Collect different sensor data and preprocess them to obtain model training data sets; Expand the data set and divide the expanded data set into a training set and a validation set; Set the training parameters and train the model using the training set. During the training process, when the cross entropy loss function converges, the model training ends. The trained model is verified through the validation set. If the mean absolute error, mean square error, and root mean square error, which are the model performance evaluation indicators, are all less than the threshold, the model verification is passed; otherwise, the training parameters are adjusted or the training set samples are expanded to retrain the model until the model verification is passed.

7. A real-time path planning and obstacle avoidance device for an unmanned aerial vehicle, characterized in that: The device comprises: The acquisition module is used to obtain the environmental perception information, terrain and weather information, and the drone's own status information during flight based on different sensors; A preprocessing module, used for preprocessing the environmental perception information, terrain and meteorological information, and self-state information; Model building and training module, used to build the UAV path planning and obstacle avoidance model and train the model; The path planning module is used to input the pre-processed environmental perception information, terrain and meteorological information, and its own state information into the trained UAV path planning and obstacle avoidance model to plan the optimal flight path of the UAV and provide the optimal obstacle avoidance mode of the UAV.

8. The real-time path planning and obstacle avoidance device for unmanned aerial vehicles according to claim 7, characterized in that: The preprocessing module comprises: A cleaning and denoising submodule, for cleaning and denoising the environmental perception information, terrain and meteorological information, and self-state information; A synchronization and alignment submodule, for performing data synchronization and timestamp alignment on the cleaned and denoised environmental perception information, terrain and meteorological information, and self-state information; The fusion submodule is used to fuse the environmental perception information, terrain and meteorological information, and self-state information after data synchronization and timestamp alignment.

9. A storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 6.

10. An electronic device, characterized in that: The electronic device comprises: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images