An unmanned aerial vehicle motion planning method and system based on depth information guided visual heat conduction

Abstract

The application discloses a kind of unmanned plane motion planning method and system based on depth information guide visual heat conduction, the unmanned plane motion planning method, the unmanned plane motion planning method is by constructing motion planning model, and depth image, unmanned plane posture and desired speed are input to motion planning model, realize that quadcopter unmanned plane is in unknown complex environment Real-time high-speed obstacle avoidance.This motion planning model uses visual heat conduction module with global receptive field as the main part of perception system, and introduces depth information guide heat conduction operator module, encodes depth information as energy weight, and guides heat conduction calculation after coupling spatial information and scene representation, so that the model can focus on close-range obstacles.Subsequently, the generated depth information guide scene representation is input to decision module to generate action instruction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence control, and particularly relates to a method and system for unmanned aerial vehicle motion planning based on depth information guided visual heat conduction. BACKGROUND

[0002] With the rapid development of unmanned aerial vehicle technology, its application scenarios have been extended from traditional aerial surveying and mapping to power inspection, post-disaster search and rescue, urban air mobility (UAM) and other fields. These emerging applications have posed severe challenges to the real-time high-speed obstacle avoidance of unmanned aerial vehicles in dense obstacle environments. Traditional obstacle avoidance algorithms usually rely on the pre-constructed high-precision three-dimensional prior map and adopt a modular layered architecture of perception-positioning-planning-control. This method leads to excessive system complexity and is susceptible to noise interference, making it difficult to meet the demand for real-time response.

[0003] In recent years, learning-based methods have gradually become an important technical route for solving the problem of autonomous navigation of unmanned aerial vehicles due to their strong representation learning and adaptive decision-making capabilities. Such methods can directly map from raw sensor data to control commands, thereby eliminating the explicit map construction and state estimation links, significantly reducing the system complexity and error accumulation risk. Among onboard sensors, lightweight depth cameras have become the mainstream input modality of end-to-end control systems because they can provide dense depth information.

[0004] However, existing learning-based end-to-end methods still have limitations. Current work mostly follows two major backbone architectures in the field of computer vision: convolutional neural networks (CNNs) and visual Transformers (ViTs). Although CNNs perform well in terms of parameter efficiency and hardware friendliness, their inherent local receptive field limits the ability to model the global spatial relationships between distant obstacles. On the other hand, ViTs can capture long-range dependencies through self-attention mechanisms, improving the reasoning accuracy of complex spatial structures, but their computational complexity grows quadratically with the number of tokens (basic units), leading to a surge in latency and power consumption, making it difficult to meet the stringent constraints of small unmanned aerial vehicles in terms of low power consumption and millisecond-level response. Therefore, how to effectively utilize depth information for high-speed obstacle avoidance of unmanned aerial vehicles while balancing global receptive field and lower computational overhead remains a challenge for current technology. SUMMARY

[0005] The purpose of the present application is to provide a method and system for unmanned aerial vehicle motion planning based on depth information guided visual heat conduction, aiming to solve the contradiction between global spatial information perception and low computational overhead in existing end-to-end visual obstacle avoidance methods in complex unknown environments, especially under high-speed flight conditions, thereby achieving efficient and robust real-time obstacle avoidance capability of quadcopters.

[0006] In a first aspect, the present application provides a UAV motion planning method based on depth information guided visual heat conduction, comprising the following steps:

[0007] A dataset containing depth images, UAV poses and desired velocities is constructed, and the dataset is labeled;

[0008] A motion planning model is constructed; the motion planning model comprises a depth information guided visual heat conduction encoder and a decision network; the depth information guided visual heat conduction encoder introduces a depth information guided heat conduction operator layer to process the depth images, obtains depth guided pre-scene features, and inputs the pre-scene features, the UAV poses and the desired velocities into the decision network to obtain the output results of the motion planning model;

[0009] In the depth information guided heat conduction operator layer, an energy weight matrix in inverse proportion to the depth values of the depth images is generated according to the depth values of the depth images; after the energy weight matrix is processed by downsampling, the processing results are multiplied with the input feature maps element by element to obtain a corrected initial temperature distribution; the depth information guided scene representation output by the depth information guided heat conduction operator layer is obtained based on the corrected initial temperature distribution;

[0010] The motion planning model is trained using the dataset, the actual velocity of the UAV is predicted using the trained motion planning model, and the motion of the UAV is controlled according to the prediction results to realize motion planning.

[0011] As a preferred, the depth information guided visual heat conduction encoder comprises a plurality of sub-encoding modules connected in series; each of the sub-encoding modules comprises a downsampling layer and one or a plurality of feature extraction modules connected in series; the depth information guided visual heat conduction encoder obtains the depth guided pre-scene features by fusing the output feature maps of the plurality of sub-encoding modules.

[0012] As a preferred, in the feature extraction module, the input feature maps are processed through a normalization layer and a depth information guided heat conduction operator module in sequence, and the processing results are fused with the input feature maps to obtain an intermediate feature map; the intermediate feature map is processed using a hybrid feedforward layer, and after the processing results are fused with the intermediate feature map, the processing results are processed using a normalization layer to obtain the output feature map of the feature extraction module.

[0013] As a preferred, the depth information guided heat conduction operator module comprises a depth convolution layer, a first normalization layer, a first linear layer, a depth information guided heat conduction operator layer, a second normalization layer and a second linear layer connected in sequence.

[0014] As a preferred, the input feature map of the depth information guided heat conduction operator layer is the output feature map of the first linear layer in the depth information guided heat conduction operator module.

[0015] As preferred, the method for obtaining the depth information guided scene representation based on the corrected initial temperature distribution is as follows:

[0016] The two-dimensional discrete cosine transform is performed on the corrected initial temperature distribution to obtain a frequency domain representation; a frequency value embedding with the same shape as the frequency domain representation is randomly initialized and then processed through a linear layer to obtain a thermal diffusion coefficient; the frequency domain representation is processed based on the thermal diffusion coefficient, and the processing result is subjected to two-dimensional inverse discrete cosine transform to obtain the depth information guided scene representation.

[0017] As preferred, the processing of the frequency domain representation based on the thermal diffusion coefficient is as follows:

[0018]

[0019] wherein, is the processing result of the thermal diffusion coefficient; is the corrected initial temperature distribution; represents the two-dimensional discrete cosine transform; is the thermal diffusion coefficient; is a fixed time; and represent the cosine wave frequency in the x direction and the y direction, respectively.

[0020] In the second aspect, the present application provides an unmanned aerial vehicle motion planning system based on depth information guided visual heat conduction, which is used to execute the unmanned aerial vehicle motion planning method described above.

[0021] The unmanned aerial vehicle motion planning system comprises an unmanned aerial vehicle and a sensor module, a map construction module and a motion planning module carried on the unmanned aerial vehicle; the sensor module is used to obtain the depth image and the attitude information of the unmanned aerial vehicle; the map construction module is used to construct a three-dimensional point cloud map; and the motion planning module is used to obtain the actual speed of the unmanned aerial vehicle in the three-dimensional point cloud map according to the depth image data, the attitude information and the expected speed.

[0022] In the third aspect, the present application provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; and the processor executes the unmanned aerial vehicle motion planning method described above.

[0023] In the fourth aspect, the present application provides a readable storage medium, which stores a computer program; and the computer program is executable on the processor to implement the unmanned aerial vehicle motion planning method described above.

[0024] The present application has the following beneficial effects:

[0025] 1. The present application adopts a visual heat conduction model as the backbone of the perception system, while maintaining relatively low computational complexity, effectively solving the problems of insufficient local receptive field of traditional CNN and excessive computational overhead of ViT, making it suitable for small unmanned aerial vehicles and other resource-constrained platforms.

[0026] 2. The depth information guided heat conduction operator module introduced in the present application can encode depth information into energy weights and guide the coupling of spatial information and scene representation, so that the model can intelligently focus on close-range and more threatening obstacles, thereby improving the accuracy and safety of the unmanned aerial vehicle obstacle avoidance decision.

[0027] 3. The present application can effectively deal with the challenges of real-time high-speed obstacle avoidance of quadrotor unmanned aerial vehicles in unknown complex environments, especially when the unmanned aerial vehicle speed climbs, it shows more excellent obstacle avoidance performance; at the same time, compared with a variety of advanced learning architectures, the present application shows significantly improved success rate and lower collision rate in different complex scenarios, especially in high-speed flight conditions, proving its excellent performance and good generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the overall flowchart of the present application.

[0029] Figure 2 is the structure diagram of the motion planning model in the present application.

[0030] Figure 3 is the structure diagram of the feature extraction module in the present application.

[0031] Figure 4 is the structure diagram of the depth information guided heat conduction operator module in the present application.

[0032] Figure 5 is the structure diagram of the depth information guided heat conduction operator layer in the present application.

[0033] Figure 6 is the simulation environment scene diagram of the present application; wherein (a) is a ball environment diagram; (b) is a forest environment diagram.

[0034] Figure 7 is the success rate and collision rate comparison diagram of the present application and other unmanned aerial vehicle motion planning methods in the forest environment; wherein (a) is a forest environment success rate diagram; (b) is a forest environment average collision times diagram.

[0035] Figure 8 is the success rate and collision rate comparison diagram of the present application and other unmanned aerial vehicle motion planning methods in the ball environment; wherein (a) is a ball environment success rate diagram; (b) is a ball environment average collision times diagram. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings.

[0037] A method for UAV motion planning based on depth information-guided visual heat conduction is disclosed. The UAV motion planning system includes a quadcopter UAV and image acquisition, map building, and motion planning modules mounted on the UAV. The image acquisition module is used to acquire depth image data; the map building module is used to construct a 3D point cloud map, thereby providing the UAV with accurate global positioning and attitude information; and the motion planning module is used to obtain the actual speed of the UAV in the 3D point cloud map based on the depth image data.

[0038] like Figure 1 As shown, the drone motion planning method includes the following steps:

[0039] Step 1: Building the dataset

[0040] A privileged expert strategy with obstacle perception capabilities was employed to collect data by performing obstacle avoidance tasks at different speeds in a simulated spherical environment. This strategy obtains the position and radius information of obstacles within 10 meters of the current UAV's location. By running a short line-of-sight (SLOS) obstacle avoidance strategy, it outputs action commands to control the simulated UAV to fly forward 60 meters at the desired speed while avoiding obstacles. Specifically, the expert strategy searches for a straight, obstacle-free trajectory from the current position to each waypoint on a lateral-vertical 2D grid plane in front of the quadcopter. It selects the waypoint closest to the grid center and applies thrust relative to its position to generate speed command actions. Sample data in the dataset were collected in the form of {(single-frame depth image, UAV attitude, desired speed), actual speed} supervision data pairs, with 588 expert runs generating 112k supervision data pairs. The depth images in the dataset require normalization. Because this expert strategy relies solely on reactive obstacle avoidance and lacks dynamic path planning, collisions still occur in the data. However, it is more timely than relying on skilled drone operators to collect data and avoids the high-risk accidents that may occur when collecting data in the real world.

[0041] Step 2: Constructing a motion planning model

[0042] like Figure 2As shown, the motion planning model comprises a depth information guided visual heat conduction encoder and a decision network. The depth information guided visual heat conduction encoder is used for processing a depth image, which comprises a first sub-encoding module and a second sub-encoding module connected in series to adapt to the limited computing power of the onboard computer; the depth image is converted into latent features coupled with spatial information through the two sub-encoding modules, which are fed into the subsequent decision network together with the quadrotor attitude and the expected speed. The first sub-encoding module and the second sub-encoding module have the same structure, each comprising a down-sampling layer and two feature extraction modules. In the depth information guided visual heat conduction encoder, the output feature map of the second sub-encoding module is processed by up-sampling, spliced with the output feature map of the first sub-encoding module, and subjected to convolution operation using a convolution layer to realize cross-level information fusion, thereby obtaining a depth-guided pre-scene representation, which is the output of the depth information guided visual heat conduction encoder and which integrates global spatial information and depth threat perception to effectively extract visual features.

[0043] As shown in Figure 3 , in the feature extraction module, the input feature map is processed by a normalization layer and a depth information guided heat conduction operator module in sequence, and the processing result is fused with the input feature map to obtain an intermediate feature map; the intermediate feature map is processed using a hybrid feedforward layer, and after the processing result is fused with the intermediate feature map, the processing result is processed using a normalization layer to obtain the output feature map of the feature extraction module.

[0044] As shown in Figure 4 , the depth information guided heat conduction operator module comprises a depth convolution layer, a first normalization layer, a first linear layer, a depth information guided heat conduction operator layer, a second normalization layer and a second linear layer connected in sequence. In the depth information guided heat conduction operator module, the input feature map is expanded along the channel dimension by the depth convolution layer, and the expanded result is processed using the first normalization layer and the first linear layer to obtain an initial temperature distribution .

[0045] As shown in Figure 5 , in the depth information guided heat conduction operator layer, in order to focus the model on the near-distance obstacles that are greater threats to the quadrotor, an energy weight matrix is generated in inverse proportion to the depth values of the depth image . After the energy weight matrix is aligned with the initial temperature distribution by down-sampling, the aligned result is multiplied element by element with the initial temperature distribution to obtain a modified initial temperature distribution , and the specific expression is as follows:

[0046]

[0047] wherein DS is a down-sampling operation; is a hyper-parameter used to adjust the scaling magnitude; is a depth value; is the initial temperature distribution before adjustment.

[0048] Ensures that the near distance obstacles have higher energy weight in the heat conduction process, thereby guiding the scene representation to pay more attention to the current threat. The modified initial temperature distribution is subjected to a two-dimensional discrete cosine transform (DCT) to convert it to the frequency domain to obtain a frequency domain representation. By randomly initializing a frequency value embedding (FVE) with the same shape as the frequency domain representation, and processing it through a linear layer, a thermal diffusion coefficient is obtained; based on the thermal diffusion coefficient the frequency domain representation is processed, and the processing result is subjected to a two-dimensional inverse discrete cosine transform (IDCT) to convert it back to the spatial domain to obtain a depth information guided scene representation output by the depth information guided heat conduction operator layer, whose specific expression is:

[0049]

[0050] wherein, is the thermal diffusion coefficient; is a fixed time; and represent the cosine wave frequency in the x direction and the y direction, respectively.

[0051] The decision network adopts a multi-layer fully connected network, and the front scene representation, as well as the unmanned aerial vehicle attitude and expected speed, are processed through the decision network to obtain the output result (linear velocity command) of the motion planning model.

[0052] Step three, use the dataset to train the motion planning model, and use the L2 loss function to optimize all time steps in the trajectory, whose expression is:

[0053]

[0054] wherein, is the label value; is the predicted speed.

[0055] Step four, use the trained motion planning model to predict the actual speed of the unmanned aerial vehicle, and control the motion of the quadrotor unmanned aerial vehicle according to the prediction result to realize obstacle avoidance.

[0056] To measure the performance of the proposed model, a flightmare simulation environment platform based on Unity is used, which mainly includes two core modules: a configurable rendering engine and a flexible physics engine. The rendering engine generates a high-fidelity three-dimensional visual environment, including a forest environment and a sphere environment, as shown in Figure 6 The physics engine focuses on the dynamics simulation of quadrotors. The sphere environment is similar to the environment used for dataset construction, with 200 white spheres of different sizes randomly distributed in different positions; the forest environment is filled with 200 similar-shaped trees to evaluate the generalization ability of the model. The baseline models for comparison include a convolutional neural network model (Convnet), a convolutional neural network model with U-shaped architecture (UNet), a visual Transformer model (ViT), and a visual heat conduction model (vHeat). In both environments, different models are tested at speeds of 3m / s, 4m / s, 5m / s, 6m / s, and 7m / s for 10 times, with the goal of flying 60 meters straight ahead while avoiding obstacles. The evaluation results are shown in Figure 7 and Figure 8 In the sphere environment, as the speed climbs to 5m / s, the success rate of the present invention is significantly higher than that of other baseline models, and the collision rate is also at a lower level. In the forest environment, only the present invention can achieve a success rate of more than 80% at different speeds. Compared with the unmodified vHeat model, the present method improves the success rate and further reduces the collision rate by coupling spatial information and scene representation to assist decision-making, which shows that the depth information guided heat conduction operator module can effectively guide the model to focus on more threatening obstacles and make more correct decisions. The experimental results fully verify the significant advantages of the present invention in performance and generalization ability.

[0057] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above examples, and the above examples and descriptions in the specification are only the principles of the present invention. Without departing from the spirit and scope of the present invention, various changes and improvements can be made to the present invention, and these changes and improvements all fall within the scope of the claimed present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for UAV motion planning based on depth information-guided visual heat transfer, characterized in that: Includes the following steps: Construct a dataset containing depth images, UAV pose, and desired velocity, and label the dataset. A motion planning model is constructed. The motion planning model includes a depth information-guided visual thermal conduction encoder and a decision network. The depth information-guided visual thermal conduction encoder introduces a depth information-guided thermal conduction operator layer to process the depth image, obtain the depth-guided foreground scene features, and inputs the foreground scene features, along with the UAV attitude and desired velocity, into the decision network to obtain the output of the motion planning model. In the depth information-guided heat conduction operator layer, an energy weight matrix that is inversely proportional to the depth value is generated based on the depth value of the depth image. After downsampling the energy weight matrix, the processing result is multiplied element-wise with the input feature map to obtain the corrected initial temperature distribution. Based on the corrected initial temperature distribution, the depth information-guided scene representation output by the depth information-guided heat conduction operator layer is obtained. The motion planning model is trained using a dataset, and the trained motion planning model is used to predict the actual speed of the drone. The drone's motion is then controlled based on the prediction results to achieve motion planning.

2. The UAV motion planning method based on depth information-guided visual heat conduction according to claim 1, characterized in that: The depth-guided visual thermal conduction encoder includes multiple sub-encoding modules connected in series; each sub-encoding module includes a downsampling layer and one or more feature extraction modules connected in series; the depth-guided visual thermal conduction encoder obtains depth-guided foreground features by fusing the output feature maps of multiple sub-encoding modules and performing convolution processing on the fusion result.

3. The UAV motion planning method based on depth information-guided visual heat conduction according to claim 2, characterized in that: In the feature extraction module, the input feature map is processed sequentially by the normalization layer and the heat conduction operator module guided by the depth information, and the processing result is fused with the input feature map to obtain the intermediate feature map. The intermediate feature map is then processed by the hybrid feedforward layer, and the processing result is fused with the intermediate feature map. Finally, the normalization layer is used to process the intermediate feature map to obtain the output feature map of the feature extraction module.

4. The UAV motion planning method based on depth information-guided visual heat conduction according to claim 3, characterized in that: The depth information-guided heat conduction operator module includes a depth convolutional layer, a first normalized layer, a first linear layer, a depth information-guided heat conduction operator layer, a second normalized layer, and a second linear layer connected in sequence.

5. The UAV motion planning method based on depth information-guided visual heat conduction according to claim 4, characterized in that: The input feature map of the depth information-guided heat conduction operator layer is the output feature map of the first linear layer in the depth information-guided heat conduction operator module.

6. The UAV motion planning method based on depth information-guided visual heat conduction according to claim 1, characterized in that: The method for guiding scene representation by obtaining depth information based on the corrected initial temperature distribution is as follows: A two-dimensional discrete cosine transform is performed on the corrected initial temperature distribution to obtain its frequency domain representation; a frequency value with the same shape as the frequency domain representation is randomly initialized and embedded, and then processed through a linear layer to obtain the thermal diffusivity. The frequency domain representation is processed based on the thermal diffusivity coefficient, and the processing result is subjected to a two-dimensional inverse discrete cosine transform to obtain a depth information-guided scene representation.

7. The UAV motion planning method based on depth information-guided visual heat conduction according to claim 6, characterized in that: The frequency domain representation is processed based on the thermal diffusivity as follows: in, The result is the thermal diffusivity. The corrected initial temperature distribution; Represents the two-dimensional discrete cosine transform; The thermal diffusivity; For a fixed time; and These represent the cosine wave frequencies in the x and y directions, respectively.

8. A drone motion planning system based on depth information-guided visual heat conduction, characterized in that: This method is used to execute the UAV motion planning method based on depth information-guided visual heat conduction as described in claim 1. The UAV motion planning system includes a UAV and a sensor module, a map building module, and a motion planning module mounted on the UAV. The sensor module is used to acquire depth images and attitude information of the UAV. The map building module is used to construct a three-dimensional point cloud map. The motion planning module is used to obtain the actual speed of the UAV in the three-dimensional point cloud map based on the depth image data, attitude information, and desired speed.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The memory stores a computer program; the processor executes a UAV motion planning method based on depth information-guided visual heat conduction as described in any one of claims 1-7.

10. A readable storage medium storing a computer program; characterized in that: When the computer program is executed by the processor, it is used to implement a UAV motion planning method based on depth information-guided visual heat conduction as described in any one of claims 1-7.

Images