Unmanned aerial vehicle motion planning method and system for guiding visual heat conduction based on depth information

Abstract

The invention discloses an unmanned aerial vehicle motion planning method and system for guiding visual heat conduction based on depth information, and the method comprises the steps: constructing a motion planning model, and inputting a depth image, an unmanned aerial vehicle attitude and an expected speed into the motion planning model, and real-time high-speed obstacle avoidance of the quad-rotor unmanned aerial vehicle in an unknown complex environment is realized. According to the motion planning model, a visual heat conduction module with a global receptive field is adopted as a sensing system trunk, a depth information guide heat conduction operator module is introduced, depth information is coded into an energy weight, and heat conduction calculation is performed after spatial information and scene representation are guided to be coupled, so that the model can focus on a close-range obstacle. The generated depth information guide scene representation is then input to a decision module to generate an action instruction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence control, and particularly relates to a method and system for unmanned aerial vehicle motion planning based on depth information guided visual heat conduction. BACKGROUND

[0002] With the rapid development of unmanned aerial vehicle technology, its application scenarios have been extended from traditional aerial surveying and mapping to power inspection, post-disaster search and rescue, urban air mobility (UAM) and other fields. These emerging applications have posed severe challenges to the real-time high-speed obstacle avoidance of unmanned aerial vehicles in dense obstacle environments. Traditional obstacle avoidance algorithms usually rely on the pre-constructed high-precision three-dimensional prior map and adopt a modular layered architecture of perception-positioning-planning-control. This method leads to excessive system complexity and is susceptible to noise interference, making it difficult to meet the demand for real-time response.

[0003] In recent years, learning-based methods have gradually become an important technical route for solving the problem of autonomous navigation of unmanned aerial vehicles due to their strong representation learning and adaptive decision-making capabilities. Such methods can directly map from raw sensor data to control commands, thereby eliminating the explicit map construction and state estimation links, significantly reducing the system complexity and error accumulation risk. Among onboard sensors, lightweight depth cameras have become the mainstream input modality of end-to-end control systems because they can provide dense depth information.

[0004] However, existing learning-based end-to-end methods still have limitations. Current work mostly follows two major backbone architectures in the field of computer vision: convolutional neural networks (CNNs) and visual Transformers (ViTs). Although CNNs perform well in terms of parameter efficiency and hardware friendliness, their inherent local receptive field limits the ability to model the global spatial relationships between distant obstacles. On the other hand, ViTs can capture long-range dependencies through self-attention mechanisms, improving the reasoning accuracy of complex spatial structures, but their computational complexity grows quadratically with the number of tokens (basic units), leading to a surge in latency and power consumption, making it difficult to meet the stringent constraints of small unmanned aerial vehicles in terms of low power consumption and millisecond-level response. Therefore, how to effectively utilize depth information for high-speed obstacle avoidance of unmanned aerial vehicles while balancing global receptive field and lower computational overhead remains a challenge for current technology. SUMMARY

[0005] The purpose of the present application is to provide a method and system for unmanned aerial vehicle motion planning based on depth information guided visual heat conduction, aiming to solve the contradiction between global spatial information perception and low computational overhead in existing end-to-end visual obstacle avoidance methods in complex unknown environments, especially under high-speed flight conditions, thereby achieving efficient and robust real-time obstacle avoidance capability of quadcopters.

[0006] In a first aspect, the present application provides a UAV motion planning method based on depth information guided visual heat conduction, comprising the following steps:

[0007] A dataset containing depth images, UAV poses and desired velocities is constructed, and the dataset is labeled;

[0008] A motion planning model is constructed; the motion planning model comprises a depth information guided visual heat conduction encoder and a decision network; the depth information guided visual heat conduction encoder introduces a depth information guided heat conduction operator layer to process the depth images, obtains depth guided pre-scene features, and inputs the pre-scene features, the UAV poses and the desired velocities into the decision network to obtain the output results of the motion planning model;

[0009] In the depth information guided heat conduction operator layer, an energy weight matrix in inverse proportion to the depth values of the depth images is generated according to the depth values of the depth images; after the energy weight matrix is processed by downsampling, the processing results are multiplied with the input feature maps element by element to obtain a corrected initial temperature distribution; the depth information guided scene representation output by the depth information guided heat conduction operator layer is obtained based on the corrected initial temperature distribution;

[0010] The motion planning model is trained using the dataset, the actual velocity of the UAV is predicted using the trained motion planning model, and the motion of the UAV is controlled according to the prediction results to realize motion planning.

[0011] As a preferred, the depth information guided visual heat conduction encoder comprises a plurality of sub-encoding modules connected in series; each of the sub-encoding modules comprises a downsampling layer and one or a plurality of feature extraction modules connected in series; the depth information guided visual heat conduction encoder obtains the depth guided pre-scene features by fusing the output feature maps of the plurality of sub-encoding modules.

[0012] As a preferred, in the feature extraction module, the input feature maps are processed through a normalization layer and a depth information guided heat conduction operator module in sequence, and the processing results are fused with the input feature maps to obtain an intermediate feature map; the intermediate feature map is processed using a hybrid feedforward layer, and after the processing results are fused with the intermediate feature map, the processing results are processed using a normalization layer to obtain the output feature map of the feature extraction module.

[0013] As a preferred, the depth information guided heat conduction operator module comprises a depth convolution layer, a first normalization layer, a first linear layer, a depth information guided heat conduction operator layer, a second normalization layer and a second linear layer connected in sequence.

[0014] As a preferred, the input feature map of the depth information guided heat conduction operator layer is the output feature map of the first linear layer in the depth information guided heat conduction operator module.

[0015] As a preferred method, the approach for guiding scene representation based on depth information obtained from the corrected initial temperature distribution is as follows:

[0016] A two-dimensional discrete cosine transform is performed on the corrected initial temperature distribution to obtain a frequency domain representation. A frequency value embedding with the same shape as the frequency domain representation is randomly initialized and then processed through a linear layer to obtain the thermal diffusivity coefficient. The frequency domain representation is processed based on the thermal diffusivity coefficient, and a two-dimensional inverse discrete cosine transform is performed on the processing result to obtain a depth information-guided scene representation.

[0017] As a preferred approach, the frequency domain representation is processed based on the thermal diffusivity as follows:

[0018]

[0019] in, The result is the thermal diffusivity. The corrected initial temperature distribution; Represents the two-dimensional discrete cosine transform; The thermal diffusivity; For a fixed time; and These represent the cosine wave frequencies in the x and y directions, respectively.

[0020] Secondly, the present invention provides a UAV motion planning system based on depth information-guided visual heat conduction, which is used to execute the aforementioned UAV motion planning method.

[0021] The UAV motion planning system includes a UAV and a sensor module, a map building module, and a motion planning module mounted on the UAV. The sensor module is used to acquire depth images and attitude information of the UAV. The map building module is used to construct a three-dimensional point cloud map. The motion planning module is used to obtain the actual speed of the UAV in the three-dimensional point cloud map based on the depth image data, attitude information, and desired speed.

[0022] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; and the processor executes the above-described unmanned aerial vehicle motion planning method.

[0023] Fourthly, the present invention provides a readable storage medium storing a computer program; when the computer program is executed by a processor, it is used to implement the above-described UAV motion planning method.

[0024] The beneficial effects of this invention are:

[0025] 1. This invention uses a visual heat conduction model as the backbone of the perception system, which achieves a global receptive field while maintaining relatively low computational complexity. It effectively solves the problems of insufficient local receptive field in traditional CNNs and excessive computational overhead in ViT, making it suitable for resource-constrained platforms such as small drones.

[0026] 2. The depth information-guided heat conduction operator module introduced in this invention can encode depth information into energy weights and guide the coupling of spatial information with scene representation, enabling the model to intelligently focus on closer and more threatening obstacles, thereby improving the accuracy and safety of UAV obstacle avoidance decisions.

[0027] 3. This invention can effectively address the challenges of real-time high-speed obstacle avoidance for quadcopter drones in unknown and complex environments, especially when the drone's speed increases, demonstrating superior obstacle avoidance performance. At the same time, under different complex scenarios, especially under high-speed flight conditions, this invention shows a significantly improved success rate and a lower collision rate compared to many state-of-the-art learning architectures, proving its excellent performance and good generalization ability. Attached Figure Description

[0028] Figure 1 This is the overall flowchart of the present invention.

[0029] Figure 2 This is a schematic diagram of the motion planning model in this invention.

[0030] Figure 3 This is a schematic diagram of the feature extraction module in this invention.

[0031] Figure 4 This is a schematic diagram of the structure of the depth information-guided heat conduction operator module in this invention.

[0032] Figure 5 This is a schematic diagram of the structure of the depth information-guided heat conduction operator layer in this invention.

[0033] Figure 6 The following is a simulation environment scene diagram of the present invention; wherein, (a) is a schematic diagram of the sphere environment; and (b) is a schematic diagram of the forest environment.

[0034] Figure 7 This is a schematic diagram comparing the success rate and collision rate of the present invention with other UAV motion planning methods in a forest environment; wherein, (a) is a schematic diagram of the success rate in a forest environment; and (b) is a schematic diagram of the average number of collisions in a forest environment.

[0035] Figure 8 This is a schematic diagram comparing the success rate and collision rate of the present invention with other UAV motion planning methods in a spherical environment; wherein, (a) is a schematic diagram of the success rate in a spherical environment; and (b) is a schematic diagram of the average number of collisions in a spherical environment. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings.

[0037] A method for UAV motion planning based on depth information-guided visual heat conduction is disclosed. The UAV motion planning system includes a quadcopter UAV and image acquisition, map building, and motion planning modules mounted on the UAV. The image acquisition module is used to acquire depth image data; the map building module is used to construct a 3D point cloud map, thereby providing the UAV with accurate global positioning and attitude information; and the motion planning module is used to obtain the actual speed of the UAV in the 3D point cloud map based on the depth image data.

[0038] like Figure 1 As shown, the drone motion planning method includes the following steps:

[0039] Step 1: Building the dataset

[0040] A privileged expert strategy with obstacle perception capabilities was employed to collect data by performing obstacle avoidance tasks at different speeds in a simulated spherical environment. This strategy obtains the position and radius information of obstacles within 10 meters of the current UAV's location. By running a short line-of-sight (SLOS) obstacle avoidance strategy, it outputs action commands to control the simulated UAV to fly forward 60 meters at the desired speed while avoiding obstacles. Specifically, the expert strategy searches for a straight, obstacle-free trajectory from the current position to each waypoint on a lateral-vertical 2D grid plane in front of the quadcopter. It selects the waypoint closest to the grid center and applies thrust relative to its position to generate speed command actions. Sample data in the dataset were collected in the form of {(single-frame depth image, UAV attitude, desired speed), actual speed} supervision data pairs, with 588 expert runs generating 112k supervision data pairs. The depth images in the dataset require normalization. Because this expert strategy relies solely on reactive obstacle avoidance and lacks dynamic path planning, collisions still occur in the data. However, it is more timely than relying on skilled drone operators to collect data and avoids the high-risk accidents that may occur when collecting data in the real world.

[0041] Step 2: Constructing a motion planning model

[0042] like Figure 2As shown, the motion planning model includes a depth-guided visual thermal conduction encoder and a decision network. The depth-guided visual thermal conduction encoder processes the depth image and includes a first sub-encoding module and a second sub-encoding module in series to accommodate the limited computing power of the onboard computer. These two sub-encoding modules transform the depth image into latent features coupled with spatial information. These latent features, along with the quadrotor's attitude and desired velocity, are then fed into the subsequent decision network. The first and second sub-encoding modules have identical structures, each including a downsampling layer and two feature extraction modules. In the depth-guided visual thermal conduction encoder, the output feature map of the second sub-encoding module is upsampled and then concatenated with the output feature map of the first sub-encoding module. A convolutional layer is then used to perform a convolution operation on the concatenated result to achieve cross-level information fusion, obtaining a depth-guided pre-scene representation as the output of the depth-guided visual thermal conduction encoder. This representation integrates global spatial information and depth threat perception, effectively extracting visual features.

[0043] like Figure 3 As shown, in the feature extraction module, the input feature map is processed sequentially by the normalization layer and the heat conduction operator module guided by the depth information, and the processing result is fused with the input feature map to obtain the intermediate feature map; the intermediate feature map is processed by the hybrid feedforward layer, and the processing result is fused with the intermediate feature map, and then processed by the normalization layer to obtain the output feature map of the feature extraction module.

[0044] like Figure 4 As shown, the depth-guided heat conduction operator module includes a depth convolutional layer, a first normalization layer, a first linear layer, a depth-guided heat conduction operator layer, a second normalization layer, and a second linear layer connected in sequence. In the depth-guided heat conduction operator module, the input feature map is expanded along the channel dimension through the depth convolutional layer, and the expansion result is processed using the first normalization layer and the first linear layer to obtain the initial temperature distribution. .

[0045] like Figure 5 As shown, in the depth-guided heat conduction operator layer, in order to make the model focus on nearby obstacles that pose a greater threat to the quadrotor, the depth values ​​of the depth image are used... This generates an energy weight matrix that is inversely proportional to the depth value. The energy weight matrix is ​​then downsampled and compared with the initial temperature distribution. After alignment, the alignment result is compared with the initial temperature distribution. Element-wise multiplication yields the corrected initial temperature distribution. Its specific expression is:

[0046]

[0047] Where DS represents the downsampling operation; This is a hyperparameter used to adjust the scaling factor; This is the depth value; This is the initial temperature distribution before adjustment.

[0048] This ensures that nearby obstacles have a higher energy weight during heat conduction, thereby guiding scene characterization to focus more on the current threat. (Regarding the corrected initial temperature distribution...) Perform two-dimensional discrete cosine transform ( The thermal diffusivity is obtained by converting the frequency value to the frequency domain and then processing it through a linear layer. A frequency value embedding (FVE) with the same shape as the frequency domain representation is randomly initialized. Based on thermal diffusivity The frequency domain representation is processed, and the processing result is subjected to a two-dimensional inverse discrete cosine transform. Transforming it back into the spatial domain yields the depth-guided scene representation output by the depth-guided heat conduction operator layer. Its specific expression is:

[0049]

[0050] in, The thermal diffusivity; For a fixed time; and These represent the cosine wave frequencies in the x and y directions, respectively.

[0051] The decision network adopts a multi-layer fully connected network. It processes the foreground scene representation, UAV attitude and desired velocity through the decision network to obtain the output result of the motion planning model (linear velocity command).

[0052] Step 3: Train the motion planning model using the dataset and apply the L2 loss function. The optimization of all time steps in the trajectory is expressed as follows:

[0053]

[0054] in, For tag values; To predict speed.

[0055] Step 4: Use the trained motion planning model to predict the actual speed of the UAV, and control the movement of the quadcopter UAV based on the prediction results to achieve obstacle avoidance.

[0056] To evaluate the performance of the proposed model, the Flightmare simulation environment platform, built on Unity, was used. This platform architecture primarily comprises two core modules: a configurable rendering engine and a flexible physics engine. The rendering engine generates highly realistic 3D visual environments, including forest and spherical environments, such as... Figure 6 As shown. The physics engine focuses on the dynamics simulation of the quadcopter. The spherical environment is similar to the environment used to build the dataset, with 200 white spheres of different sizes randomly distributed in different locations; the forest environment is filled with 200 trees of similar shapes to evaluate the model's generalization ability. The baseline models for comparison include a convolutional neural network model (ConvNet), a U-shaped convolutional neural network model (UNet), a visual Transformer model (ViT), and a visual heat conduction model (vHeat). In both environments, different models were tested 10 times at speeds of 3m / s, 4m / s, 5m / s, 6m / s, and 7m / s, with the goal of flying 60 meters straight ahead while avoiding obstacles. The evaluation results are as follows. Figure 7 and Figure 8 As shown, in a spherical environment, as the speed increases to 5 m / s, this invention significantly outperforms other baseline models in success rate, while maintaining a low collision rate. In a forest environment, only this invention achieves a success rate exceeding 80% across different speed conditions. Compared to the unmodified vHeat model, this method significantly improves the success rate and further reduces the collision rate by coupling spatial information with scene representation to aid decision-making. This demonstrates that the depth-guided heat conduction operator module can effectively guide the model to focus on more threatening obstacles and make more accurate decisions. The experimental results fully validate the significant advantages of this invention in terms of performance and generalization ability.

[0057] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for unmanned aerial vehicle (UAV) motion planning based on depth information guided visual heat conduction, characterized in that: The method comprises the following steps: constructing a dataset comprising a depth image, a UAV attitude and a desired speed, and labeling the dataset; constructing a motion planning model; the motion planning model comprises a depth information guided visual heat conduction encoder and a decision network; the depth information guided visual heat conduction encoder introduces a depth information guided heat conduction operator layer to process the depth image, to obtain a depth guided front scene feature, and inputs the front scene feature, the UAV attitude and the desired speed into the decision network, to obtain an output result of the motion planning model; in the depth information guided heat conduction operator layer, an energy weight matrix in inverse proportion to a depth value of the depth image is generated according to the depth value; the energy weight matrix is processed by downsampling, and the processed result is multiplied with an input feature map element by element, to obtain a corrected initial temperature distribution; and a depth information guided scene representation output by the depth information guided heat conduction operator layer is obtained based on the corrected initial temperature distribution; the motion planning model is trained using the dataset, the trained motion planning model is used to predict an actual speed of the UAV, and the motion of the UAV is controlled according to the prediction result, to realize motion planning. 2.The method of claim 1, wherein: The depth information guided visual heat conduction encoder comprises a plurality of sub-encoding modules connected in series; each of the sub-encoding modules comprises a downsampling layer and one or a plurality of feature extraction modules connected in series; the depth information guided visual heat conduction encoder fuses output feature maps of the plurality of sub-encoding modules, and performs convolution processing on the fused result, to obtain the depth guided front scene feature. 3.The method of claim 2, wherein: In the feature extraction module, an input feature map is processed through a normalization layer and a depth information guided heat conduction operator module in sequence, and the processed result is fused with the input feature map, to obtain an intermediate feature map; the intermediate feature map is processed using a hybrid feedforward layer, and the processed result is fused with the intermediate feature map, and then processed using a normalization layer, to obtain an output feature map of the feature extraction module. 4.The method of claim 3, wherein: The depth information guided heat conduction operator module comprises a depth convolution layer, a first normalization layer, a first linear layer, a depth information guided heat conduction operator layer, a second normalization layer and a second linear layer connected in sequence.

5. The unmanned aerial vehicle motion planning method based on depth information guided visual heat conduction according to claim 4, characterized in that: The input feature map of the depth information guided heat conduction operator layer is an output feature map of the first linear layer in the depth information guided heat conduction operator module. 6.The method of claim 1, wherein: The method for obtaining the depth information guided scene representation based on the corrected initial temperature distribution is as follows: a two-dimensional discrete cosine transform is performed on the corrected initial temperature distribution, to obtain a frequency domain representation; a frequency value embedding with the same shape as the frequency domain representation is randomly initialized, and then processed through a linear layer, to obtain a thermal diffusion coefficient; the frequency domain representation is processed based on the thermal diffusion coefficient, and a two-dimensional inverse discrete cosine transform is performed on the processed result, to obtain the depth information guided scene representation.

7. The unmanned aerial vehicle motion planning method based on depth information guided visual heat conduction according to claim 6, characterized in that: The method for processing the frequency domain representation based on the thermal diffusion coefficient is as follows: wherein is a processing result of a thermal diffusion coefficient; is a corrected initial temperature distribution; denotes a two-dimensional discrete cosine transform; is a thermal diffusion coefficient; is a fixed time; and denote cosine wave frequencies in the x direction and the y direction, respectively. 8.A UAV motion planning system based on depth information guided visual heat conduction, characterized in that: The method is used for executing the unmanned aerial vehicle motion planning method based on depth information guided visual heat conduction according to claim 1 The adopted unmanned aerial vehicle motion planning system comprises an unmanned aerial vehicle and a sensor module, a map construction module and a motion planning module carried on the unmanned aerial vehicle; the sensor module is used for acquiring a depth image and attitude information of the unmanned aerial vehicle; the map construction module is used for constructing a three-dimensional point cloud map; and the motion planning module is used for acquiring an actual speed of the unmanned aerial vehicle in the three-dimensional point cloud map according to the depth image data, the attitude information and a desired speed.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: The memory stores a computer program; and the processor executes a kind of unmanned aerial vehicle motion planning method based on depth information guiding visual heat conduction as claimed in any one of claims 1-7.

10. A readable storage medium, storing a computer program; characterized in that: The computer program is executed by the processor to implement a kind of unmanned aerial vehicle motion planning method based on depth information guiding visual heat conduction as claimed in any one of claims 1-7.

Images