Robot real-time path planning method and system based on depth-space information combined guidance

Abstract

The invention discloses a robot real-time path planning method and system based on depth-space information combined guidance, and the method achieves the efficient obstacle avoidance of a mobile robot through constructing a path planning model, and inputting a depth image, attitude information and an expected speed into the path planning model. The path planning model generates an obstacle avoidance correlation weight matrix in the sensing network through depth-space information coding, guides a visual Transform network to focus effective information, and suppresses redundant interference; the generated depth-space information guide scene representation is input into a decision network, and a motion control instruction is output; through targeted distribution of the feature weights, the representation intensity of obstacle avoidance related features is improved, interference of irrelevant features is reduced, the method is suitable for various mobile robots needing autonomous obstacle avoidance such as unmanned aerial vehicles and ground mobile robots, and the obstacle avoidance success rate and the real-time response speed of the robots in a complex environment are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of artificial intelligence and robot control, specifically relating to a real-time path planning method and system for robots based on joint guidance of depth-space information. Background Technology

[0002] With the popularization of robotics technology, its application scenarios have expanded to complex fields such as power line inspection, disaster search and rescue, urban air transportation, warehousing and logistics, and ground inspection, which places higher demands on the real-time obstacle avoidance capabilities of mobile robots in dense obstacle environments. Traditional obstacle avoidance algorithms rely on high-precision prior maps and a hierarchical architecture of "perception-localization-planning-control," resulting in high system complexity, weak anti-interference capabilities, difficulty in meeting the real-time response requirements of high-speed movement, and poor versatility, making it difficult to adapt to different types of mobile robots.

[0003] In recent years, end-to-end learning-based methods have become the mainstream technology for autonomous robot navigation due to their advantage of directly mapping sensor data to control commands. Lightweight depth cameras, as core sensors, provide dense spatial depth information, supporting end-to-end control. However, existing end-to-end methods still have significant limitations: Convolutional Neural Networks (CNNs) are limited by their local receptive fields, making it difficult to model the global spatial relationships of distant obstacles; while traditional Visual Transformers (ViTs) can capture long-range dependencies, they lack an effective mechanism to distinguish between obstacle avoidance-related and irrelevant features—depth images contain a large amount of non-threatening background areas and irrelevant distant spaces, which are mixed with obstacle avoidance-related features such as obstacle contours, distances, and spatial distributions, resulting in low model feature extraction efficiency and insufficient weighting of effective information, directly affecting the accuracy and real-time performance of obstacle avoidance decisions; furthermore, existing methods are mostly designed for specific types of robots (such as single drones), lacking versatility and making it difficult to quickly migrate to other mobile robot platforms.

[0004] Therefore, how to integrate deep-spatial information guidance into the network architecture, strengthen obstacle avoidance-related features and suppress obstacle avoidance-irrelevant features through differentiated allocation of feature weights, while maintaining global perception capabilities and improving the universality of the method to adapt to the obstacle avoidance needs of various mobile robots, has become a technical problem that urgently needs to be solved in the field of high-speed obstacle avoidance for robots. Summary of the Invention

[0005] This invention provides a real-time path planning method and system for robots based on joint guidance of depth and spatial information. It aims to solve the core problems of "confusion between obstacle avoidance-related features and irrelevant features and insufficient weight of effective information" in existing end-to-end obstacle avoidance methods. At the same time, it improves the versatility of the method, adapts to various mobile robots such as UAVs and ground mobile robots, ensures global perception capabilities, and improves the obstacle avoidance robustness and real-time performance of robots when moving at high speed in unknown and complex environments.

[0006] In a first aspect, the present invention provides a real-time path planning method for robots based on joint guidance of depth and spatial information, the method comprising:

[0007] Construct a dataset containing depth images, pose information, and desired velocities, and then label the dataset.

[0008] A path planning model is constructed; the path planning model includes a perception network and a decision network; the perception network is used to convert the depth image into a foreground scene representation, and inputs the foreground scene representation, pose information and desired velocity into the decision network for processing to obtain the predicted action command;

[0009] The perception network introduces a depth-spatial information guidance module to assist in feature extraction of depth images. In the depth-spatial information guidance module, the input feature map is weighted and scaled element-wise using a depth-spatial guidance weight matrix, and the weighted feature map is residually connected to the input feature map to obtain the output feature map of the depth-spatial information guidance module. The depth-spatial guidance weight matrix is ​​constructed based on the depth guidance weight matrix and the spatial guidance weight matrix.

[0010] The path planning model is trained using a dataset, and the trained model is used to predict the robot's actual movement speed. The robot's movement is then controlled based on the actual movement speed, thereby achieving real-time path planning for the robot.

[0011] Preferably, the perception network includes multiple encoding modules connected in series and a feature fusion module; the multiple encoding modules process the depth image sequentially, and the feature fusion module fuses the output feature maps of each encoding module to obtain the foreground scene representation output by the perception network.

[0012] Preferably, the encoding module includes a downsampling layer and multiple feature extraction units connected in sequence; the feature extraction unit includes a depth-spatial information guidance module and a visual Transformer multi-head attention module connected in series.

[0013] Preferably, in the feature fusion module, a pixel recombination operation is used to upsample the output feature map except for the first encoding module, so that the resolution of its output feature map is the same as that of the first encoding module; after channel concatenation of the output feature maps of each encoding module, dimensional fusion is performed through a convolutional layer to obtain the output feature map of the feature fusion module.

[0014] Preferably, the depth guidance weight matrix is ​​constructed based on depth image information; the spatial guidance weight matrix is ​​generated based on the two-dimensional spatial position information of the obstacle relative to the robot.

[0015] Preferably, the depth-space guided weight matrix The construction method is as follows:

[0016]

[0017] in, These are the weighting coefficients; The depth value for each pixel; x and y are the horizontal and vertical coordinates of each pixel, respectively; and , where x and y are the x and y coordinates of the center point of the Gaussian distribution, respectively; and These are the standard deviations of the Gaussian distribution in the width and height directions, respectively. This is the error compensation coefficient.

[0018] Preferably, in the visual Transformer multi-head attention module, the input feature map is split into multiple feature blocks, and the feature blocks are mapped to queries, keys, and values ​​respectively through independent linear layers; after downsampling the keys and values, multi-head attention is calculated jointly by the queries, and the attention output is obtained by fusion; the attention output is nonlinearly transformed by a hybrid feedforward layer to obtain the output feature map of the visual Transformer multi-head attention module.

[0019] As a preferred approach, in the decision network, the foreground scene representation, pose information, and desired velocity are flattened and vectorized before being input into a multi-layer cascaded fully connected layer to obtain the output of the decision network.

[0020] Preferably, the path planning model is trained using the preprocessed dataset; the preprocessing process involves normalizing the depth images in the dataset and converting the depth values ​​I... d Map to the [0,1] interval; use Kalman filtering algorithm to filter and denoise the attitude information; align all data with timestamps.

[0021] Secondly, the present invention provides a robot real-time path planning system based on joint guidance of depth-spatial information, which is used to execute the above-mentioned robot real-time path planning method; the robot real-time path planning system includes an image acquisition module, a map construction module, and a motion planning module; the image acquisition module is used to acquire depth image data; the map construction module is used to construct a three-dimensional point cloud map and obtain the robot's posture information; the motion planning module is used to determine the robot's actual movement speed in the three-dimensional point cloud map based on the depth image, posture information, and desired speed.

[0022] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described real-time path planning method for robots.

[0023] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described real-time path planning method for robots.

[0024] The beneficial effects of this invention are:

[0025] 1. This invention introduces a depth-spatial information guidance module into the perception network. The depth-spatial information guidance module generates an adaptive weight matrix, which assigns high weights to obstacle avoidance-related features such as nearby obstacles and flight path areas, and low weights to irrelevant features such as non-threatening backgrounds and distant areas. This effectively solves the feature confusion problem, enabling the model to focus on core obstacle avoidance information. In high-speed motion scenarios, the obstacle avoidance performance is superior to other methods.

[0026] 2. This invention uses a depth-spatial information-guided visual Transformer as the backbone of the perception network. While enhancing effective features, it fully retains ViT's ability to model global spatial relationships, ensuring that various robots can comprehensively perceive the distribution of obstacles in complex environments and improve the global rationality of obstacle avoidance decisions.

[0027] 3. This invention adopts a modular and universal design. The perception network is not dependent on a specific robot type, and the decision network can be fine-tuned to adapt to various mobile robots such as drones, ground mobile robots, and robotic dogs, which has strong versatility and reduces cross-platform migration costs. In terms of improvement, this invention adopts an end-to-end model architecture, which eliminates the explicit map construction and state estimation links, reducing system complexity and error accumulation. Moreover, due to the feature selection mechanism, the model exhibits high success rate and low collision rate in different complex scenarios such as spherical environments and forest environments, and has strong generalization ability. Attached Figure Description

[0028] Figure 1 This is the overall flowchart of the present invention.

[0029] Figure 2 This is a schematic diagram of the feature extraction unit in this invention.

[0030] Figure 3 This is a visualization heatmap of the depth-spatial information-guided weight matrix in this invention.

[0031] Figure 4This is a schematic diagram comparing the success rates of the present invention with other path planning methods; wherein, (a) is a schematic diagram of the success rate in a sphere environment; and (b) is a schematic diagram of the success rate in a forest environment.

[0032] Figure 5 This is a schematic diagram comparing the average number of collisions of the present invention with other path planning methods; wherein, (a) is a schematic diagram of the average number of collisions in a sphere environment; and (b) is a schematic diagram of the average number of collisions in a forest environment. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings.

[0034] This invention is applicable to various mobile robots (including but not limited to quadcopter drones, multi-rotor drones, wheeled ground mobile robots, tracked mobile robots, and mechanical dogs) that need to autonomously avoid obstacles in unknown and complex environments, and is especially suitable for high-speed obstacle avoidance scenarios; to verify the effectiveness of the method, this invention uses a quadcopter drone as an example for illustration.

[0035] A real-time path planning method for robots based on joint guidance of depth and spatial information is proposed. The robot's real-time path planning system consists of an image acquisition module, a map building module, and a motion planning module mounted on a quadcopter UAV. The core function of the image acquisition module is to acquire depth image data. The map building module provides accurate global positioning data for the quadcopter UAV. The map building module includes a 3D LiDAR and an inertial measurement unit (IMU); the 3D LiDAR is used to construct a 3D point cloud map; the IMU is used to acquire the attitude information of the quadcopter UAV. The motion planning module determines the actual movement speed of the UAV in the 3D point cloud map based on the depth image data.

[0036] In this embodiment, the image acquisition module uses a depth camera.

[0037] like Figure 1 As shown, the real-time path planning method for this robot includes the following steps:

[0038] Step 1: Building the dataset

[0039] Single-frame depth images (640×480 pixels resolution) are acquired using a depth camera, and attitude information (including roll, pitch, yaw, and corresponding angular velocity) of the quadcopter drone is collected using an inertial measurement unit. A privileged expert strategy with obstacle perception capabilities is employed to perform obstacle avoidance tasks in various complex scenarios (including spherical obstacles and forest scenarios) at different speeds (3D vectors, corresponding to x / y / z axis motion speeds). The privileged expert strategy can obtain the position and radius information of obstacles within 10 meters of the current quadcopter drone's location. By running a short line-of-sight obstacle avoidance strategy, action commands are output to control the quadcopter drone to fly forward 60 meters at the desired speed while avoiding obstacles. Specifically, the expert strategy searches for a straight, obstacle-free trajectory from the current position to each waypoint on a lateral-vertical 2D grid plane in front of the quadcopter drone, selects the waypoint closest to the grid center, and applies thrust to its relative position to generate speed command actions. The sample data in the dataset was collected in the form of {(depth image, attitude information, desired velocity), actual velocity} supervised data pairs, with 588 expert runs generating 112k supervised data pairs. The depth images in the dataset require normalization. Because this expert strategy relies solely on reactive obstacle avoidance and lacks dynamic path planning, collisions still occur in the data. However, compared to relying on skilled quadcopter drone operators to collect data, it offers better timeliness and avoids the high-risk accidents that can occur when collecting data in the real world.

[0040] Step 2: Data Preprocessing

[0041] The depth images in the dataset are normalized, and the depth values ​​I are normalized. d Map to the [0,1] interval. Filter and denoise the attitude information (using the Kalman filter algorithm) to remove sensor noise interference. Timestamp align all data to ensure data synchronization.

[0042] Step 3: Construct a path planning model

[0043] The path planning model consists of a perception network and a decision network. The perception network transforms the depth image into a foreground scene representation coupled with spatial information, and then inputs the foreground scene representation, along with pose information and desired velocity, into the decision network to output predicted action commands.

[0044] The perception network comprises a first encoding module, a second encoding module, and a feature fusion module to accommodate the limited computing power of the onboard computer. Within the perception network, the depth image is processed sequentially by the first and second encoding modules. The feature fusion module then fuses the output feature maps from the first and second encoding modules to obtain a multi-scale fusion feature (pre-scene representation) guided by depth-spatial information, which serves as the output of the perception network. This representation integrates global spatial information and depth threat perception, effectively extracting visual features.

[0045] like Figure 2 As shown, the first and second encoding modules have the same structure, both including a downsampling layer and two feature extraction units. The downsampling layer uses convolution operations to reduce the spatial resolution of the input feature map, thereby increasing the channel dimension and reducing computational overhead while preserving key features. The feature extraction units include a cascaded depth-spatial information guidance module and a visual Transformer multi-head attention module.

[0046] In this embodiment, the convolution kernel sizes of the downsampling layers in the first encoding module and the second encoding module are 7×7 and 3×3, respectively, the stride sizes are 4 and 2, and the number of channels is 16→32→64 in sequence.

[0047] In the depth-spatial information guidance module, the depth-spatial guidance weight matrix is ​​used. For the input feature map Element-wise weighted scaling is performed, and the weighted feature map is residually concatenated with the input feature map to obtain the output feature map of the depth-spatial information guidance module. It is represented as:

[0048]

[0049] in, This is the adjustment factor (default is 1.0). This indicates element-wise multiplication, ensuring that the original feature details are not lost.

[0050] Depth-space guided weight matrix Based on deep guided weight matrix and spatial guided weight matrix Constructing and visualizing heatmaps with different weight matrices, as shown below. Figure 3 As shown. The depth-guided weight matrix, constructed based on depth image information, quantifies the distance correlation between obstacles and the quadcopter drone; a depth-guided weight matrix negatively correlated with depth values ​​is constructed. This approach assigns high weights to feature regions corresponding to nearby obstacles and low weights to distant obstacles, while background regions are hard-coded to have a weight of 0, thus achieving differentiated feature weight allocation. (Depth-guided weight matrix) The construction method is as follows:

[0051]

[0052] in, The depth value for each pixel; This is the error compensation coefficient. .

[0053] The spatial guidance weight matrix is ​​generated based on the two-dimensional spatial position information of obstacles relative to the quadcopter drone. It can accurately characterize the spatial correlation between different regions in the image and obstacle avoidance decisions. Its core function is to filter key obstacle avoidance regions from the "position dimension," and it works in conjunction with the depth guidance weight matrix to achieve a dual enhancement of obstacle avoidance-related features. The construction method is as follows:

[0054] Where x and y are the horizontal coordinate (width direction) and vertical coordinate (height direction) of each pixel, respectively. and , where x and y are the x and y coordinates of the center point of the Gaussian distribution, respectively; and These are the standard deviations of the Gaussian distribution in the width and height directions, respectively.

[0055] Based on deep guided weight matrix and spatial guided weight matrix Constructed depth-space guided weight matrix Represented as:

[0056]

[0057] in, These are the weighting coefficients. .

[0058] In the multi-head attention module of the Visual Transformer, the input feature map is... Split into There are 1 single-pixel feature blocks, each feature block having a dimension of 1. The feature blocks are mapped to query (Q), key (K), and value (V) through three independent linear layers, respectively, where the Q dimension is preserved. After downsampling, K and V have the following dimensions: , where n < N; calculate the multi-head attention output based on the queries, keys, and values corresponding to each feature block, fuse the multi-scale feature correlation information to obtain the attention output; perform a non-linear transformation on the attention output through a hybrid feed-forward layer with a "linear + convolution + activation + linear" structure (convolution kernel size of 3×3) to enhance the feature expression ability, obtain the output feature map of the visual Transformer multi-head attention module, and use it as the output result of the feature extraction unit.

[0059] In the feature fusion module, perform pixel reorganization operations on the output feature map of the second encoding module for upsampling to unify the resolution of the output feature map of the second encoding module and the output feature map of the first encoding module; after channel concatenation of the output feature map of the first encoding module and the output feature map of the second encoding module, perform dimension fusion through a 3×3 convolution to achieve multi-scale feature complementarity.

[0060] In the decision network, flatten and vector concatenate the multi-scale fusion features, pose information, and desired speed and then input them into a three-layer cascaded fully connected layer; the number of neurons in the three-layer fully connected layer decreases sequentially, and the output dimension is adaptively adjusted according to the robot type. In the UAV scenario, a 3D linear velocity command (x / y / z-axis velocities) is output, and in the ground robot scenario, a 3D command (x / y-axis velocities + steering angle) is output. The output layer selects an appropriate activation function according to different robot types to output the final action command.

[0061] Step Four: Model Training

[0062] Use the dataset to train the path planning model and adopt the mean square loss function Optimize the model parameters to minimize the error between the predicted motion parameters and the expert motion parameters, which is expressed as:

[0063]

[0064] Where is the model's predicted action command; is the expert action command; T is the number of samples.

[0065] During the training process, the model gradually optimizes the distribution strategy of the obstacle avoidance correlation weight matrix by learning the association between depth information and the obstacle avoidance actions of different types of robots, ensuring that the obstacle avoidance-related features dominate the feature representation. After training, the decision network can be fine-tuned according to the target robot type (such as UAV, ground robot).

[0066] Step Five: Motion Control

[0067] Deploy the trained path planning model to the on-board computing platform of the corresponding robot, receive sensor data in real-time and output control commands to achieve autonomous obstacle avoidance.

[0068] Step Six: Model Evaluation

[0069] To evaluate the performance of the proposed model, the Flightmare simulation environment platform, built on the Unity engine, was used. This platform architecture primarily comprises two core modules: a configurable rendering engine and a flexible physics engine. The rendering engine generates highly realistic 3D visual environments, including a forest environment and a sphere environment. The physics engine focuses on the dynamics simulation of the quadcopter. The sphere environment is similar to the environment used to construct the dataset, with 200 white spheres of different sizes randomly distributed at various locations. The forest environment is filled with 200 trees of similar shapes to evaluate the model's generalization ability. Baseline models for comparison include a convolutional neural network model (ConvNet), a U-shaped convolutional neural network model (UNet), a visual Transformer model (ViT), and a visual heat conduction model (vHeat). In both environments, different models were tested 10 times at speeds of 3m / s, 4m / s, 5m / s, 6m / s, and 7m / s, with the goal of flying 60 meters straight ahead while avoiding obstacles. Evaluation results are as follows: Figure 4 and Figure 5 As shown, in a spherical environment, except for a slightly lower success rate than the Convnet model at a speed of 6 m / s, this invention significantly outperforms other baseline models in both success rate and collision rate at other speeds. In a forest environment, as the speed increases to 4 m / s, this invention surpasses other baseline models in both success rate and collision rate. Compared to the unmodified ViT model, this method significantly improves the success rate and further reduces the collision rate by coupling depth-spatial information with scene representation to assist decision-making. This indicates that the depth-spatial information guidance module can effectively guide the model to focus on more threatening obstacles and make more accurate decisions. The experimental results fully verify the significant advantages of this invention in terms of performance and generalization ability.

[0070] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A real-time path planning method for robots based on joint guidance of depth and spatial information, characterized in that: The method includes: Construct a dataset containing depth images, pose information, and desired velocities, and then label the dataset. A path planning model is constructed; the path planning model includes a perception network and a decision network; the perception network is used to convert the depth image into a foreground scene representation, and inputs the foreground scene representation, pose information and desired velocity into the decision network for processing to obtain the predicted action command; The perception network introduces a depth-spatial information guidance module to assist in feature extraction of depth images. In the depth-spatial information guidance module, the input feature map is weighted and scaled element-wise using a depth-spatial guidance weight matrix, and the weighted feature map is residually connected to the input feature map to obtain the output feature map of the depth-spatial information guidance module. The depth-spatial guidance weight matrix is ​​constructed based on the depth guidance weight matrix and the spatial guidance weight matrix. The path planning model is trained using a dataset, and the trained model is used to predict the robot's actual movement speed. The robot's movement is then controlled based on the actual movement speed, thereby achieving real-time path planning for the robot.

2. The robot real-time path planning method based on joint guidance of depth-spatial information according to claim 1, characterized in that: The perception network includes multiple encoding modules connected in series and a feature fusion module; the multiple encoding modules process the depth image sequentially, and the feature fusion module fuses the output feature maps of each encoding module to obtain the foreground scene representation output by the perception network.

3. The robot real-time path planning method based on joint guidance of depth-spatial information according to claim 2, characterized in that: The encoding module includes a downsampling layer and multiple feature extraction units connected in sequence; The feature extraction unit includes a cascaded depth-spatial information guidance module and a visual Transformer multi-head attention module.

4. The robot real-time path planning method based on joint guidance of depth-spatial information according to claim 2, characterized in that: In the feature fusion module, pixel recombination is used to upsample the output feature map except for the first encoding module; after channel concatenation of the output feature maps of each encoding module, dimensionality fusion is performed through a convolutional layer to obtain the output feature map of the feature fusion module.

5. The robot real-time path planning method based on joint guidance of depth-spatial information according to claim 1, characterized in that: The depth guidance weight matrix is ​​constructed based on depth image information; the spatial guidance weight matrix is ​​generated based on the two-dimensional spatial position information of the obstacle relative to the robot.

6. The robot real-time path planning method based on joint guidance of depth-spatial information according to claim 1, characterized in that: The depth-space guided weight matrix The construction method is as follows: in, These are the weighting coefficients; The depth value for each pixel; x and y are the horizontal and vertical coordinates of each pixel, respectively; and and represent the x and y coordinates of the center point of the Gaussian distribution, respectively; and These are the standard deviations of the Gaussian distribution in the width and height directions, respectively. This is the error compensation coefficient.

7. The robot real-time path planning method based on joint guidance of depth-spatial information according to claim 1, characterized in that: In the Visual Transformer multi-head attention module, the input feature map is split into multiple feature blocks, and each feature block is mapped to a query, key, and value through an independent linear layer. After downsampling the key and value, the multi-head attention is calculated by jointly querying the query and then fused to obtain the attention output. The attention output is then nonlinearly transformed by a hybrid feedforward layer to obtain the output feature map of the Visual Transformer multi-head attention module.

8. The real-time path planning method for robots based on joint guidance of depth-spatial information according to claim 1, characterized in that: In the decision network, the foreground scene representation, pose information, and desired velocity are flattened and vectorized before being input into a multi-layer cascaded fully connected layer to obtain the output of the decision network.

9. A real-time path planning method for robots based on joint guidance of depth-spatial information according to claim 1, characterized in that: The path planning model is trained using the preprocessed dataset; the preprocessing process involves normalizing the depth images in the dataset and converting the depth values ​​I... d Map to the [0,1] interval; use Kalman filtering algorithm to filter and denoise the attitude information; align all data with timestamps.

10. A real-time path planning system for robots based on joint guidance of depth and spatial information, characterized in that: This system is used to execute a robot real-time path planning method based on joint guidance of depth and spatial information as described in claim 1. The robot real-time path planning system includes an image acquisition module, a map construction module, and a motion planning module. The image acquisition module is used to acquire depth image data. The map construction module is used to construct a three-dimensional point cloud map and obtain the robot's posture information. The motion planning module is used to determine the robot's actual movement speed in the three-dimensional point cloud map based on the depth image, posture information, and desired speed.

Images