Improved RRT path planning method and device based on deep reinforcement learning

Abstract

The invention provides an improved RRT path planning method and device based on deep reinforcement learning, which are suitable for efficient barrier-free path planning of an air-ground cross-domain robot in a complex ground and air environment so as to ensure that the robot safely and quickly arrives at a target position. The method comprises the following steps: constructing a deep reinforcement learning model, and designing a distance heuristic exploration mechanism in combination with an epsilon greedy strategy, so that an algorithm can adaptively adjust a sampling strategy; a target bias factor and a dynamic sampling space adjustment strategy are introduced, and randomness and redundant node generation in the expansion process are reduced; and path simplification and smooth optimization processing are adopted, so that the safety of the generated path and the performability of the robot are improved. Compared with the traditional RRT and RRTConnect algorithms on the key indexes such as planning time, path length and node number, the method shows higher planning efficiency, path quality and environment robustness, and can better meet the application requirements of the path planning task of the air-ground amphibious robot.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and more particularly, to an improved RRT path planning method and device based on deep reinforcement learning. BACKGROUND

[0002] Modern post-disaster urban environments (such as after earthquakes, floods) often present extremely complex terrains such as road damage, building collapse, and blocked passages, which pose a serious challenge to the adaptability of traditional single motion mode robots. New land-air cross-domain robots have attracted attention due to their rich application scenarios and strong terrain adaptability. Compared to traditional robots that only support a single motion mode, new land-air cross-domain robots have a significant advantage in comprehensive performance in complex post-disaster urban environments.

[0003] Path planning is a core technology for intelligent agents such as mobile robots and unmanned aerial vehicles to achieve autonomous navigation, and its goal is to plan a collision-free path from the starting point to the end point under constraints. The Rapidly-exploring Random Tree (RRT) algorithm is widely used because it does not require environment preprocessing and is suitable for high-dimensional spaces. However, the traditional RRT algorithm relies on uniform random sampling of the entire space, which has slow convergence speed, high path redundancy, and low success rate in complex environments. To improve the RRT algorithm, researchers have proposed RRT, RRT-Connect, Informed-RRT, and other algorithms. RRT gradually optimizes the path through rewiring but converges slowly; RRT-Connect uses bidirectional growth to accelerate connection but the path quality is unstable; Informed-RRT limits the sampling area to improve efficiency but has difficulty finding the initial path. These improved algorithms have some improvements in specific aspects, but they generally have weak environmental adaptability and difficulty in balancing path planning time and path quality.

[0004] In recent years, deep reinforcement learning (DRL) has provided a new approach to path planning. Existing research has attempted to combine DRL with RRT, such as using DRL's experience replay mechanism to enhance exploration ability, or dynamically adjusting RRT parameters through a DRL model. However, existing fusion methods still have obvious shortcomings: lack of a complete system framework to optimize sampling strategy, guidance mechanism, and search space simultaneously; unable to deeply couple DRL decision-making with RRT random exploration to balance exploration and utilization; still lacking in environmental adaptive adjustment and path smoothness and safety guarantees. In addition, when facing land-air cross-domain robots with multiple motion modalities, traditional and existing improved algorithms are difficult to effectively adapt to their structural characteristics and motion constraints. Such robots typically have a wheel-leg-wing structure and can achieve multiple motion modalities such as crawling or sliding and flying. Such robots have higher requirements for path planning algorithms in terms of three-dimensional space adaptability, dynamic constraint handling, and multi-modal trajectory optimization during motion. SUMMARY

[0005] The present application aims at the technical problems existing in the prior art, and provides an improved RRT path planning method and device based on deep reinforcement learning, which can effectively improve the robot path planning efficiency and the self-adaptive ability in complex environment by designing and developing the path planning method.

[0006] According to a first aspect of the present application, an improved RRT path planning method based on deep reinforcement learning is provided, comprising the following steps: Obtaining robot path planning environment information, defining a starting point and a target point, and setting maximum iteration number, expansion step, discrete step and safety distance parameters; Initializing an RRT tree structure, taking the starting point as a root node, constructing a deep reinforcement learning model, defining a state space, an action space and a reward function; An improved ε-greedy strategy combining distance heuristic information and dynamic safety constraints is used to select actions based on the current state and guide RRT expansion; Performing the expansion action and interacting with the environment, obtaining the next state and reward, and judging whether the target is reached or the obstacle is collided; The state transition data is stored in an experience replay buffer, and when the data amount in the buffer reaches a set threshold, a small batch of transition data is randomly sampled from the buffer to train the deep learning network; Repeating the expansion action until the maximum iteration number is reached or the path is successfully planned, and sequentially performing redundant node elimination and trajectory smoothing processing on the planned path, using the trained greedy strategy for final path planning, and outputting the optimal trajectory that is smooth and meets the multi-modal motion constraints of the robot.

[0007] On the basis of the above technical solution, the present application can also be improved as follows.

[0008] Optionally, the path planning environment information includes the three-dimensional map size of the ground and the air space, the position and shape of the obstacle; the definition of the starting point and the target point, and the setting of the maximum iteration number, the expansion step, the discrete step and the safety distance parameters include: The environment map size is obtained by reading an image file, the obstacle boundary is extracted by image processing, the obstacle coordinate set is obtained by using the Canny edge detection algorithm, the discrete step is controlled by setting parameters, and the robot radius, the obstacle safety radius and the maximum expansion distance are set.

[0009] Optionally, the state space contains the motion mode information of the robot currently in, and the motion mode includes a four-legged crawling mode, a four-wheel sliding mode and a four-rotor flight mode.

[0010] Optionally, the ε-greedy strategy adaptively adjusts the exploration probability according to the robot's current motion pattern, using a higher exploration rate in flight mode to adapt to three-dimensional space search, and a lower exploration rate in crawling mode to improve ground path planning efficiency; the action selection mechanism includes randomly selecting actions with ε probability and selecting the action with the largest Q value output by the network with 1-ε probability.

[0011] Optionally, the step of selecting an action based on the current state to guide the RRT extension includes: RRT is extended by calling the fun_rrt or fun_rrt_star function. The extension process takes into account the motion constraints of the robot's wheel-leg-wing structure and adopts different extension step sizes in different motion modes, including: using a large step size in flight mode to improve search efficiency, and using a small step size in crawling mode to adapt to complex terrain. Calculate the distance from the expanded new node to the target point, and obtain the location of the expanded new node and the tree structure update information.

[0012] Optionally, a target bias factor and a dynamic sampling space strategy are introduced in the environmental interaction. Random numbers are generated in each iteration, and the deep reinforcement learning model outputs an adaptive threshold based on the current state. If the random number is less than the adaptive threshold, the sampling point is set as the target point; otherwise, global random sampling is performed.

[0013] Optionally, the dynamic sampling space strategy is adjusted in real time according to the motion constraints of the robot's wheel-leg-wing integrated structure. A dynamically adjusted rectangular sampling area is constructed with the newly added node of the random tree as the center. The length and height of the rectangular sampling area are adaptively adjusted according to the density of environmental obstacles: the sampling space size is increased in areas with sparse obstacles to accelerate exploration, and the sampling range is reduced in areas with dense obstacles or complex structures to enhance the local search precision.

[0014] Optionally, the repeated execution of the extension action until the maximum number of iterations is reached or a path is successfully planned includes: Set the maximum number of training rounds and the maximum number of steps per round. Record the cumulative reward and historical best path for each round in real time. Record the reward curve and historical best path. When the path is successfully planned for several consecutive rounds and the reward curve converges, terminate the training early.

[0015] Optionally, the step of sequentially removing redundant nodes and smoothing the trajectory of the planned path includes: The exploration probability parameter ε=0 is set, and a pure greedy strategy is used to select actions. Starting from the end node, the movement is traced backward along the parent pointer to the starting point. A continuous and smooth motion trajectory is generated using a curve fitting method. The final output includes performance metrics such as path length, planning time, and number of nodes.

[0016] According to a second aspect of the present invention, a land-air cross-domain robot is provided, applied to the improved RRT path planning method based on deep reinforcement learning, the robot comprising: an integrated wheel-leg-wing structure, a torso connector, a sensing and detection module, and a motion control module; wherein, The wheel leg The integrated wing structure is used to achieve the transformation between three modes: crawling, rollerblading, and flight. The torso connector is used to connect and position the various structural components; The sensing and detection module is used to perceive the robot's motion state and environmental information; The motion control module integrates wireless communication functionality to receive control commands and coordinate the actions of various actuators, thereby enabling the control and switching of multimodal motion.

[0017] The technical effects and advantages of this invention are as follows: This invention provides an improved RRT path planning method and apparatus based on deep reinforcement learning. By guiding the RRT expansion direction through deep reinforcement learning and combining a distance-heuristic exploration strategy, a target bias factor, and a dynamic sampling space mechanism, it significantly improves sampling efficiency and path quality, reduces the generation of invalid nodes, and accelerates convergence speed. It is particularly suitable for the multi-morphic motion requirements of land-air cross-domain robots. The path planning fully considers the motion characteristics of the wheel-leg-wing integrated structure and provides optimal path planning for different motion modes. Attached Figure Description

[0018] Figure 1 A flowchart illustrating an improved RRT path planning method based on deep reinforcement learning, provided for an embodiment of the present invention; Figure 2 The reward curve of the DRL-RRT algorithm provided in this embodiment of the invention; Figure 3 A schematic diagram illustrating the path planning of the DRL-RRT algorithm and its comparison algorithm in a narrow channel, as provided in an embodiment of the present invention. Figure 4 This is a schematic diagram illustrating the path planning of the DRL-RRT algorithm and its comparison algorithm in a simple two-dimensional environment, as provided in an embodiment of the present invention. Figure 5 A schematic diagram illustrating path planning in a two-dimensional complex environment using the DRL-RRT algorithm and its comparison algorithm provided in this embodiment of the invention; Figure 6 A schematic diagram illustrating path planning in a three-dimensional environment for the DRL-RRT algorithm and its comparison algorithm provided in this embodiment of the invention; Figure 7A comparison chart of path length simulation results between the DRL-RRT algorithm and the comparison algorithm provided in this embodiment of the invention; Figure 8 This is a comparison chart of the simulation results of the planning time of the DRL-RRT algorithm and the comparison algorithm provided in the embodiments of the present invention; Figure 9 This is a comparison chart of the simulation results of the number of nodes of the DRL-RRT algorithm and the comparison algorithm provided in the embodiments of the present invention; Figure 10 This is a schematic diagram of the overall structure of the land-air cross-domain robot provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of a single-leg structure of a land-air cross-domain robot provided in an embodiment of the present invention.

[0019] The attached diagram lists the components represented by each number as follows: 1. Wheel leg 1. Integrated wing structure; 2. Torso connector; 3. Motion control module; 4. Sensor detection module; 5. Long support; 6. Short support; 7. Brushless motor; 8. Rotor; 9. Servo 1; 10. Servo 2; 11. Encoder motor; 12. Servo 3; 13. Swing arm; 14. Pinion; 15. Lower leg; 16. Fixture; 17. Roller. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Understandably, given the shortcomings in the background technology, this invention proposes an improved RRT path planning method based on deep reinforcement learning, specifically as follows: Figure 1 As shown, it includes the following steps: S1. Obtain robot path planning environment information, define the starting point and target point, and set parameters such as maximum number of iterations, extended step size, step size and safety distance. In this embodiment, the path planning environment information for the land-air cross-domain robot includes: the dimensions of the three-dimensional map of the ground and airspace, and the location and shape of obstacles. The robot's starting point and target point are defined, and parameters such as the maximum number of iterations, extended step size, step size, and safe distance are set based on the robot's physical dimensions and mobility.

[0022] The specific operations for defining the robot's starting point and target point are as follows: The map size is obtained by reading the environment image file, specifically P.size. Obstacle information is obtained by extracting obstacle boundaries through image processing and using the Canny edge detection algorithm to extract obstacle coordinates, resulting in the obstacle coordinate set P.obstales.cor. The starting point of the amphibious robot is defined as P.air.start=[18,8], and the target point is defined as P.air.end=[193,195].

[0023] Based on the robot's physical dimensions and mobility, the following settings are provided: Set the maximum number of iterations P.iterations=100000 and the step size P.dx=3, and control the step size using the step_size parameter; set the robot radius P.air.r=2, the obstacle safety radius P.obstales.r=3, and the maximum extension distance P.dmax=P.obstales.r·2=6.

[0024] S2. Initialize the RRT tree structure, take the starting point as the root node, construct the deep reinforcement learning model, and define the state space and reward function. The initialization of the RRT tree structure includes: The `initial_rrt(P.air.r, P.air.start, P.air.end)` function initializes the Quick Explore Random Tree (RRT) structure, which contains the following fields: Node coordinate matrix rrt.tree: A matrix that stores the coordinates of all nodes, with the starting point being the first node; Parent node index rrt.parent: Stores the parent node index of each node, with the root node's parent node index being 0; The distance to the target point, rrt.d2aim, is the distance from the current node to the target point, which serves as the state input for reinforcement learning. The set of expansion directions is defined as P.direct, defining the possible expansion angles; the state space is defined as the normalized distance from the current node to the target point, with state dimension P.dim = 1, and state values ​​are represented as follows:

[0025] in, This is the distance normalization factor, i.e., a preset maximum possible distance value; rrt.d2aim is the distance to the target point.

[0026] The construction of the deep reinforcement learning model includes the following steps: State space design: State s is defined as the current state (position coordinates) of the agent. Indicates the action of selection. Indicates the location of the target point. Let represent the set of obstacles. The mathematical expression of the state transition function is as follows:

[0027] in, It is an action The corresponding movement vector, It is a free state space. The following two constraints must be met simultaneously: Boundary constraints: and obstacle constraints: ,in, Indicates the mesh environment at the 1st Dimensions.

[0028] Reward function design: The reward function provides learning signals to the agent, guiding it to move towards the target direction.

[0029] in, It is a large positive reward upon reaching the goal. It is a negative Euclidean distance, meaning the farther away from the target, the greater the penalty. In practice, successfully reaching the target grants a large positive reward of 10,000, while in other cases the reward is a negative normalized distance, to encourage the agent to move closer to the target.

[0030] Termination condition: Determines whether the current round has ended.

[0031] The neural network adopts a multilayer perceptron structure, including an input layer (1 node), a hidden layer (32 nodes, ReLU activation), a hidden layer (16 nodes, ReLU activation), and an output layer (5 nodes, Sigmoid activation). The network parameters are initialized using the Xavier initialization method.

[0032] S3. An improved ε-greedy strategy that integrates distance heuristic information and dynamic security constraints is adopted to select actions based on the current state and guide the RRT expansion. In this embodiment, an improved ε-greedy strategy that integrates distance heuristic information and dynamic safety constraints is adopted, wherein the exploration strategy is adaptively adjusted according to the robot's current motion pattern.

[0033] The ε-greedy strategy adaptively adjusts the exploration probability based on the robot's current motion pattern. A higher exploration rate is used in flight mode to adapt to 3D spatial search, while a lower exploration rate is used in crawling mode to improve ground path planning efficiency. The action selection mechanism includes randomly selecting actions with ε probability and selecting the action with the maximum Q-value output by the network with 1-ε probability. The deep Q-network inference process includes state preprocessing, forward propagation, and action selection. The action selection formula is:

[0034] In the formula, Q(s,a) is the action value output by the neural network; collision detection is combined to filter possible actions to ensure the safety of the expansion direction.

[0035] In each iteration, the agent selects an action based on the current state. An improved ε-greedy strategy is employed, and the action space consists of N predefined movement directions, denoted as . Each action Corresponding to a movement vector The improved ε-greedy strategy is mathematically expressed as:

[0036] in, Indicates the probability of exploration. This refers to the action with the greatest value.

[0037] The step of selecting an action based on the current state to guide rapid exploration of random tree expansion includes: Set initial exploration rate =1, attenuation coefficient -Decay=0.99, minimum exploration -Min=0.1, updates every round: =max( · -Decay, -Min).

[0038] Exploration Phase (Probability ε): A biased exploration based on distance heuristics is adopted, instead of completely random selection. For each action... Calculate heuristic weights:

[0039] in, It is execution The expected Euclidean distance from the new state to the target point is... It is a collision detection function (safety is 1, collision is 0). To prevent division by zero constant, this case takes... Then, actions are selected using a roulette wheel based on weights.

[0040]

[0041] Utilization phase (probability 1-ε): Calculate the adjusted value of each action:

[0042] Here, λ is the weighting coefficient, which ultimately selects the action with the highest adjusted value. The weighting coefficient λ is set to 0.5 here.

[0043] S4. Execute the extended strategy and interact with the environment to obtain the next state and reward, and determine whether the target has been reached or an obstacle has been encountered. In this embodiment, a target bias factor and a dynamic sampling space strategy are introduced during the environmental interaction process. The dynamic sampling space strategy is adjusted in real time according to the motion constraints of the robot's wheel-leg-wing integrated structure.

[0044] The execution of the extended strategy and interaction with the environment includes: (1) Target bias: Generate random numbers in each iteration

[0045] Random points are generated according to the following formula. :

[0046] Among them, the bias threshold Dynamically generated by a deep reinforcement learning network based on the current environmental state, The probability of resetting the sampling point to the starting point guides the tree to grow in the target direction. These are random points sampled uniformly throughout the entire space.

[0047] In practice, random numbers are generated in each iteration. Deep reinforcement learning (DRL) networks output adaptive thresholds based on the current state (obstacle density, etc.). ,like Then the sampling points Set as target point Otherwise, random sampling is performed globally.

[0048] (2) Dynamic sampling space: The environmental status includes the obstacle density, obstacle orientation distribution, and number of expanded nodes in the neighborhood of the current sampling point.

[0049] The above is a new node added to the random tree. Centered around this, a dynamically adjustable rectangular sampling area is constructed. The length and height of the rectangular sampling area are adaptively adjusted according to the environmental obstacle density: the sampling space size is increased in sparse obstacle areas to accelerate exploration, and the sampling range is reduced in dense or complex-structured obstacle areas to enhance the local search fineness.

[0050] The size of the sampling area is adaptively adjusted according to the obstacle density:

[0051] where L and W are the length and width of the sampling area respectively, 、 is the reference size, is the obstacle density, is the adjustment coefficient. In specific operations, the length , width , when the obstacles are sparse ( is small), the area becomes larger to accelerate exploration; when the obstacles are dense ( is large), the area becomes smaller for fine search.

[0052] State transition and termination judgment: The extension is executed through the fun_rrt function to calculate the new state. Collision detection is performed through the judge_ok and judge_ok_2 functions. If the target area is reached (distance < δ) or a collision occurs, the round terminates.

[0053] S5. Store the state transition data in the experience replay buffer. When the data volume in the buffer reaches the set threshold, randomly sample a small batch of transition data from the buffer to train the deep learning network; In this embodiment, the state transition data (state, action, reward, nextState, done) is stored in the experience replay buffer. The buffer capacity replayBufferSize = 1000, and a circular queue is used for management. To improve the data quality, only store when nextState < state (i.e., the state improves). The action uses one-hot encoding.

[0054] The buffer configuration sets the capacity to replayBufferSize = 1000; the data structure includes fields such as states, actions, rewards, nextStates, and dones; the data storage strategy adopts selective storage, and only stores the experience when the state improves, that is, stores when nextState < state; the action encoding uses one-hot encoding with a dimension of P.dN; it supports pre-filling the buffer with initial trajectory data.

[0055] Training begins when the amount of data in the buffer, replayBuffer.count, is greater than or equal to batchSize (batchSize=128). Prioritized experience replay: Calculate the priority of each experience. ,in For TD error. Sampling probability:

[0056] In the formula, This indicates the priority of all experiences in the experience replay buffer. The sum of α powers, specifically, Indicates the first A single experience, i.e., a single state transition data point. Indicates the first The priority of the experience points, α controls the strength of the priority, which is 0.5 in this case.

[0057] Network training: Randomly sample a small batch of transition data from the buffer and calculate the target Q-value. The network parameters are updated using stochastic gradient descent with momentum as the loss function: momentum coefficient = 0.9, learning rate = 1e. -3 .

[0058] The training trigger condition is set to replayBuffer.count ≥ batchSize, and the batch size is batchSize = 128. The sampling strategy includes random sampling and priority experience replay. Priority experience replay uses TD-error to calculate the sampling weights, and the weight formula is:

[0059] Where δ is the TD-error and α is a hyperparameter with a value of 0.5; Network training parameters: learning rate = 1e -3 The momentum factor is 0.9; the training process includes data normalization, loss function calculation, and gradient update, with the loss function being the mean squared error.

[0060] The optimization algorithm employs stochastic gradient descent with momentum:

[0061] Where μ is the momentum coefficient and η is the learning rate. This is the velocity vector.

[0062] S6. Repeat steps S3 to S5 until the maximum number of iterations is reached or a path is successfully planned. Redundant nodes are removed and the trajectory is smoothed sequentially on the planned path. The trained greedy strategy is used for the final path planning, and the optimal trajectory that is smooth and meets the robot's multi-morphic motion constraints is output.

[0063] In this embodiment, the maximum number of training rounds (numEpisodes) is set to 1000, and the maximum number of steps per round (maxSteps) is set to P.iterations. The cumulative reward (episodeRewards) and the historical best path (best.rrt) for each round are recorded in real time. Performance monitoring records the reward curve and the historical best path; training can be terminated early when multiple rounds of successful path planning and the reward curve converge.

[0064] The process of sequentially removing redundant nodes and smoothing the trajectory of the planned path includes: During the testing phase, the exploration probability parameter ε=0 is set, and a pure greedy strategy is used to select actions. Path backtracking starts from the end node and traces backward along the parent pointer to the starting point. Path smoothing is achieved by curve fitting to generate a continuous and smooth motion trajectory, ensuring that the trajectory meets the dynamic constraints of the robot in different motion modes, especially the smooth transition of the wheel-leg-wing structure at the form switching point. The final output includes performance indicators such as path length, planning time, and number of nodes.

[0065] The specific steps are as follows: After training is complete, set the exploration probability parameter. The final plan is made using a purely greedy strategy. Path backtracking: Using the `backward` function, starting from the endpoint node, backtrack along the parent pointer to the starting point to obtain the initial path.

[0066] Redundant node removal: from Begin by checking in sequence. and Check if the connections are collision-free. If CollisionFree( , If )==True, then delete all intermediate nodes. Repeat this process until it cannot be simplified further.

[0067] Trajectory smoothing: Cubic non-uniform rational B-spline curves are used.

[0068] Adaptive control point generation: using three consecutive path points As a set of processing units, calculate the unit vector of adjacent path segments. and its included angle Based on path segment length The minimum value determines the reference length. Five control points are generated accordingly:

[0069] After curve fitting, if the curvature exceeds the motion limit of the amphibious robot by 0.2 m... -1 Then the control point spacing will be automatically adjusted to reduce the curvature.

[0070] Output: The final output is the smoothed path, and the performance metrics are calculated: path length, total planning time and number of nodes.

[0071] In summary, the improved RRT path planning method based on deep reinforcement learning described in this embodiment of the invention systematically improves the performance of path planning by constructing a complete framework that integrates deep reinforcement learning and RRT algorithms, and through three core improvement strategies and two path optimization strategies.

[0072] Specific examples are as follows: A series of simulation environments were built based on the Matlab platform, with a 2D map size of 100 m × 100 m and a 3D environment size of 100 m × 100 m × 100 m. RRT (Real-Time Reduction) was tested in four typical test scenarios: narrow passage, simple 2D scene, complex 2D scene, and 3D space. A systematic comparison of four algorithms—RRT-Connect, DRL-RRT, and others—was conducted. Each experiment was run ten times independently, and the arithmetic mean of the results was used as the final performance metric. Through comprehensive analysis of three commonly used evaluation metrics—path length, planning time, and number of generated nodes—the path planning performance of each algorithm under the same conditions was fully assessed.

[0073] To verify the superior performance of the proposed algorithm, an experimental scenario was designed using Matlab. The simulation experiment aimed to compare and test the time and path length of each algorithm in the same environment. Table 1 shows the network parameter settings for the deep reinforcement learning algorithm.

[0074] Table 1 Network parameter settings for deep reinforcement learning algorithms

[0075] Figure 2The graph shows the reward curves of the DRL-RRT algorithm after training in the constructed narrow channel, simple 2D environment, complex 2D environment, and 3D environment. In the graph, the purple curve is the smoothed reward curve after mean smoothing, which better reflects the algorithm's training trend. The smoothed reward curve reaches a certain value quickly and fluctuates around it, demonstrating the convergence in different scenarios and the ability to quickly search for near-optimal strategies and reach a stable state. The simulation-generated path results are shown below. Figures 3-6 As shown, the path length results are compared for reference. Figure 7 As shown, the comparison of planning time results is as follows: Figure 8 As shown, the comparison of node count results is as follows: Figure 9 As shown in the figure, the DRL-RRT improvement method used in this embodiment of the invention outperforms the traditional RRT in key indicators such as planning time, path length, number of nodes, and path smoothness. and RRT The Connect algorithm demonstrates higher planning efficiency, path quality, and environmental robustness, making it suitable for path planning tasks of amphibious robots.

[0076] RRT, Table 2 shows the simulation data of four algorithms—RRT-Connect, DRL-RRT, and RRT-Connect—in narrow channels, simple two-dimensional environments, complex two-dimensional environments, and three-dimensional environments. The simulation data is obtained through RRT, RRT-Connect, DRL-RRT, and DRL-Connect algorithms in narrow channels, simple two-dimensional environments, complex two-dimensional environments, and three-dimensional environments. A comparison of simulation results of four algorithms—RRT-Connect, DRL-RRT, and RRT-Connect—in narrow passages, simple environments, complex environments, and three-dimensional environments reveals that the adaptive ε-decay strategy balances exploration and utilization. It intelligently adjusts the exploration probability based on the robot's current motion form, ensuring that the algorithm fully explores in the early stages of training and converges stably in the later stages. Through redundant node removal and B-spline curve smoothing, it generates concise, safe trajectories that conform to the motion constraints of novel land-air cross-domain robots. It also takes into account the smooth transition requirements of shape switching points, improving the feasibility and practicality of the path.

[0077] Table 2 Simulation results of the four algorithms

[0078] In summary, the improved RRT path planning method based on deep reinforcement learning described in this embodiment of the invention takes into account the characteristics of the three motion modes of the robot. By constructing a complete deep reinforcement learning model and a neural network update strategy, it enables the robot to adaptively learn the optimal exploration strategy in each motion mode, ensuring the learning efficiency and stability of the algorithm.

[0079] Furthermore, this invention also provides a land-air cross-domain robot applied to the aforementioned improved RRT path planning method based on deep reinforcement learning. The robot includes: an integrated wheel-leg-wing structure 1, a torso connector 2, a sensing and detection module 4, and a motion control module 3; wherein, The wheel leg The wing-integrated structure 1 is used to achieve the transformation between three modes: crawling, roller skating, and flight. The torso connector 2 is used to connect and position the various structural components; The sensing and detection module 4 is used to sense the robot's motion state and environmental information; The motion control module 3 integrates wireless communication functionality, which is used to receive control commands and coordinate the actions of each actuator to achieve control and switching of multimodal motion.

[0080] The aforementioned land-air cross-domain robot, through its compact integrated structure and precise transmission mechanism, combined with the coordinated control of multiple servos and motors, enables the robot to quickly and smoothly switch between three motion modes: crawling, roller skating, and flying, and ensures that it maintains its flexibility and stability in complex cross-domain environments.

[0081] Specifically, the motion control module 3 uses a Raspberry Pi 4B as its core processor. The motion control module 3 works in conjunction with the servo control board and the motor control board through the IIC communication protocol to achieve precise control of the robot's crawling, roller skating and flying multimodal motions. The robot integrates a remote control module, which can send commands through a host computer or remote controller, and use a data transmission module to send the internal and external sensor data collected by the flight control board back to the host computer to achieve remote monitoring of the robot's motion status and environmental information.

[0082] The motion control module 3 includes a hardware control module and a path planning algorithm. The hardware control module integrates a communication unit for receiving instructions and coordinating the actions of each actuator.

[0083] like Figure 10 As shown, the land-air cross-domain robot includes wheels. leg The structure comprises an integrated wing structure 1, a torso connector 2, a motion control module 3, and various sensing and detection modules 4; wherein, the wheels... leg The integrated wing structure 1 includes a main support frame, a flight propulsion module, and a land motion module. The main support frame, composed of a long support frame 5 and a short support frame 6, serves as the core load-bearing frame of the overall structure. The wheels... leg The integrated wing structure 1 achieves precise control of the leg joints and wheels through the cooperation of the servo motor and the encoder motor. Stable switching of leg form; the core load-bearing frame fixes the flight propulsion module and the land motion module; the servo of the land motion module drives the lower leg structure with fixed rollers through the meshing of the swing arm and the pinion gear, thereby realizing the precise movement and control of the leg joints.

[0084] The flight propulsion module includes a brushless motor 7 and a rotor 8 driven by it, which is fixedly mounted on the main support. The brushless motor 7 of the flight propulsion module directly drives the rotor 8 to generate lift, and together with the land motion module, it forms a complete cross-domain mobility system.

[0085] The land motion module includes a servo motor 9, a servo motor 10, an encoder motor 11, a servo motor 12, a swing arm 13, a pinion 14, a lower leg 15, a fixing component 16, and a roller 17. The output shafts of the servo motors 9 and 10 are respectively connected to the swing arm 13. The swing arm 13 meshes with the pinion 14 for transmission. The pinion 14 is connected to the encoder motor 11. The encoder motor 11 and the servo motor 12 work together to drive the lower leg 15. The roller 17 is mounted at the end of the lower leg 15 via the fixing component 16. The outer ring of the roller 17 is gear-shaped, and the inner ring is connected to the center hole ring through the hub. During roller skating, the fixing component 16 and the swing arm 13 restrict the position of the roller 17, ensuring that the outer ring gear of the roller and the pinion 14 remain meshed. The rotor 8 and the roller 17 are structurally integrated to form a compact wheel-wing fusion unit. The connector 2 is used to connect and position the wheels. leg The various mechanical components in the wing-integrated structure 1 form a compact overall layout; The land motion module drives the swing arm 13 through servo motor 19 and servo motor 2 10, which is then transmitted to the encoder motor 11 via the pinion 14. The encoder motor 11 and servo motor 3 12 work together to control the movement of the lower leg 15, which ultimately drives the roller 17 at the end of the fixed part 16 to complete the crawling and sliding actions. The motion control module 3 uses the environmental information acquired by the sensing and detection module 4, combined with the path planning algorithm, to generate the optimal path in real time and control the wheel. leg The wing-integrated structure 1 executes the corresponding motion modes to ensure that the robot always maintains the best motion state in complex environments.

[0086] The sensing and detection module 4 is used to perceive the robot's own motion state and surrounding environment information in real time; through the wheels leg The wing-integrated structure 1, connector 2, motion control module 3, and sensing and detection module 4 work together to enable the robot to autonomously switch and move stably between three motion modes: crawling / H-shaped flight mode, roller skating mode, and X-shaped flight mode.

[0087] likeFigure 11 As shown, the single-leg structure of the land-air cross-domain robot integrates a rotor and a roller, with the outer ring of the roller being gear-shaped. During roller skating, the fixed component and swing arm restrict the position of the roller, ensuring that the roller and the pinion remain engaged at all times, thus guaranteeing the reliability of power transmission. The brushless motor and rotor of the flight propulsion module, along with the encoder motor and servo motor group of the land motion module, are integrated onto the main support via connectors, forming a compact, integrated layout that meets miniaturization requirements.

[0088] In summary, the land-air cross-domain robot described in this embodiment of the invention, by adopting an integrated wheel-leg-wing structure, can achieve flexible switching and stable control of three motion modes: crawling, roller skating, and flight, adapting to complex land and air cross-domain environments; through a compact integrated layout and a precision transmission mechanism, it ensures efficient coordination and reliability of multimodal motion.

[0089] The improved RRT path planning method based on deep reinforcement learning and the resulting land-air cross-domain robot described in this invention are applicable to land-air cross-domain robots with multimodal motion capabilities for efficient and barrier-free path planning in complex ground and air environments, ensuring the robot safely and quickly reaches its target location. It can be widely applied to path planning tasks in intelligent systems with multimodal motion, such as mobile robots, autonomous vehicles, drones, and especially land-air cross-domain robots, providing targeted path planning support for their integrated structures, and has significant theoretical and practical value.

[0090] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0091] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0092] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An improved RRT path planning method based on deep reinforcement learning, characterized in that, The method comprises the following steps: acquiring robot path planning environment information, defining a starting point and a target point, and setting maximum iteration number, expansion step, discrete step, and safety distance parameters; initializing an RRT tree structure, taking the starting point as a root node, constructing a deep reinforcement learning model, defining a state space, an action space, and a reward function; adopting an improved ε-greedy strategy that fuses distance heuristic information and dynamic safety constraints to select an action based on a current state and guide RRT expansion; performing an expansion action and interacting with the environment to obtain a next state and a reward, and judging whether the target is reached or an obstacle is collided with; storing state transition data in an experience replay buffer, and when the data amount in the buffer reaches a set threshold, training a deep learning network by randomly sampling a small batch of transition data from the buffer; repeatedly performing an expansion action until the maximum iteration number is reached or a path is successfully planned, sequentially performing redundant node elimination and trajectory smoothing on the planned path, using a trained greedy strategy for final path planning, and outputting an optimal trajectory that is smooth and satisfies robot multi-modal motion constraints.

2. The improved RRT path planning method based on deep reinforcement learning according to claim 1, characterized in that, The path planning environment information comprises three-dimensional map size, obstacle position, and shape of the ground and airspace; The definition of the starting point and the target point, and the setting of the maximum iteration number, the expansion step, the discrete step, and the safety distance parameters comprise: acquiring environment map size by reading an image file, extracting obstacle boundaries through image processing, obtaining an obstacle coordinate set by using a Canny edge detection algorithm, controlling the discrete step by setting parameters, setting a robot radius, an obstacle safety radius, and a maximum expansion distance.

3. The improved RRT path planning method based on deep reinforcement learning according to claim 1, wherein, The state space contains information about the motion mode of the robot currently in, and the motion mode comprises a four-legged crawling mode, a four-wheel sliding mode, and a four-rotor flight mode.

4. The improved RRT path planning method based on deep reinforcement learning according to claim 1, wherein, The ε-greedy strategy adaptively adjusts the exploration probability according to the current motion mode of the robot, adopts a higher exploration rate in the flight mode to adapt to three-dimensional space search, and adopts a lower exploration rate in the crawling mode to improve the efficiency of ground path planning; the action selection mechanism comprises randomly selecting an action with an ε probability and selecting a maximum Q value action output by the network with a 1-ε probability.

5. The improved RRT path planning method based on deep reinforcement learning according to claim 1, wherein, The action selection based on the current state and the guiding of RRT expansion comprise: performing RRT expansion by calling a function, considering the motion constraints of the robot wheel-leg-wing structure in the expansion process, and using different expansion steps in different motion modes, including using a large step in the flight mode to improve search efficiency and using a small step in the crawling mode to adapt to complex terrain; calculating the distance from the new node after expansion to the target point, obtaining the position of the new node after expansion, and updating the tree structure information.

6. The improved RRT path planning method based on deep reinforcement learning according to claim 1, wherein, In the environment interaction, a target bias factor and a dynamic sampling space strategy are introduced, a random number is generated at each iteration, the deep reinforcement learning model outputs an adaptive threshold value according to the current state, if the random number is less than the adaptive threshold value, the sampling point is set as the target point, otherwise, the sampling point is randomly sampled in the global space.

7. The improved RRT path planning method based on deep reinforcement learning according to claim 6, wherein, The dynamic sampling space strategy is adjusted in real time according to the motion constraint of the robot wheel-leg-wing integrated structure, and a dynamically adjusted rectangular sampling region is constructed above the new node added to the last random tree as the center; the length and height of the rectangular sampling region are adaptively adjusted according to the environment obstacle density, the sampling space size is increased in the sparse obstacle region to accelerate exploration, and the sampling range is reduced in the dense obstacle or complex structure region to enhance the local search precision.

8. The improved RRT path planning method based on deep reinforcement learning according to claim 1, wherein, The repeated execution of the extended action is performed until a maximum iteration number is reached or a path is successfully planned, and the maximum iteration number comprises: setting a maximum number of training rounds, a maximum number of steps per round, recording cumulative rewards and historical optimal paths for each round in real time, recording reward curves and historical optimal paths, and terminating training in advance when a path is successfully planned for a plurality of consecutive rounds and the reward curve converges.

9. The improved RRT path planning method based on deep reinforcement learning according to claim 1, wherein, The planned path is sequentially subjected to redundant node elimination and trajectory smoothing, and the redundant node elimination and trajectory smoothing comprise: setting an exploration probability parameter ε=0, selecting an action using a pure greedy strategy, tracing from the end node to the starting point in reverse along the parent pointer, generating a continuous and smooth motion trajectory using a curve fitting method, and finally outputting performance indicators including path length, planning time and node number.

10. An air-ground cross-domain robot, applied to the improved RRT path planning method based on deep reinforcement learning according to any one of claims 1 to 9, characterized in that, The robot comprises a wheel-leg-wing integrated structure, a trunk connecting member, a sensing and detection module and a motion control module, wherein, The wheels Legs The wing-integrated structure is used to realize the conversion among the three forms of the crawling state, the wheel-skating state and the flying state. The trunk connecting member is used to connect and position the structural components; The sensing and detection module is used to sense the motion state of the robot and environmental information; The motion control module is integrated with wireless communication function and is used to receive control instructions and coordinate the actions of the execution mechanisms to realize the control and switching of multi-modal motion.

Images