Snakelike robot three-dimensional path planning method based on deep reinforcement learning

Abstract

The invention belongs to the technical field of snakelike robot path planning, and discloses a snakelike robot three-dimensional path planning method based on deep reinforcement learning. According to the method, an improved A * algorithm is combined with a D3QN deep reinforcement learning model, so that the path planning efficiency and the energy consumption of the snakelike robot in a complex three-dimensional environment are optimized. Dynamic optimization in a path is realized by adopting a double-depth Q network (D3QN), communication with a control system is realized through a TCP-IP protocol, and energy consumption minimization and path selection precision improvement in robot path planning are realized based on an improved energy consumption evaluation model. According to the method, improvement is carried out on the basis of a traditional A * algorithm, the heuristic function is optimized through the D3QN model, the calculated amount in path planning in a three-dimensional space is effectively reduced, meanwhile, the adaptability and accuracy of path planning are improved, and efficient energy management and accurate path planning of the snake-shaped robot are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot path planning, and in particular relates to a three-dimensional path planning method for a snake-like robot based on deep reinforcement learning. The method optimizes the path selection and energy consumption of the snake-like robot in a complex three-dimensional environment by combining an improved A* algorithm with a D3QN model, thereby achieving efficient path planning and low-energy motion control. Background Art

[0002] With the continuous development of robotics technology, snake-like robots have shown remarkable adaptability in complex environments with their highly flexible body structure. Such robots can move efficiently in complex terrains such as jungles, deserts, and rocks, and are suitable for tasks such as search, rescue, and environmental detection. However, although snake-like robots have the potential to move in diverse environments, their path planning efficiency and energy management still have many shortcomings when facing complex three-dimensional environments.

[0003] Existing path planning methods, such as the classic A* algorithm, perform well in finding the path from the starting point to the end point, but have limitations in energy consumption, complex path selection, and three-dimensional space adaptability. Traditional algorithms are usually based on distance optimization and lack consideration of energy consumption, making it difficult to cope with the continuous and efficient operation of snake-like robots in complex environments. To solve these problems, artificial intelligence methods, especially deep reinforcement learning, have been gradually introduced in recent years, hoping to dynamically optimize path selection and energy consumption through autonomous learning. However, current deep reinforcement learning algorithms still face challenges such as high algorithm complexity and lack of real-time performance in practical applications, especially for resource-constrained embedded systems, where real-time execution of the algorithm is difficult to guarantee. Summary of the invention

[0004] The purpose of the present invention is to provide a three-dimensional path planning method for a snake-like robot based on deep reinforcement learning, so as to solve the technical problems of path planning efficiency and energy management of snake-like robots in complex three-dimensional environments. In response to the path planning needs of snake-like robots in three-dimensional space, the present invention combines the D3QN (dual deep Q network) model with the improved A* algorithm, and proposes a new energy consumption evaluation model based on the consideration of path length, angle change and vertical movement consumption. This model significantly improves the path planning efficiency of snake-like robots in complex three-dimensional environments by optimizing heuristic functions. Compared with traditional methods, the present invention can not only improve the accuracy of path planning, but also effectively reduce energy consumption, providing better technical guarantee for the long-term operation of snake-like robots in complex terrains. The specific technical scheme is as follows: A three-dimensional path planning method for a snake robot based on deep reinforcement learning is proposed. By combining an improved A* algorithm with a dual deep Q network (D3QN) model, the snake robot can achieve efficient path planning and minimize energy consumption in a complex three-dimensional environment. The method optimizes the heuristic function in the A* algorithm through the D3QN model, thereby significantly improving the efficiency of path planning. The D3QN model dynamically evaluates the energy consumption in the three-dimensional environment and provides the optimal route for the path planning of the snake robot.

[0005] Furthermore, based on the A* algorithm, the method expands the path search direction from 8 directions in two dimensions to 26 directions in three dimensions, including the basic movement directions along the x, y, and z axes and the diagonal directions between the three axes, to meet the complex three-dimensional path planning requirements.

[0006] Furthermore, the path planning method includes an energy consumption evaluation model (ECEM), which comprehensively evaluates the path energy consumption through factors such as path length, angle change and vertical movement to achieve path optimization. The ECEM model evaluates the energy consumption in different path selections to reduce the total energy consumption of the snake robot on the path.

[0007] Furthermore, the D3QN model estimates the value of the current state and the value of the future state respectively through the dual Q network to reduce the deviation in path planning. The D3QN model uses a reward function to reward each node based on the path length and energy consumption, so that the node that can achieve the shortest path and save energy to the maximum extent can be selected in path planning.

[0008] The present invention also discloses the implementation steps of a three-dimensional path planning method for a snake-like robot based on deep reinforcement learning: Step 1: Run the D3QN model, initialize the state space and action space of the snake robot, and perform preliminary path selection in a three-dimensional environment; Step 2: Evaluate the path consumption in real time through the reward function, and dynamically adjust the heuristic weights of the D3QN model; Step 3: Perform path search based on the improved A* algorithm, taking into account the path length, angle change, and energy consumption of vertical movement; Step 4: After completing the path planning, pass the optimal path to the control system of the snake robot; Step 5: Repeat steps 1-4 until the snake robot reaches the target position, realizing path planning with efficient energy management.

[0009] The three-dimensional path planning method of a snake-like robot based on deep reinforcement learning of the present invention has the following advantages: The path planning method of the present invention is suitable for snake robot applications in complex three-dimensional environments, effectively improves the accuracy and energy efficiency of path planning, and is suitable for long-term continuous operation. The present invention optimizes the heuristic function through a dual deep Q network (D3QN) model, significantly reduces energy consumption, and enables the snake robot to achieve more economical and efficient path selection in three-dimensional space. The present invention realizes a comprehensive evaluation of factors such as path length, angle change, and vertical movement through an energy consumption evaluation model (ECEM), effectively improving the adaptability and accuracy of path planning. The path planning method of the present invention is simple in equipment, suitable for a variety of embedded systems, and can be quickly deployed and applied in complex three-dimensional environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is the overall system deployment diagram of the present invention; Figure 2 This is the overall structure diagram of the improved A* algorithm based on deep reinforcement learning of the present invention; Figure 3 The improved A* algorithm path planning diagram of the present invention; Figure 4 A traditional A* algorithm path planning diagram for comparison of the present invention; Figure 5 Improved A* path planning graph for the weight of the control of the present invention; Figure 6 A bidirectional A* path planning diagram for comparison of the present invention; Figure 7 A comparison diagram of energy consumption between the improved A* algorithm of the present invention and various algorithms; DETAILED DESCRIPTION

[0011] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a three-dimensional path planning method for a snake-like robot based on deep reinforcement learning in conjunction with the accompanying drawings.

[0012] like Figure 1As shown, the three-dimensional path planning method of the snake robot based on deep reinforcement learning of the present invention first captures the global information of the three-dimensional environment through a camera and transmits the information to the control system. The control system runs the D3QN deep reinforcement learning model to dynamically optimize the path of the snake robot in a complex three-dimensional environment. The D3QN model learns the characteristics of the three-dimensional environment by analyzing factors such as the path length, angle change and vertical movement in the environment, adjusts the heuristic function weight of the A* algorithm in real time, and generates an energy consumption evaluation model (ECEM) for the optimal path. The improved A* algorithm combines the energy evaluation results of ECEM into path planning to select the energy-optimal path for the snake robot. Finally, the planning results are transmitted to the control system inside the snake robot through the TCP-IP protocol. The control system parses the position and energy consumption data of each path point and issues motion instructions to ensure that the snake robot achieves efficient path planning and low-energy path tracking in a three-dimensional environment.

[0013] The energy consumption evaluation model (ECEM) calculates the total energy consumption through path length, angle change and vertical movement consumption. The path length weight, angle weight and vertical movement weight are adjusted according to specific application requirements to balance the energy consumption of different path selections. The energy consumption calculation formula of the ECEM is: path_energy=path length consumption×path length weight+vertical movement consumption×vertical movement weight+angle change consumption×angle weight, so as to adjust according to different path planning requirements to minimize energy consumption.

[0014] The deep reinforcement learning model (D3QN) dynamically adjusts the heuristic function weights through the reward function, and dynamically evaluates and adjusts the optimal path of the snake robot according to energy consumption and path selection during the path planning process. The reward function rewards each state based on distance and energy consumption, and the distance difference weight k d The real-time performance of the control path optimization is controlled. When the snake robot is less than the preset distance from the target point, an additional reward is added to ensure that the target is reached quickly. The D3QN model contains two Q networks that are updated alternately, and the Q value is corrected after each action, thereby reducing the path planning deviation and improving the stability and accuracy of the path planning.

[0015] like Figure 2 As shown in the figure, the improved A* algorithm of the present invention combines the D3QN model and the energy consumption evaluation model to form an overall path planning structure. The D3QN model uses environmental information to adjust the weight of the heuristic function of the A* algorithm in real time, thereby generating an optimal path suitable for complex three-dimensional environments. Through the dual deep Q network model of D3QN, the system estimates the value of the current state and the future state respectively, ensuring that path planning not only considers the path length, but also minimizes energy consumption.

[0016] like Figure 3 As shown in the figure, the improved A* algorithm of the present invention realizes the path planning of the snake robot in a three-dimensional environment. The movement direction of the improved A* algorithm is expanded from 8 directions in two dimensions to 26 directions in three dimensions, including the basic movement directions along the x, y, and z axes and the diagonal directions between the three axes, so as to improve the flexibility and efficiency of the three-dimensional space path search. The figure shows the optimal path of the snake robot from the starting point to the target point. After the weight adjustment of the D3QN model, the path has higher adaptability and accuracy. The path planning process comprehensively considers the path length, angle change and vertical movement, so that the robot reaches the target with the lowest energy consumption.

[0017] like Figure 4 As shown in the figure, the traditional A algorithm was used for path planning in the control experiment. The path in the figure goes directly from the starting point to the target point, and the heuristic function is not optimized. The traditional A algorithm only considers the path length and does not evaluate the energy consumption, resulting in low energy efficiency in complex environments.

[0018] like Figure 5 As shown in the comparison of the weight-improved A algorithm path planning diagram, the algorithm partially optimizes the heuristic function, but still does not combine deep reinforcement learning for dynamic weight adjustment. Compared with the improved A algorithm, the weight-improved A* path consumes more energy, and the path planning effect still has a lot of room for improvement.

[0019] like Figure 6 As shown in the figure, the bidirectional A algorithm is used for path planning in the control experiment, and the search speed is improved by searching the path at the starting point and the target point at the same time. Although bidirectional A reduces the search time in some scenarios, it is not as energy efficient as the improved A* algorithm because it is not optimized for energy consumption in three-dimensional space.

[0020] like Figure 7 As shown in the figure, the improved A algorithm is compared with the traditional A algorithm, the weighted improved A algorithm and the bidirectional A algorithm in terms of energy consumption. The experimental results show that the improved A* algorithm can significantly reduce the energy consumption of the snake robot in the same environment. By introducing the D3QN model to optimize the heuristic function, the path planning is more energy-efficient and suitable for long-term task requirements.

[0021] In practical applications, the present invention implements three-dimensional path planning of the snake-like robot through the following steps: Initialization: Run the D3QN model, set the initial state space and action space, and generate preliminary path selection in a three-dimensional environment; Path selection optimization: Evaluate the energy consumption in the path based on the reward function, and adjust the heuristic function weight of the A* algorithm in real time to achieve path optimization; Path planning: Use the improved A* algorithm to search for paths in three-dimensional space, taking into account path length, angle changes, and energy consumption of vertical movement; Path transmission: transmit the path planning results to the control system of the snake robot for the next step of path tracking; Repeat path planning: Repeat the above steps until the snake robot reaches the target point to achieve optimal path planning and low-energy operation.

[0022] The present invention verifies the effectiveness of the path planning method through a large number of simulation experiments. In a three-dimensional simulation environment, the snake-like robot path planning method of the present invention effectively reduces energy consumption and improves path planning accuracy, is applicable to various complex three-dimensional scenes, and provides technical support for the long-term operation of the snake-like robot.

[0023] It is to be understood that the present invention is described by some embodiments, and those skilled in the art are aware that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, the various embodiments of the present invention may be adjusted according to specific circumstances to adapt to the needs of different application environments without departing from the core idea of ​​the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments within the scope of the claims of this application belong to the protection scope of the present invention.

Claims

1. A three-dimensional path planning method for a snake-like robot based on deep reinforcement learning, characterized in that Based on the improved A* algorithm and D3QN model, the path planning and energy consumption of the snake robot in three-dimensional space are optimized. The three-dimensional environment characteristics are learned through the D3QN model and the heuristic function weights of the A* algorithm are improved to achieve path optimization. The improved A* algorithm is then extended to path search in three-dimensional space.

2. The three-dimensional path planning method of a snake-like robot based on deep reinforcement learning according to claim 1 is characterized in that: The D3QN model estimates the current state value and the future state value respectively through a dual Q network, thereby achieving dynamic optimization of the heuristic function and reducing energy consumption in the path planning process.

3. The three-dimensional path planning method of a snake-like robot based on deep reinforcement learning according to claim 1 or 2, characterized in that: In the heuristic function of the A* algorithm, the energy consumption of path length, angle change and vertical movement are comprehensively considered to form an energy consumption evaluation model to optimize the energy consumption of path planning.

4. The three-dimensional path planning method of a snake-like robot based on deep reinforcement learning according to claim 3 is characterized in that: The movement directions of the A* algorithm are expanded from 8 directions in two dimensions to 26 directions in three dimensions, including basic movement directions along the x, y, and z axes and diagonal directions between the three axes.

5. The three-dimensional path planning method of a snake-like robot based on deep reinforcement learning according to claim 3 is characterized in that: The energy consumption evaluation model calculates the total energy consumption through path length, angle change and vertical movement consumption. The path length weight, angle weight and vertical movement weight are adjusted according to specific application requirements to balance the energy consumption of different path selections.

6. The three-dimensional path planning method for a snake-like robot based on deep reinforcement learning according to claim 5, characterized in that: The energy consumption calculation formula of the energy consumption evaluation model is: path_energy=path length consumption×path length weight+vertical movement consumption×vertical movement weight+angle change consumption×angle weight, so as to adjust according to different path planning requirements to minimize energy consumption.

7. The three-dimensional path planning method of a snake-like robot based on deep reinforcement learning according to claim 1, characterized in that: The D3QN model dynamically adjusts the heuristic function weights of the A* algorithm through the reward function, and dynamically evaluates and adjusts the optimal path of the snake robot movement according to energy consumption and path selection during the path planning process.

8. The method for three-dimensional path planning of a snake-like robot based on deep reinforcement learning according to claim 7, characterized in that: The reward function rewards each state based on distance and energy consumption, and the distance difference weight k d Control the real-time performance of path optimization. When the snake robot is less than the preset distance from the target point, an additional reward is added to ensure that the target is reached quickly.

9. The three-dimensional path planning method for a snake-like robot based on deep reinforcement learning according to any one of claims 1 to 8, characterized in that: The D3QN model includes two Q networks that are updated alternately, and the Q value is corrected after each action, thereby reducing the path planning deviation and improving the stability and accuracy of path planning.

10. A three-dimensional path planning method for a snake-like robot based on deep reinforcement learning according to any one of claims 1 to 9, characterized in that: The specific steps include: Step 1: Run the initial state and action space in the D3QN model to select a path in a three-dimensional environment; Step 2: Use the reward function to evaluate the energy consumption of the current path in real time, and adjust the heuristic function weight of the A* algorithm for path selection through the D3QN model; Step 3: Perform path search based on the improved A* algorithm, taking into account the comprehensive consumption of path length, angle change, and vertical movement; Step 4: After completing the path planning, the optimal path is transmitted to the motion control system of the snake robot; Step 5: Repeat steps 1-4 until the snake robot reaches the target point, achieving high-precision path planning and low-energy motion.

Images