A hexapod robot path planning method based on improved reinforcement learning

Abstract

The present invention discloses a hexapod robot path planning method based on improved reinforcement learning, comprising the steps of Markov process, neural network construction, global information acquisition, exploration rate adjustment, action selection improvement, experience pool management, training step number judgment, and target neural network update. Through the present invention, the efficiency of experience pool data accumulation in the reinforcement learning algorithm can be effectively improved, and the cross-plane planning decision-making time of the hexapod robot in a multidimensional environment can be shortened. The particle swarm algorithm is used to replace the original strategy, making the robot action selection more purposeful, thereby increasing the amount of valid data in the experience pool. Compared with the existing technology, the present invention has higher path planning efficiency and stronger action selection ability, and can be widely used in the field of hexapod robots.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a hexapod robot path planning method based on improved reinforcement learning. Background Art

[0002] Mobile robots can replace humans in exploring dangerous or unknown environments, such as detecting underground pipelines and exploring dangerous plateaus and mountainous areas. Hexapod robots, which rely on discrete landing points for traversal, have a higher degree of freedom and are more suitable for operations in complex environments.

[0003] Path planning is a core technology for autonomous robots and a research hotspot in robotics. Reinforcement learning, a field within machine learning, emphasizes how intelligent agents can learn through trial and error based on their environment, acquiring valuable information and knowledge. In unknown environments, robots must not only respond to unexpected situations but also avoid obstacles. Therefore, empowering robots with autonomous learning capabilities can improve their adaptability and robustness.

[0004] The reinforcement learning path method utilizes a reinforcement learning algorithm as the primary driver. The robot continuously interacts with the environment during movement, gradually accumulating experience and then iterating to achieve the optimal strategy. For environments with large state spaces, the Deep Reinforcement Learning algorithm (Deep QNetwork, DQN) is currently the primary method used, but this algorithm suffers from overestimation during training. Subsequently, researchers improved this approach by proposing a dual-layer deep reinforcement learning algorithm (Double Deep QNetwork, DDQN) for path planning. This algorithm uses a local neural network and a target neural network to mitigate Q-value estimation bias, improving planning speed and decision-making efficiency.

[0005] Currently, most path planning for hexapod robots relies primarily on reinforcement learning. Path planning using the DDQN algorithm can mitigate overestimation during path planning and improve algorithm stability. However, like other reinforcement learning algorithms, the DDQN algorithm relies heavily on data from the experience pool. Therefore, the validity of this data determines the algorithm's decision-making speed. Excessive erroneous data in the experience pool can slow the algorithm's convergence and reduce its accuracy. Summary of the Invention

[0006] The purpose of the present invention is to provide a hexapod robot path planning method based on improved reinforcement learning.

[0007] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0008] The present invention comprises the following steps:

[0009] S1: A Markov process includes an agent, a state set, and an action set. The state set contains the position of the agent in the environment; the action set contains the actions that can be performed in each state set, and the local neural network and the target neural network with the same structure are constructed;

[0010] S2: Update the parameters of the local neural network to the target neural network so that the initial parameters of the two neural networks are the same;

[0011] S3: The running hexapod robot obtains global information including scene layout, own position, target position, and running status.

[0012] S4: Initialize the exploration rate. The value of the exploration rate gradually decreases with the number of iterations of the entire algorithm.

[0013] S5: Improve the action selection, select the action according to the local policy neural network, put the executed action in the environment to observe the next state, and update the four environment matrices with the observed state; use the accumulation in the experience pool to construct the map data, update the speed and position of the particles, and update the global optimal position. Finally, extract the next fuzzy point of the current hexapod robot's position from the planned route point, select the action in the direction of the fuzzy point, and after executing the action, observe whether there are obstacles in the state. If there are calibrated coordinates, punish and reward, and update the state matrix and map data. Otherwise, update the matrix and map data.

[0014] S6: The obtained state matrix is ​​stored in the experience pool. When the experience pool reaches the upper limit of the specified storage data, the specified amount of old data is deleted;

[0015] S7: Determine whether the maximum number of training steps has been reached. If so, proceed to the next round of training until all training cycles are completed. Otherwise, proceed to the next step.

[0016] S8: Check whether the specified number of steps is reached to update the target neural network. If so, an improved reinforcement learning algorithm is obtained for path planning.

[0017] The beneficial effects of the present invention are:

[0018] This invention is a hexapod robot path planning method based on improved reinforcement learning. Compared with existing technologies, this method not only improves the effectiveness of the experience pool data accumulated in the reinforcement learning algorithm, but also shortens the time it takes for the robot to make cross-plane planning decisions in a multidimensional environment. By replacing the original strategy with a particle swarm algorithm, the hexapod robot's action selection becomes more targeted and the amount of valid data in the experience pool is increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1is a flow chart of the present invention;

[0020] Figure 2 It is a convolutional neural network structure diagram of the present invention;

[0021] Figure 3 It is the exploration rate variation graph of the present invention;

[0022] Figure 4 It is the random strategy disadvantage diagram of the present invention;

[0023] Figure 5 is the environmental data processing diagram of the present invention;

[0024] FIG6 is a comparison diagram of the improved method of the present invention; FIG6 (a) ... Figure 1 The total reward value graph (b) randomly Figure 1 Completion time graph (c) randomly Figure 2 The total reward value graph (d) randomly Figure 2 Completion time graph (e) randomly Figure 3 The total reward value graph (f) is randomly Figure 3 Completion time diagram. DETAILED DESCRIPTION

[0025] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0026] like Figure 1 As shown: The present invention comprises the following steps:

[0027] Step (1): Construct the algorithm framework: First, a Markov process is included in the agent (hexapod robot), state set S, and action set A. S contains the position of the agent in the environment; A contains the actions that can be performed in each S. Construct two neural network models with the same structure. Different from the traditional use of backpropagation neural network (BP) to build a network structure, both the local neural network and the target neural network use convolutional neural networks with two convolutional layers, two pooling layers, and three linear layers. Figure 2 As shown in Figure 1, the entire neural network is mainly divided into three parts: the first part: the first convolution layer and the first maximum pooling layer; the second part: the second convolution layer and the second maximum pooling layer; the third part: the data obtained from the first two parts are linearly fitted through three layers of fully connected layers. Initialize the parameters θ of the local policy neural network Q and initialize the target policy neural network Q - The parameter θ - , set the hyperparameter learning rate α, discount rate λ, and experience pool D. When the agent interacts with the environment, the current position s is generatedt , action a generated by the action strategy, transfer to the next position s t+1 、Rewards and punishments given by the environment t+1 , based on the update, the Q value is calculated. The goal of the agent is to maximize the accumulated reward in the environment, so that the agent can perform the action with the maximum reward value in each state. The Q value update calculation method is as follows:

[0028] y t =r t+1 +γQ - (s t+1 ,argmax a′ Q(s t+1 ,a′;θ);θ - )(1)

[0029] The value of λ is between 0 and 1, where the larger the value, the more importance the agent attaches to the reward of the previous action. The more importance it attaches, the larger the value. a′ represents the value of using the current neural network Q - Select the next action.

[0030] Step (2): After the previous step, update the parameters θ in the local neural network Q to the target neural network Q - , so that the initial parameters of the two neural networks are the same.

[0031] Step (3): Unlike the previous one-dimensional list data storage of map data, the running hexapod robot can obtain a large amount of global information, including the scene layout (dynamic obstacles, static obstacles, exploration points), its own position, target position, and operation status. The multidimensional environment is abstracted as multiple planes composed of a gridded current layer and a gridded next layer. The overall path planning is divided into multiple local plans. The target points are located at the crossing points of the current layer that can lead to the next layer, realizing the preprocessing of the map data.

[0032] Create four matrices to represent the state of the current environment: path matrix (PM), obstacle matrix (OM), location matrix (LM) and target matrix (TM).

[0033] Step (4): Initialize the exploration rate K. The exploration rate determines the degree of subsequent guidance using the particle swarm algorithm and also affects the selection of local target points in the particle swarm algorithm in subsequent steps. The value of K will gradually decrease with the number of iterations of the entire algorithm. The K value changes as follows: Figure 3 shown.

[0034] Step (5): Different from the random strategy used in the traditional DDQN algorithm, the action selection strategy is improved here: first, the action selection is improved.

[0035]

[0036] P is a number with 8 decimal places between 0 and 1 that is randomly generated in each round, and A represents the action guided by the particle swarm optimization (PSO).

[0037] If P is greater than or equal to K: select an action based on the local policy neural network Q, and place the executed action in the environment to observe the next state s t+1 , the observed state updates the four environment matrices.

[0038] If P is less than K:

[0039] (a) Data preparation: Using the accumulated experience pool, the map data is constructed. Initialize the position and velocity of the particle swarm. For each particle i, randomly initialize its position vector X i and velocity vector V i Initialize the individual optimal position P of each particle i , initially with its current position X i Initially, the position P of the particle closest to the current position of the hexapod robot is selected. i Similarly, initialize the global optimal position G. The selection of G requires a combination of angle and distance screening with the final target point. First, calculate the angle and distance between each particle and the final target point. The formula is as follows:

[0040]

[0041] (x i ,y i ) represents the coordinates of the initial particles in the selected map data, (x g ,y g ) represents the global map target coordinates, d represents the distance, and ag represents the angle between the particle and the target. Then, a range screening is performed to determine the global optimal position G. The screening formula is as follows:

[0042]

[0043] C rand It represents the collection of particle coordinates with the random map boundary particle coordinates as the center.

[0044] (b) Update the velocity and position of the particles. For each particle i, update the velocity vector:

[0045] V i (t+1)=ω*V i (t)+c1*rand()*(P i -X i)+c2*rand()*(GX i ) (5)

[0046] Among them, V i (t) is the velocity of particle i at time t, ω is the inertia weight, c1 and c2 are learning factors, and rand() is the random number generator function; update the position vector:

[0047] Xi(t+1)=Xi(t)+Vi(t+1) (6)

[0048] Evaluate the new position X i (t+1) fitness; update the best position of the individual: if fitness (X i (t+1))>Fitness(P i ), then update P i =X i (t+1).

[0049] (c): Update the global optimal position. For each particle i, if the fitness (P i )>Fitness(G), then update G=P i .

[0050] (d) Repeat steps 2 and 3 until the stopping condition is met.

[0051] Finally, the planned route point is extracted from the next fuzzy point of the current hexapod robot's position, and the direction of the action is selected. After executing the action, observe whether there is an obstacle in the state. If there is a calibration coordinate, and the penalty reward r crash , update the state matrix and map data, update the matrix and map data anyway.

[0052] Step (6): Substitute the obtained state matrix into (S t ,a,r t+1 ,S t+1 ,over) into the experience pool D. When the experience pool reaches the upper limit of the specified storage data, the specified number of old data will be deleted. Randomly sample a batch of experience from the experience pool for training, and use Q - Estimate y t , calculate the action a that Q takes in the current state t The Q value is calculated using the mean square error loss function:

[0053]

[0054] Use gradient descent to minimize the loss and update θ:

[0055] θ←θ-α▽ θ L(θ)(8)

[0056] Step (7): Determine whether the maximum number of training steps has been reached. If so, proceed to the next round of training until all training cycles are completed. Otherwise, proceed to the next step.

[0057] Step (8) checks whether the specified number of steps is reached to update the target neural network. If so, update: θ - ←θ.

[0058] Finally, we improve the DDQN algorithm for path planning.

[0059] This embodiment selects the hardware environment: a computer with Intel (R) Core (TM) i9-13900HX, 2.20GHz CPU and NIVIDIA GeForce RTX 4060Laptop GPU.

[0060] (a) Action space adoption: The actions given by the improved policy are represented as up, down, left, and right in the abstract space.

[0061] (b) Reward value: Here, the map reward value is combined with the penalty value given by the PSO strategy. The map reward value defines different reward values ​​according to obstacles and target points:

[0062]

[0063] The final reward value: r = r map +r PSO

[0064] (c) State Space: The combined map consists of two 40x40 grid maps, one representing the current layer and the next layer. During movement, the hexapod robot matrices the state space of the current layer and gradually updates it based on the experience pool data to distinguish different objects.

[0065] This invention is aimed at the path planning of a hexapod robot in a multi-dimensional environment, and makes three improvements:

[0066] (1) Strategy optimization of traditional DDQN algorithm: In the early stage of training, the traditional DDQN algorithm uses random selection for action selection, which will lead to the accumulation of too much error data, such as Figure 4 Using the PSO algorithm to replace the original strategy will make the hexapod robot more purposeful in action selection and increase the amount of effective data in the experience pool.

[0067] Improvement of action selection strategy: First, improve the action selection.

[0068]

[0069] P is a number between 0 and 1 randomly generated in each round, K is the exploration rate, and A represents the action guided by the particle swarm algorithm.

[0070] (2) Optimize the network structure:

[0071] Both the local neural network and the target neural network use convolutional neural networks, with two convolutional layers, two pooling layers, and three linear layers. The entire neural network is mainly divided into three parts. The first part: the matrix data is passed into the convolution layer of the neural network. The convolution layer has 16 convolution kernels, each of which is 5*5 in size and has a stride of 1 (stride=1). An additional 2-unit border of 0 is added around the input data (padding=2). After the convolution operation, 16 feature matrices of size 40*40 are obtained. Create a maximum pooling layer in which the pooling window size (kernelsize) is 2*2 and the stride (stride) is also 2x2. After calculation, a feature matrix of 16*20*20 is obtained. Part 2: Using a 16*20*20 feature matrix as input data, the convolution layer has 32 convolution kernels, each 5*5 in size, with a stride of 1 (stride=1), and an extra 2-unit border of 0 is added around the input data (padding=2). After the convolution operation, 32 feature matrices of size 20*20 are obtained. Create a maximum pooling layer with a pooling window size (kernel size) of 2x2 and a stride (stride) of 2x2. After calculation, a 32*10*10 feature matrix is ​​obtained. Part 3: Perform a linear fit on the data obtained in the first two parts.

[0072] (3) The abstract multi-dimensional environment is composed of multiple planes composed of a gridded current layer and a gridded next layer. The overall path planning is divided into multiple local plans. The crossing points that can lead to the next layer on the current layer are located at the target point to achieve the preprocessing of the map data. The data obtained from the map is represented in the form of four matrices. Compared with the traditional data represented by one-dimensional lists, the map data is effectively utilized and contains more feature data. Figure 5 As shown, splitting into four matrices can extract more feature data.

[0073] The improved path planning method based on the DDQN algorithm described in this paper abstracts path planning in a multidimensional environment into multiple local plans consisting of the current layer and the next layer. Using a four-input, four-output convolutional neural network, the particle swarm algorithm replaces the original random search strategy. Three random maps were used to conduct comparative experiments comparing the method described in this paper, a path planning method based on the DDQN algorithm, and a path planning method based on the DQN algorithm. All three methods used a comprehensive reward and path completion time evaluation.

[0074] The comprehensive reward value is the sum of the rewards obtained by the agent during the entire planning process, which is calculated as follows:

[0075] Total reward=reward current layer +reward next layer (11)

[0076] Path completion time calculation:

[0077]

[0078] Construct three random maps, and the map parameters are shown in Table 1. The experimental results with the same number of iterations are shown in Figure 6. In Figure (a), the random Figure 1 The total reward value graph (b) randomly Figure 1 Completion time graph (c) randomly Figure 2 The total reward value graph (d) randomly Figure 2 Completion time graph (e) randomly Figure 3 The total reward value graph (f) is randomly Figure 3 The completion time graph shows the improved DDQN algorithm with a particle swarm algorithm strategy, the blue line represents the DDQN algorithm with a traditional random search strategy, and the yellow line represents the DQN algorithm with a traditional random search strategy. Comparing Figures (a), (c), and (e) on three random maps, the improved algorithm, due to the guidance of the particle swarm algorithm, reaches a total reward value of 80 faster than the DQN algorithm with a traditional random search strategy and the DDQN algorithm with a traditional random search strategy at the same number of iterations, eventually reaching a stable stage, while the other two algorithms have not yet reached a stable value, indicating that the improved algorithm converges faster. Furthermore, comparing Figures (b), (d), and (f), at the same movement speed, the improved algorithm takes less time to complete the path, indicating that the path planned by the improved algorithm is shorter than the paths planned by the other two algorithms in the figure.

[0079] Table 1

[0080]

[0081] The training results were tested, as shown in Table 2. Time refers to the decision-making time from the initial method; total steps refers to the number of action steps required to complete the plan to the target point; and time saved refers to the time saved by the improved method compared to the other two methods in terms of decision-making time. The results demonstrate that the method described in this invention achieves higher overall rewards and has the shortest decision-making time of the three methods.

[0082] Table 2

[0083]

[0084]

[0085] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. Any technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.

Claims

1. A hexapod robot path planning method based on improved reinforcement learning, characterized in that: The following steps are involved: S1: A Markov process includes an agent, a state set, and an action set. The state set contains the position of the agent in the environment; the action set contains the actions that can be performed in each state set, and the local neural network and the target neural network with the same structure are constructed; S2: Update the parameters of the local neural network to the target neural network so that the initial parameters of the two neural networks are the same; S3: The running hexapod robot obtains global information including scene layout, own position, target position, and running status. S4: Initialize the exploration rate. The value of the exploration rate gradually decreases with the number of iterations of the entire algorithm. S5: Improve the action selection, select the action according to the local policy neural network, put the executed action in the environment to observe the next state, and update the four environment matrices with the observed state; use the accumulation in the experience pool to construct the map data, update the speed and position of the particles, and update the global optimal position. Finally, extract the next fuzzy point of the current hexapod robot's position from the planned route point, select the action in the direction of the fuzzy point, and after executing the action, observe whether there are obstacles in the state. If there are calibrated coordinates, punish and reward, and update the state matrix and map data. Otherwise, update the matrix and map data. The action selection is improved as follows: P is a number with 8 decimal places between 0 and 1 that is randomly generated in each round, and A represents the action guided by the particle swarm algorithm; If P is greater than or equal to K: select an action based on the local neural network Q, and place the executed action in the environment to observe the next state s t+1 , the observed state updates the four environment matrices; If P is less than K: (a) Constructing the map data and initializing the position and velocity of the particle swarm: For each particle i, its position vector X is randomly initialized. i and velocity vector V i ; Initialize the individual optimal position P of each particle i , initially with its current position X i Same; initially, the position P of the particle closest to the current hexapod robot position is selected i Similarly, initialize the global optimal position G; the selection of G requires a combination of angle and distance screening with the final target point. First, calculate the angle and distance between each particle and the final target point. The formula is as follows: (x i ,y i ) represents the coordinates of the initial particles in the selected map data, (x g ,y g ) represents the global map target coordinates, d represents the distance, and ag represents the angle between the particle and the target; Then perform range screening to determine the global optimal position G. The screening formula is as follows: C rand It represents the set of particle coordinates with the coordinates of the random map boundary particle as the center; (b) Update the velocity and position of the particles: For each particle i, update the velocity vector: V i (t+1)=ω*V i (t)+c1*rand()*(P i -X i )+c2*rand()*(G-X i ) (4) Among them, V i (t) is the velocity of particle i at time t, ω is the inertia weight, c1 and c2 are learning factors, and rand() is the random number generator function; update the position vector: Xi(t+1)=Xi(t)+Vi(t+1) (5) Evaluate the new position X i (t+1) fitness; update the best position of the individual: if fitness (X i (t+1))>Fitness(P i ), then update P i =X i (t+1); (c): Update the global optimal position: For each particle i, if the fitness (P i )>Fitness(G), then update G=P i； (d): Repeat steps S2 and S3 until the stop condition is met; finally, extract the next fuzzy point of the current hexapod robot's position from the planned route point, select the direction action of the point, and after executing the action, observe whether there is an obstacle in the state. If there is a calibrated coordinate, and punish the reward r crash , update the state matrix and map data, update the matrix and map data anyway; S6: The obtained state matrix is ​​stored in the experience pool. When the experience pool reaches the upper limit of the specified storage data, the specified amount of old data is deleted; S7: Determine whether the maximum number of training steps has been reached. If so, proceed to the next round of training until all training cycles are completed. Otherwise, proceed to the next step. S8: Check whether the specified number of steps is reached to update the target neural network. If so, an improved reinforcement learning algorithm is obtained for path planning.

2. The hexapod robot path planning method based on improved reinforcement learning according to claim 1, characterized in that: Both the local neural network and the target neural network use convolutional neural networks. The convolutional neural network has two convolutional layers, two pooling layers and three linear layers. The convolutional neural network is mainly divided into three parts: the first part: the first convolutional layer and the first maximum pooling layer; the second part: the second convolutional layer and the second maximum pooling layer; the third part: the data obtained from the first two parts are linearly fitted through three fully connected layers.

3. The hexapod robot path planning method based on improved reinforcement learning according to claim 2, characterized in that: In step S1, the state set S, the action set A, the parameters θ of the local neural network Q and the target neural network Q are initialized. - The parameter θ - , set the hyperparameter learning rate α, discount rate λ, experience pool D; when the hexapod robot interacts with the environment, the current position s t , action a generated by the action strategy, transfer to the next position s t+1 、Rewards and punishments given by the environment t+1 , based on the updated Q value calculation; the purpose of the hexapod robot is to obtain the maximum cumulative reward in the environment, so that the hexapod robot can perform the action with the maximum reward value in each state; the Q value update calculation method is as follows: y t =r t+1 +γQ - (s t+1 ,argmax a′ Q(s t+1 ,a′;θ);θ - )(6) The value of λ is between 0 and 1, where the larger the value, the more importance the agent attaches to the reward of the previous action. The larger the value, the greater the importance. a′ represents the value of using the current neural network Q - Select the next action.

4. The hexapod robot path planning method based on improved reinforcement learning according to claim 1, characterized in that: The step S6 is specifically as follows: the obtained state matrix is ​​converted into (S t ,a,r t+1 ,S t+1 ,over) into the experience pool D. When the experience pool reaches the upper limit of the specified storage data, the specified number of old data will be deleted. A batch of experience will be randomly sampled from the experience pool for training. - Estimate y t , calculate the action a that Q takes in the current state t The Q value is calculated using the mean square error loss function: Use gradient descent to minimize the loss and update θ: θ←θ-α▽θL(θ)(8).

Images