A robot path exploration method based on dual-agent competitive reinforcement learning
Through the dual-agent competitive reinforcement learning method, the intrinsic rewards are generated by using trajectory similarity, which solves the problems of low exploration efficiency and strategy convergence in sparse reward environments in robot path exploration, and achieves faster training speed and higher sample utilization efficiency.
Patent Information
- Application Number
- CN202111637946.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-29
- Publication Date
- 2025-09-23
- Estimated Expiration
- 2041-12-29
AI Technical Summary
Existing robot path exploration reinforcement learning methods have the problem that the agent cannot converge to the ideal strategy in sparse reward environments, and it is difficult to balance exploration and utilization, resulting in low exploration efficiency.
A dual-agent competitive reinforcement learning method is adopted. By constructing a Markov decision model, the trajectory similarity and cross-entropy function between the two agents are used to generate intrinsic rewards, which promotes competition and exploration between the agents and avoids the complex reward function design and the introduction of domain knowledge.
It effectively solves the problem of ineffective exploration of intelligent agents in sparse reward environments, improves exploration efficiency and training speed, enhances robustness to random noise, and avoids strategy convergence to local optimality.
Smart Images
Figure CN114372520B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot trajectory planning, and in particular to a robot path exploration method based on dual-agent competitive reinforcement learning. Background Art
[0002] Reinforcement learning technology has achieved remarkable success in the field of robotic control. However, reinforcement learning is based on a reward mechanism, that is, the goal of the intelligent agent is to obtain the maximum cumulative reward. However, most existing reinforcement learning application scenarios for robot path exploration are sparse reward environments, that is, the intelligent agent will only receive positive rewards when it reaches the final goal, and no rewards in other cases. Intelligent agents that do not receive feedback will lack an effective mechanism to update their own strategies and cannot converge to the ideal strategy.
[0003] The current solution for sparse rewards is as follows:
[0004] 1. Hard hand-coded reward function: By introducing a large amount of relevant domain knowledge to construct the reward function, the agent is guided to explore and learn along the ideal trajectory. However, this method requires sufficient understanding of domain knowledge and it is difficult to design an ideal reward function in a more complex environment.
[0005] 2. Imitation learning, that is, learning expert strategies by introducing expert samples to directly clone behaviors or constructing reward functions to guide the agent strategy to be close to the expert strategy, thereby obtaining the ideal strategy. However, this method requires obtaining a certain number of expert samples in advance, which is not possible in some environments.
[0006] 3. Intrinsic reward-driven algorithms, exemplified by curiosity, estimate the next state from the current state-action pair, treating the difference (curiosity) as an intrinsic reward. This intrinsic reward encourages the agent to explore newer states, while also leveraging adjacent states to fit actions for feature selection and robustness. However, this approach is subject to model constraints and is susceptible to random noise. Summary of the Invention
[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a robot path exploration method based on dual-agent competitive reinforcement learning.
[0008] The purpose of the present invention can be achieved by the following technical solutions:
[0009] A robot path exploration method based on dual-agent competitive reinforcement learning, the method comprising the following steps:
[0010] S1. Build a Markov decision model, initialize Agent1 and Agent2, and initialize experience pools Buffer1 and Buffer2;
[0011] S2, record the current state s of Agent1 t , Agent1 explores k steps according to the existing strategy and records the current trajectory sequence to the experience pool Buffer1;
[0012] S3, put the agent Agent2 into state s t At , Agent2 explores k steps according to its strategy and records the current trajectory sequence to the experience pool Buffer2;
[0013] S4. Use the cross entropy function to evaluate the similarity between the exploration trajectories of two agents As an additional reward for Agent1, similarity The opposite of As an additional reward for Agent 2, and update the reward information of the corresponding tracks in the experience pool Buffer1 and Buffer2;
[0014] S5. When the number of data in the experience pool meets the requirements, update the strategies of Agent1 and Agent2;
[0015] S6, repeat steps S2-S5 until Agent1 reaches the target state or exceeds the set time t limit ;
[0016] S7. Repeat steps S1-S6 until the set number of training episodes is completed.
[0017] In the step S1, a Markov decision model is constructed, and the image of the area observable by the agent at the current position is taken as the current state. t Under this condition, the agent selects the current action a according to the strategy π(a|s) t Interact with the environment and reach the next state s t+1 , get reward r t , the goal of the agent is to obtain the optimal strategy π ＊ To maximize the cumulative reward, the reward r t The specific settings are:
[0018] When the agent reaches the target position, it is given a reward of +100, and the remaining positions are given a reward of -1.
[0019] In step S1, the agent uses the DDQN reinforcement learning method to initialize the network Q. The agent's strategy π(a|s) is given according to the DDQN reinforcement learning method. Specifically, the ∈-greedy strategy is selected, and then:
[0020]
[0021] Where m is the number of actions of the agent, that is, the total number of actions in the action set A = {a1, a2, a3, a4, a5, a6, a7}, a1, a2, a3, a4, a5, a6, a7 respectively represent the agent moving one square forward, backward, left, right, turning left, turning right and staying in place, ∈ is a constant with a value of 0.1.
[0022] In step S2, the step length k is set, and the agent Agent1 explores k steps to obtain the trajectory sequence {s 1 t ,a 1 t ,s 1 t+1 ,r 1 t},{s 1 t+1 ,a 1 t+1 ,s 1 t+2 ,r 1 t+1}…{s 1 t+k ,a 1 t+k ,s 1 t+k ,r 1 t+k} and stored in the experience pool Buffer1. The exploration strategy of Agent1 at each step uses a greedy algorithm to obtain the current action.
[0023] In step S3, the position of Agent2 is set to the initial position of the current segment exploration of Agent1. Explore k steps to get the trajectory sequence And stored in the experience pool Buffer2, the exploration strategy of Agent2 at each step uses a greedy algorithm to obtain the current action.
[0024] In step S4, the exploration trajectory is taken out from the experience pool Buffer1 and Buffer2 respectively. After normalizing them respectively, we get trajectories T1 and T2, and get the absolute value of the cross entropy between the two trajectories clip(abs(-∑T1log(T2)),0,1) as the similarity between the two trajectories T1 and T2 And update the track reward λ is the attenuation coefficient.
[0025] To ensure convergence, the decay coefficient λ becomes 0.9 times of the previous step after each update.
[0026] In step S5, for Agent1 and Agent2, the DDQN method is used to update the strategy, and then:
[0027] For the agent Agent1, randomly sample l trajectory sequences from the experience pool Buffer1 And calculate the current Q value target y 1 j =r 1 j +γQ′(s 1 j+1 ,argmax a′ Q(s 1 j+1 ,a,w),w′), where Q is the current network in the DDQN algorithm, w is its network parameter, Q′ is the target network in the DDQN algorithm, w′ is the network parameter of the target network, γ is the attenuation coefficient, and then the mean square error loss function is obtained Update the current network Q through the gradient of the neural network, and update the network parameters of Q to Q' every q times.
[0028] In step S6, the time t is set limit Set to 100.
[0029] In step S7, the number of training episodes is set to 500.
[0030] Compared with the prior art, the present invention has the following advantages:
[0031] 1. The first problem faced by existing robot reinforcement learning path exploration methods is that due to the lack of rewards, the agent's exploration will converge to a local position and cannot reach the target point. The cost of manually designing a reward function to guide the agent's exploration is high. To address this problem, the present invention does not require the introduction of task-related domain knowledge, nor does it require complex manual reward design. The trajectory similarity information between agents is used as an internal reward to avoid the agent from conducting ineffective exploration due to insufficient external rewards, which makes the algorithm unable to converge. At the same time, the randomness of the agent's exploration is used to eliminate the random noise of the state estimation of the ordinary curiosity algorithm, making the agent more robust to random noise. Therefore, this method enables the agent to more effectively solve the sparse reward problem.
[0032] 2. The second problem faced by existing robot reinforcement learning path exploration methods is how to balance the exploration and exploitation of the agent, that is, when should the agent explore the environment and when should it fit the reward estimate to update the strategy. Existing reinforcement learning methods either over-pursue exploration (such as curiosity-based algorithms), making the agent strategy unable to converge, or over-pursue updating of existing strategies (such as manually designed rewards), causing the agent strategy to fall into a local optimum. The present invention creatively uses two agents to compete to construct internal rewards. After agent Agent 1 explores, agent Agent 2 learns the trajectory of agent Agent 1. When the two trajectories are relatively close, agent Agent 1 is given a smaller internal reward and agent Agent 2 is given a larger internal reward. That is, the additional goal of agent Agent 1 is to move away from agent Agent 2 and reach a more difficult goal, thereby prompting agent Agent 1 to increase exploration. This balances exploration and exploitation, which not only speeds up training and improves sample utilization efficiency, but also effectively prevents the agent strategy from converging to a local optimal solution. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 This is the overall architecture diagram of the method of the present invention.
[0034] Figure 2 Schematic diagram of the implementation environment of the present invention.
[0035] Figure 3 Flowchart of the algorithm of the method of the present invention.
[0036] Figure 4 Schematic diagram of the Q network structure.
[0037] Figure 5 This is a diagram showing the verification results of the method of the present invention. DETAILED DESCRIPTION
[0038] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0039] Example
[0040] The present invention provides a method, which is a robot path exploration method based on dual-agent competitive reinforcement learning. It uses the similarity of the exploration states of two agents (in this case, two identical robots or a robot in a repeatable experimental environment) to generate intrinsic rewards, thereby enhancing agent exploration. It does not require complex reward function design, avoids the introduction of domain knowledge, and uses the certainty of the agent's explored trajectories to eliminate the influence of random noise.
[0041] The framework diagram of this method is as follows Figure 1As shown in , the verification environment uses the reinforcement learning test environment MultiRooms trajectory exploration environment, that is, a series of 4 to 6 room exploration environments are randomly generated in a two-dimensional grid plane, such as Figure 2 As shown, the triangle represents the position of the agent, and the bright square represents the target position that the agent needs to reach. The main flow chart is as follows Figure 3 As shown, the present invention specifically includes the following steps:
[0042] S1. Build a Markov decision model, initialize Agent1 and Agent2, and initialize experience pools Buffer1 and Buffer2;
[0043] S2, record the current state s of Agent1 t ,Agent Agent1 explores k steps according to the existing strategy and records the ,current trajectory sequence to Buffer1;
[0044] S3, place Agent2 in state s t At , Agent2 explores k steps according to its strategy and records the current trajectory sequence to Buffer2;
[0045] S4. Use the cross entropy function to evaluate the similarity between the exploration trajectories of two agents As an additional reward for Agent1, similarity The opposite of As an additional reward for Agent 2, and update the reward information of the corresponding trajectory in the experience pool Buffer1 and Buffer2;
[0046] S5. When the number of data in the experience pool meets the requirements, update the strategies of Agent1 and Agent2;
[0047] S6, repeat S2--S5 until Agent1 reaches the target state or exceeds the set time t limit ;
[0048] S7, repeat S1--S6 until the set number of training episodes is completed;
[0049] During the implementation of step S1, a Markov decision model is constructed for the environment MultiRooms, such as Figure 2 As shown, the two-dimensional plane is gridded, and the image of the area that the agent can observe at the current position is recorded as the state s of the agent, that is, Figure 2The light-colored 7x7 square covered with a triangle symbol is shown as follows; the agent's action set A = {a1, a2, a3, a4, a5, a6, a7} is denoted as the agent's movement forward, backward, left, right, and left, respectively, turning left and right, and stopping. The reward r is kept sparse, and to ensure that the reward is maximized only when the target position is reached, the specific settings are as follows: when the agent reaches the target position, a reward of +100 is given, and a reward of -1 is given for all other positions. The reinforcement learning algorithm uses DDQN, and the network Q is initialized. The network structure is as follows Figure 4 As shown; the agent's strategy π(a|s) is given according to the selected reinforcement learning method. Here, the DDQN reinforcement learning method is selected, and the strategy selection adopts the ∈-greedy strategy, then:
[0050]
[0051] Where m is the number of actions of the agent, which is 7 here, and ∈ is 0.1 here. The entire Markov decision model is as follows. In the current state s t Under this condition, the agent selects the current action a according to the strategy π(a|s) t Interact with the environment and reach the next state s t+1 , get reward r t , the goal of the agent is to obtain the optimal strategy π ＊ Maximize the cumulative reward, that is, explore to reach the target point.
[0052] During the implementation of step S2, in this example, the step length k is selected as 3, and the agent Agent1 explores k steps according to the strategy π(a|s) to obtain the trajectory sequence {s 1 t ,a 1 t ,s 1 t+1 ,r 1 t},{s 1 t+1 ,a 1 t+1 ,s 1 t+2 ,r 1 t+1},{s 1 t+2 ,a 1 t+2 ,s 1 t+3 ,r 1 t+2} is stored in the experience pool Buffer1, where the agent's exploration strategy at each step uses a greedy algorithm to obtain the current action a t 1 .
[0053] During the implementation of step S3, the position of Agent2 is set to the initial position of the current segment exploration of Agent1. Get the trajectory sequence {s 2 t ,a 2 t ,s 2 t+1 ,r 2 t},{s 2 t+1 ,a 2 t+1 ,s 2 t+2 ,r 2 t+1},{s 2 t+2 ,a 2 t+2 ,s 2 t+3 ,r 2 t+2}Stored in experience pool Buffer2.
[0054] During the implementation of step S4, the exploration trajectory is taken out from Buffer1 and Buffer2 respectively. After normalizing them, we get trajectories T1 and T2, and calculate the absolute value of the cross entropy between the two trajectories clip(abs(-∑T1log(T2)),0,1) as the similarity s between the two trajectories. 12 , update trajectory reward λ is the attenuation coefficient. To ensure the convergence of the algorithm, λ = 0.9λ after each update.
[0055] During the implementation of step S5, for Agent1 and Agent2, the strategy is updated according to the DDQN method. Taking Agent1 as an example, we will randomly sample l trajectory sequences from the experience pool. Calculate the current Q value target y 1 j =r 1 j +γQ′(s 1 j+1 ,argmax a′ Q(s 1 j+1 ,a,w),w′), where Q is the current network in the DDQN algorithm, w is its network parameter, Q′ is the target network in the DDQN algorithm, w′ is its network parameter, and γ is the attenuation coefficient. Then calculate the mean square error loss function Then the Q network is updated inversely through the gradient of the neural network, and the network parameters of Q are updated to Q′ every q times.
[0056] During the implementation of step S6, the timeout time t limit Set to 100.
[0057] During the implementation of step S7, the number of training episodes is set to 500. It should be noted that both Agent 1 and Agent 2 have performance above the baseline level after strategy convergence, but Agent 1 performs better. At the same time, in order to verify the effectiveness of the algorithm, three different random seeds are selected to record the mean and variance of the reward value curve during the training process and compare them with the baseline algorithm DDQN, as shown in the figure below. Figure 5 shown.
[0058] The dual-agent reinforcement learning exploration method provided by the present invention has the following innovations compared to existing reinforcement learning methods: it leverages competition between the two agents to autonomously generate intrinsic rewards, and uses these intrinsic rewards to guide the agents to increase their exploration. This solves the problem of agents being unable to obtain rewards and having to engage in ineffective exploration under sparse reward conditions, allowing the agents to explore more quickly, accelerating algorithm training and improving sample utilization efficiency. This method does not require the introduction of task-related domain knowledge or complex manual reward design, thereby reducing the design cost of solving the problem. Furthermore, the randomness of the agent's exploration is leveraged to eliminate the random noise in the state estimation of the ordinary curiosity algorithm, making the agent more robust to random noise.
[0059] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.
Claims
1. A robot path exploration method based on dual-agent competitive reinforcement learning, characterized in that: The method comprises the following steps: S1. Build a Markov decision model, initialize Agent1 and Agent2, and initialize experience pools Buffer1 and Buffer2; S2, record the current state s of Agent1 t , Agent1 explores k steps according to the existing strategy and records the current trajectory sequence to the experience pool Buffer1; S3, put the agent Agent2 into state s t At , Agent2 explores k steps according to its strategy and records the current trajectory sequence to the experience pool Buffer2; S4. Use the cross entropy function to evaluate the similarity between the exploration trajectories of two agents As an additional reward for Agent1, similarity The opposite of As an additional reward for Agent 2, and update the reward information of the corresponding tracks in the experience pool Buffer1 and Buffer2; S5. When the number of data in the experience pool meets the requirements, update the strategies of Agent1 and Agent2; S6, repeat steps S2-S5 until Agent1 reaches the target state or exceeds the set time t limit ; S7, repeat steps S1-S6 until the set number of training episodes is completed; in step S1, a Markov decision model is constructed, with the image of the area observable by the agent at the current position as the current state, and given the current state s t Under this condition, the agent selects the current action a according to the strategy π(a|s) t Interact with the environment and reach the next state s t+1 , get reward r t , the goal of the agent is to obtain the optimal strategy π ★ To maximize the cumulative reward, the reward r t The specific settings are: When the agent reaches the target position, it is given a reward of +100, and the rest of the positions are given a reward of -1. In step S1, the agent uses the DDQN reinforcement learning method to initialize the network Q. The agent's strategy π(a|s) is given according to the DDQN reinforcement learning method. Specifically, the ∈-greedy strategy is selected, and then: Where m is the number of actions of the agent, that is, the total number of actions in the action set A = {a1, a2, a3, a4, a5, a6, a7}, a1, a2, a3, a4, a5, a6, a7 respectively represent the agent moving forward, backward, left, right one square, turning left, turning right and stopping at the same place, ∈ is a constant; in the step S2, the step length k is set, and the agent Agent1 explores k steps to obtain the trajectory sequence {s 1 t ,a 1 t ,s 1 t+1 ,r 1 t },{s 1 t+1 ,a 1 t+1 ,s 1 t+2 ,r 1 t+1 }…{s 1 t+k ,a 1 t+k ,s 1 t+k ,r 1 t+k } and stored in the experience pool Buffer1, the exploration strategy of Agent1 in each step uses the greedy algorithm to obtain the current action; in the step S3, the position of Agent2 is set to the initial position of the current exploration segment of Agent1 Explore k steps to get the trajectory sequence {s 2 t ,a 2 t ,s 2 t+1 ,r 2 t },{s 2 t+1 ,a 2 t+1 ,s 2 t+2 ,r 2 t+1 }…{s 2 t+k ,a 2 t+k ,s 2 t+k ,r 2 t+k} and stored in the experience pool Buffer2, the exploration strategy of Agent2 at each step uses the greedy algorithm to obtain the current action; in the step S4, the exploration trajectory is taken from the experience pool Buffer1 and Buffer2 respectively After normalizing them respectively, we get trajectories T1 and T2, and get the absolute value of the cross entropy between the two trajectories clip(abs(-∑T1log(T2)),0,1) as the similarity between the two trajectories T1 and T2 And update the track reward λ is the attenuation coefficient.
2. A robot path exploration method based on dual-agent competitive reinforcement learning according to claim 1, characterized in that: To ensure convergence, the decay coefficient λ becomes 0.9 times of the previous step after each update.
3. A robot path exploration method based on dual-agent competitive reinforcement learning according to claim 1, characterized in that: In step S5, for Agent1 and Agent2, the DDQN method is used to update the strategy, and then: For the agent Agent1, randomly sample l trajectory sequences from the experience pool Buffer1 And calculate the current Q value target y 1 j =r 1 j +γQ′(s 1 j+1 ,argmax a′ Q(s 1 j+1 ,a,w),w′), where Q is the current network in the DDQN algorithm, w is its network parameter, Q′ is the target network in the DDQN algorithm, w′ is the network parameter of the target network, γ is the attenuation coefficient, and then the mean square error loss function is obtained Update the current network Q through the gradient of the neural network, and update the network parameters of Q to Q' every q times.
4. A robot path exploration method based on dual-agent competitive reinforcement learning according to claim 1, characterized in that: In step S6, the time t is set limit Set to 100.
5. The robot path exploration method based on dual-agent competitive reinforcement learning according to claim 1 is characterized in that: In step S7, the number of training episodes is set to 500.