Robot path exploration method based on double-agent competitive reinforcement learning

Abstract

The invention relates to a robot path exploration method based on double-agent competitive reinforcement learning, and the method comprises the following steps: S1, constructing a Markov decision model, and initializing agents and an experience pool; s2, recording a current state st of an agent Agent1, exploring k steps, and recording a current track sequence to an experience pool Buffer 1; s3, the intelligent agent Agent2 is placed at the state st, the intelligent agent Agent2 explores k steps, and a current track sequence is recorded to an experience pool Buffer 2; s4, taking the similarity between the exploration trajectories as an additional reward of the agent Agent1, and taking an opposite number as an additional reward of the agent Agent2; s5, updating strategies of the agents Agent1 and Agent2 when the number of data in the experience pool meets the requirement; s6, repeatedly executing the steps S2-S5 until the intelligent agent Agent1 reaches the target state or exceeds the set time tlimit; and S7, repeatedly executing the steps S1-S6 until the set training episode number is completed. Compared with the prior art, the method has the advantages that the intelligent agent can explore more effectively, the training speed is increased, the utilization efficiency of samples is improved, random noise can be effectively eliminated, and the robustness is higher.

Description

technical field [0001] The invention relates to the field of robot trajectory planning, in particular to a robot path exploration method based on dual-agent competitive reinforcement learning. Background technique [0002] Reinforcement learning technology has made remarkable achievements in the field of robot control, but reinforcement learning is based on the reward mechanism, that is, the goal of the agent is to obtain the cumulative maximum reward, and for most of the existing reinforcement learning applications for robot path exploration The scenarios are all sparsely rewarded environments, that is, the agent will only receive positive rewards when it reaches the final goal, and there will be no rewards in other cases, and the agent that does not receive feedback will lack an effective mechanism to update its own strategy. Converge to the ideal policy. [0003] The current solution to sparse rewards is as follows: [0004] 1. Hardly hand-coded reward functions, by int...

AI Technical Summary

Problems solved by technology

[0004] 1. Hardly hand-coded reward functions, by introducing a large amount of relevant domain knowledge to construct reward functions, to guide the agent to explore and learn according to the ideal trajectory, but this method needs to achieve sufficient understanding of domain knowledge, and it is difficult Design an ideal reward function under the environment

[0005] 2. Imitation learning, that is, to learn expert strategies, by introducing expert samples to directly clone behaviors or construct reward functions to guide agent strategies close to expert strategies, so as to obtain ideal strategies, but this method requires a certain number of expert samples in advance , which is not possible in some environments

However, this method has model constraints and is easily affected by random noise

Images