Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a policy neural network for use in controlling an agent using relative variational intrinsic control. In one aspect, a method includes selecting a skill from a set of skills, generating a trajectory by controlling an agent using a policy neural network while the policy neural network is conditioned on the selected skill,
processing an initial observation and a last observation using a relative
discriminator neural network to generate a relative
score,
processing the last observation using an absolute
discriminator neural network to generate an absolute
score, generating a reward for the trajectory as a function of the absolute
score corresponding to the selected skill and the relative score corresponding to the selected skill, and training the policy neural network based on the reward for the trajectory.