A diffusion-based dual generative playback method for continuous off-policy reinforcement learning

Abstract

A diffusion-based dual-generation replay continuous offline reinforcement learning method first decouples the continuous learning strategy into a diffusion-based behavior generation model and a multi-head action evaluation model. Second, a task-conditional diffusion model is trained to simulate the state distribution of the old task, and the generated states are paired with the corresponding responses of the behavior generation model to represent the old task using high-fidelity replay pseudo-samples. Finally, by interleaving pseudo-samples with real samples of the new task, the state and behavior generation models are continuously updated to model increasingly diverse behaviors, and the multi-head action evaluation model is standardized using behavior cloning to reduce forgetting. This invention proposes a dual-generation replay framework that retains knowledge of the old task through concurrent replay of generated pseudo-data. Experiments demonstrate that the proposed method achieves better forward transfer and retains less forgetting in continuous offline learning, and closely approximates results using real data from the old task due to its high-fidelity replay of the sample space.

Images