A diffusion model for generating audio data based on descriptive text prompts

Abstract

A corpus of text data is generated using a machine learning text generation model. The corpus of text data includes a plurality of sentences. Each sentence describes a type of audio. For each of a plurality of audio recordings, the audio recording is processed using a machine learning audio classification model to obtain training data including the audio recording and one or more sentences from the plurality of sentences that are closest to the audio recording in a joint audio-text embedding space of the machine learning audio classification model. The sentence(s) are processed using a machine learning generative model to obtain intermediate representations of the one or more sentences. The intermediate representations are processed using a machine learning cascade diffusion model to obtain audio data. The machine learning cascade diffusion model is trained based on differences between the audio data and the audio recording.

Images