Diffusion model for audio data generation based on descriptive text prompts

Abstract

A corpus of text data is generated using a machine learning text generation model. The corpus of text data includes a plurality of sentences. Each sentence describes a type of audio. For each of a plurality of audio recordings, the audio recording is processed with a machine learning audio classification model to obtain training data including the audio recording and one or more of the plurality of sentences that are closest to the audio recording in a joint audio-text embedding space of the machine learning audio classification model. The sentence is processed with a machine learning generation model to obtain an intermediate representation of the one or more sentences. The intermediate representation is processed with a machine learning cascaded diffusion model to obtain audio data. The machine learning cascaded diffusion model is trained based on a difference between the audio data and the audio recording. In a computing system, the machine learning generator is used to generate an intermediate representation of text content corresponding to a query that describes a user-desired type of audio content, the machine learning diffusion model is used to obtain audio data based on the intermediate representation of text content, the obtained audio data including audio of the desired type of audio content.

Description

[0001] Priority requirements

[0002] This application is based on and claims priority to U.S. Provisional Application 63 / 481,746, filed January 26, 2023. The applicant claims priority and benefit from this application, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to generative models. More specifically, this disclosure relates to a diffusion model trained to generate audio data based on audio type descriptions. Background Technology

[0004] Generative models have proven increasingly useful in the field of machine learning. Deep generative models are used across a wide range of domains. Some types of generative models, such as diffusion models, can be trained to generate high-quality output in a format different from the input. For example, some diffusion models can be trained to process text prompts to generate high-quality image output. Summary of the Invention

[0005] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0006] One example aspect of this disclosure relates to a computing system. The computing system includes one or more processors. The computing system includes a machine learning generative model trained to generate an intermediate representation of text content. The computing system includes a machine learning diffusion model trained to generate audio data from the intermediate representation of the text content, wherein the audio data is in response to a query described by the text content. The computing system includes one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause participating computing devices to perform operations. The operations include processing the text content with the machine learning generative model to generate an intermediate representation of the text content, wherein the text content describes a query of a desired type indicating audio content. The operations also include processing the intermediate representation with the machine learning diffusion model to obtain audio data, wherein the audio data includes audio of the desired type of audio content.

[0007] Another exemplary aspect of this disclosure relates to a computer-implemented method. The method includes generating a corpus of text data using a machine learning text generation model by a computing system including one or more computing devices, wherein the corpus of text data comprises multiple sentences, and each sentence describes a type of audio. The method includes: for each of a plurality of audio records, processing the audio record by the computing system using a machine learning audio classification model to obtain training data, the training data including the audio record and one or more sentences among the plurality of sentences that are closest to the audio record within a joint audio-text embedding space of the machine learning audio classification model. The method includes processing the one or more sentences by the computing system using a machine learning generation model to obtain intermediate representations of the one or more sentences. The method includes processing the intermediate representations by the computing system using a machine learning cascade diffusion model to obtain audio data. The method includes training the machine learning cascade diffusion model by the computing system based on the differences between the audio data and the audio record.

[0008] Another example aspect of this disclosure relates to a non-transitory computer-readable medium storing one or more instructions that, when executed by one or more processors, cause participating computing devices to perform operations. The operations include processing text content with a machine learning generative model to generate an intermediate representation of the text content, wherein the text content describes a query of a desired type indicating audio content, and wherein the machine learning generative model is trained to generate the intermediate representation of the text content. The operations also include processing the intermediate representation with a machine learning diffusion model to obtain audio data, wherein the audio data includes audio of a desired type of audio content, and wherein the machine learning diffusion model is trained to generate the audio data from the intermediate representation of the text content.

[0009] Other aspects of this disclosure relate to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to illustrate the relevant principles. Attached Figure Description

[0011] Referring to the accompanying drawings, a detailed discussion of embodiments is set forth in this specification for those skilled in the art, in which:

[0012] Figure 1A A block diagram of an example computing system for training a generative model to generate audio data from text content, according to an example embodiment of the present disclosure, is depicted.

[0013] Figure 1B A block diagram of an example computing device for performing high-quality generation of audio data conditionally based on text prompts, according to an example embodiment of the present disclosure, is depicted.

[0014] Figure 1C A block diagram of an example computing device for training a machine learning model to generate high-quality audio data, according to an example embodiment of the present disclosure, is depicted.

[0015] Figure 2 A data flow diagram for training a joint audio-text embedding model for audio classification is depicted according to an example embodiment of the present disclosure.

[0016] Figure 3 A data flow diagram is depicted according to an example embodiment of the present disclosure for training a machine learning generative model and a machine learning diffusion model to generate high-quality audio data conditioned on text prompts.

[0017] Figure 4 A data flow diagram is depicted illustrating the use of a machine learning generative model and a machine learning diffusion model, which are jointly trained to generate high-quality audio data, to generate high-quality audio from text prompts, according to an example embodiment of this disclosure.

[0018] Figure 5 A flowchart is depicted illustrating an example method for training a machine learning diffusion model to generate high-quality audio data according to an example embodiment of the present disclosure.

[0019] The repeated reference numerals across multiple figures are intended to identify the same features in various implementations. Detailed Implementation

[0020] Overview

[0021] In general, this disclosure relates to a diffusion model trained to generate audio data based on descriptions of audio genres. More specifically, generative models capable of producing high-quality audio from different types of input, such as text prompts, are highly desirable. However, achieving such models with sufficient accuracy presents numerous obstacles. As an example, training a model to generate audio data from text prompts requires extensive text descriptions of audio clips, which can be extremely difficult to extract (or generate). As another example, architectures commonly used for such models often fail to produce audio with sufficient accuracy.

[0022] Therefore, this disclosure proposes a diffusion model for generating audio data based on descriptive text prompts. More specifically, the computational system can obtain a large corpus of associated audio samples and descriptive text data. For example, the computational system can extract audio samples from audio data of videos hosted by an audiovisual data hosting entity, and the computational system can extract a corresponding corpus of descriptive text data (e.g., music videos and user-provided comments for the music videos) from text content provided by users to describe the corresponding audio samples. The computational system can use the corpus of audio samples and descriptive text data to train a machine learning audio classification model (e.g., a joint audio-text embedding model, etc.) using a contrastive loss function.

[0023] The computational system can utilize machine learning audio classification models to train and optimize models for audio generation. More specifically, the computational system can first generate a corpus of text data using a machine learning text generation model (e.g., a large language model (LLM)). This corpus of text data can include a large number of sentences, each describing a different type of audio (e.g., “A light EDM drumbeat carries a bass guitar, strings, and a simple piano”). The computational system can also obtain a large number of audio recordings. The computational system can then use the machine learning audio classification model to evaluate the corpus of audio recordings and text data to select the most accurate pairs of audio recordings and descriptive sentences for training data.

[0024] For each pair of audio recordings and text content (i.e., sentences), the computational system can first process the text content using a machine learning generative model to obtain an intermediate representation of the text content (e.g., latent representation, low-fidelity audio samples, spectrograms, etc.). The computational system can then process this intermediate representation using a machine learning diffusion model to obtain audio data. The computational system can train the machine learning diffusion model and / or the machine learning generative model using a loss function that evaluates the difference between the audio data and the audio recordings corresponding to the text content. In this manner, the implementation of this disclosure can efficiently and effectively train a series of models to generate high-quality audio samples conditioned on text content.

[0025] The various aspects of this disclosure provide several technical effects and benefits. As an example, conventional generative models typically generate low-quality audio data from text prompts, or lack the ability to generate audio data from text content altogether. Furthermore, training such models requires substantial computational resources (e.g., power, memory, energy, computation cycles, bandwidth, etc.). However, the implementation of this disclosure provides the ability to generate specially tailored training data to more effectively and efficiently train novel architectures for generating high-quality audio data, thus reducing the amount of computational resources required for training while providing the ability to generate high-quality audio from text prompts.

[0026] Exemplary embodiments of this disclosure will now be discussed in further detail with reference to the accompanying drawings.

[0027] Example devices and systems

[0028] Figure 1A A block diagram of an example computing system 100 for training a generative model to generate audio data from text content, according to an example embodiment of the present disclosure, is depicted. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0029] User computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop computer or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0030] User computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 may store data 116 and instructions 118, which are executed by processor 112 to cause user computing device 102 to perform operations.

[0031] In some implementations, the user computing device 102 may store or include one or more models 120. For example, model 120 may or may otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, diffuse networks, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models (e.g., transformer models).

[0032] Specifically, in some implementations, model 120 may be a diffusion model, or otherwise include a diffusion model. As described herein, a diffusion model typically refers to a generative machine learning model that works to generate audio by iteratively denoising random noise. For example, the input to a diffusion model may be a conditional signal. The time step of random sampling and via a normal or Gaussian diffusion process (using time-varying methods) The standard deviation of noise Parameterized noise scheduling) corrupts the original samples The obtained samples .time The range can be set to [0,1], from which uniform sampling can be performed during training.

[0033] The distribution can be obtained through a single noise vector that belongs to the standard normal distribution. To parameterize, because It can be written as original samples, deterministic noise scheduling, and noise vectors. The function that makes The loss function used to train model 120 can be defined as follows: ,in This represents a fixed weight function that can be used as or considered a hyperparameter. The training of models 120 and / or 140 will be discussed in more detail with regard to model trainer 160.

[0034] In some implementations, the user computing device 102 can be controlled by time. Random noise is acquired at a location (e.g., via an audio sampler capable of sampling random audio) and model 120 is used to denoise the random noise based on noise predictions provided by model 120. For example, ancestor sampling can be used to control the quality of the generated audio data. For example, the randomness parameter of the audio sampler can be adjusted. This controls the degree of randomness in the denoising process. Other examples include denoising step size (e.g., how frequently denoising occurs), variance scheduling, loss weights, etc.

[0035] In some implementations, one or more models 120 may be received from server computing system 130 via network 180, stored in user computing device memory 114, and then used by one or more processors 112 or otherwise implemented. In some implementations, user computing device 102 may implement multiple parallel instances of a single model 120 or a single set of models 120 (e.g., multiple instances across a model or set of models perform parallel generation of audio data from text prompts).

[0036] More specifically, in some implementations, model 120 may include a large language model (LLM). An LLM can be used to generate descriptive sentences for music (e.g., general descriptive sentences for many different types of audio data). Model 120 may include a machine learning audio classification model. Model 120 may include a machine learning generative model (e.g., a diffusion model, etc.) and a machine learning diffusion model. The machine learning generative model and the machine learning diffusion model can be combined to generate high-quality audio data based on text content. For example, a user of user computing device 102 may (e.g., using user input component 122, etc.) provide text content including a query for a specific type of music. The user computing device can process the text content using the machine learning generative model of model 120 to obtain an intermediate representation of the text content. The user computing device 102 can then process the intermediate representation using the machine learning diffusion model of model 120 to obtain high-quality audio corresponding to the query.

[0037] Alternatively or concurrently, one or more models 140 may be included in or otherwise stored and implemented by the server computing system 130, which communicates with the user computing device 102 in a client-server relationship. For example, model 140 may be implemented by the server computing system 130 as part of a web service (e.g., an audio generation service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0038] User computing device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). Touch-sensitive components can be used to implement a virtual keyboard. Other example user input components include microphones, conventional keyboards, or other devices that a user can use to provide user input.

[0039] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 may store data 136 and instructions 138, which are executed by processor 132 to cause server computing system 130 to perform operations.

[0040] In some implementations, the server computing system 130 includes one or more server computing devices or is otherwise implemented by such one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, these server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0041] As described above, the server computing system 130 may store or otherwise include one or more models 140. For example, model 140 may or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models (e.g., transformer models).

[0042] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150, which is communicatively coupled through network 180. Training computing system 150 may be separate from or part of server computing system 130.

[0043] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158, which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices or is otherwise implemented by such one or more server computing devices.

[0044] The training computing system 150 may include a model trainer 160 that uses various training or learning techniques, such as, for example, error backpropagation, to train machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters across multiple training iterations.

[0045] In some implementations, performing error backpropagation may include performing backpropagation through time with truncation. The model trainer 160 may perform various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0046] Specifically, model trainer 160 can train models 120 and / or 140 based on training dataset 162. Training data 162 may include, for example, audio samples, corpora of text content, audio recordings, etc. More specifically, for example, training data may include audio samples and corresponding descriptive text from audiovisual hosting entities (e.g., video hosting sites, etc.). For example, audio samples may be extracted from music videos, and descriptive text may be extracted from text content provided by users for those music videos. Model trainer 160 can use the audio samples and descriptive text content to train machine learning audio classification models of models 120 / 140.

[0047] Training data 162 may also include a corpus of text data generated using the LLM of model 120 / 140. Training data 162 may include a large number of audio recordings. Model trainer 160 can use the corpus of audio recordings and text data to train the machine learning generative model and machine learning diffusion model of model 120 / 140.

[0048] In some implementations, training examples may be provided by the user computing device 102 if the user has provided consent. Therefore, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process may be referred to as model personalization.

[0049] Model trainer 160 includes computer logic for providing the desired functionality. Model trainer 160 may be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0050] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. Generally, communication over Network 180 can be conducted using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL) via any type of wired and / or wireless connection.

[0051] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0052] In some implementations, the input to the machine learning model of this disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate output. For example, the machine learning model may process natural language data to generate a language-encoded output. For another example, the machine learning model may process text or natural language data to generate a latent text embedding output. For another example, the machine learning model may process text or natural language data to generate a translation output. For another example, the machine learning model may process text or natural language data to generate a classification output. For another example, the machine learning model may process text or natural language data to generate a text segmentation output. For another example, the machine learning model may process text or natural language data to generate a semantic intent output. For another example, the machine learning model may process text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). For yet another example, the machine learning model may process text or natural language data to generate a predictive output.

[0053] In some implementations, the input to the machine learning model of this disclosure may be latent encoded data (e.g., a latent spatial representation of the input). The machine learning model may process the latent encoded data to generate an output. As an example, the machine learning model may process the latent encoded data to generate an identification output. As another example, the machine learning model may process the latent encoded data to generate a reconstruction output. As another example, the machine learning model may process the latent encoded data to generate a search output. As another example, the machine learning model may process the latent encoded data to generate a re-clustering output. As yet another example, the machine learning model may process the latent encoded data to generate a prediction output.

[0054] Figure 1A An example computing system that can be used to implement this disclosure is shown. Other computing systems may also be used. For example, in some implementations, user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, model 120 may be trained locally at user computing device 102 and both may be used. In some such implementations, user computing device 102 may implement model trainer 160 to personalize model 120 based on user-specific data.

[0055] Figure 1B A block diagram of an example computing device 10, which performs high-quality generation of audio data conditionally based on text prompts according to an exemplary embodiment of the present disclosure, is depicted. The computing device 10 may be a user computing device or a server computing device.

[0056] The computing device 10 includes multiple applications (e.g., application 1 to application N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0057] like Figure 1B As shown, each application can communicate with multiple other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0058] Figure 1C A block diagram of an example computing device 50, according to an exemplary embodiment of the present disclosure, is depicted for training a machine learning model to generate high-quality audio data. The computing device 50 may be a user computing device or a server computing device.

[0059] The computing device 50 includes multiple applications (e.g., application 1 to application N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application may use an API (e.g., a common API across all applications) to communicate with the central intelligence layer (and the models stored therein).

[0060] The central intelligence layer comprises multiple machine learning models. For example, such as... Figure 1C As shown, a corresponding machine learning model can be provided for each application, and the corresponding machine learning model can be managed by a central intelligent layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligent layer can provide a single model for all applications. In some implementations, the central intelligent layer is included within or otherwise implemented by the operating system of the computing device 50.

[0061] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized repository for data from the computing device 50. For example... Figure 1C As shown, the central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0062] Figure 2A data flow diagram 200 depicts an example embodiment of this disclosure for training a joint audio-text embedding model for audio classification. More specifically, the computational system (e.g., Figure 1A The training computing system 150, etc., can obtain a set of corresponding audio samples 202 and descriptive text content 204 from the audiovisual hosting entity 205 (e.g., a video hosting site, etc.). For example, the audio samples 202 may be sampled from music videos hosted by the audiovisual hosting entity, and the set of corresponding descriptive text content 204 may be comments, titles, descriptions, etc. provided by users of the audiovisual hosting entity for the corresponding music videos.

[0063] The computational system can train a machine learning audio classification model 206 (e.g., a joint audio-text embedding model) using a set of corresponding audio samples 202 and text content 204. More specifically, the computational system can process the audio samples 202 using the audio embedding part 208 of the machine learning audio classification model 206 to obtain audio embeddings 212. The computational system can process the set of descriptive text content 204 using the text embedding part 210 of the machine learning audio classification model 206 to obtain text embeddings 214. The computational system can use a contrastive loss function 218 to evaluate the audio embeddings 212 and text embeddings 214 within the embedding space 216. Based on the differences, the computational system can train the machine learning audio classification model 206.

[0064] Figure 3 A data flow diagram 300 depicts an example embodiment of the present disclosure for training a machine learning generative model and a machine learning diffusion model to generate high-quality audio data conditionally based on text prompts. More specifically, a computing system (e.g., Figure 1A The training computational system 150, etc., can utilize a machine learning text generation model 302 (e.g., a conventional large language model) to generate a corpus 304 of text data. The corpus 304 of text data may include multiple sentences. These sentences can generally describe various types of audio data, characteristics of the audio data, etc. For example, if the desired type of audio data is music, the corpus 304 of text data may include sentences generally describing music (e.g., "some types of music utilize synths, violins, piano, drums, and vocals!"). The computational system can also obtain a series of audio recordings 306. Audio recordings 306 can be any type or kind of audio recording desired for the model's training. For example, audio recording 306 can be a 30-second clip of music.

[0065] It can be used (for example, as about) Figure 2A machine learning audio classification model 206 (described) processes a corpus 304 of audio recordings 306 and text data to obtain training data 308. Training data 308 may include multiple training pairs, each pair comprising an audio recording 308A and a descriptive sentence 308B that most accurately describes audio recording 308A. More specifically, the machine learning audio classification model 206 can be used to evaluate the corpus 304 of text data about audio recordings 306 within the embedding space of the machine learning audio classification model 206. By doing so, the computational system can determine the sentence that most accurately describes each audio recording 306. For example, for the training pairs of training data 308, sentence 308B of the training pair would be the sentence that is closest to audio recording 308A within the embedding space of the machine learning audio classification model 206.

[0066] The computing system can process sentence 308B using machine learning generative model 310. Machine learning generative model 310 can be any type or kind of generative model. For example, machine learning generative model 310 can be a diffusion model or a cascaded diffusion model. Machine learning generative model 310 can output an intermediate representation 312 of sentence 308B. Intermediate representation 312 can be any type of intermediate representation (e.g., low-fidelity audio samples, spectrograms, embeddings, latent representations, etc.). For example, sentence 308B can be vectorized and fed as a cross-attention sequence to machine learning generative model 310. As output, machine learning generative model 310 can output intermediate representation 312, which can be or otherwise include a "low-fidelity" (e.g., ~3kHz) audio signal.

[0067] The computing system can process the intermediate representation 312 using a machine learning diffusion model 314. The machine learning diffusion model 314 (e.g., a cascaded diffusion model) can output audio data 316. It should be noted that the audio data 316 can be any type or kind of audio data (e.g., an .mp3 file, raw audio data, 16kHz audio, etc.). Following the previous example, the machine learning diffusion model can process the intermediate representation 312 (e.g., “low-fidelity” audio) and output audio data 316 that includes audio with a higher fidelity (e.g., ~16kHz) than the low-fidelity audio of the intermediate representation 312. In some implementations, the machine learning diffusion model 314 can upsample the intermediate representation 312 to generate the audio data 316. For example, the machine learning diffusion model 314 can upsample the 3kHz audio included in the intermediate representation 312 by applying a Fast Fourier Transform (FFT) to the audio sequence and then applying an inverse FFT to obtain higher-fidelity audio based on the lower-fidelity Fourier coefficients.

[0068] Alternatively, in some implementations, the machine learning generative model 310 may be a spectrogram model trained to generate a spectrogram representation of audio based on input data (e.g., a vectorized representation of sentence 308B). For example, the intermediate representation 312 may be a spectrogram comprising ~80 channels and ~100 features per second. In some implementations, the pixel values ​​of the spectrogram may be normalized to the range [-1, 1]. Similarly, in some implementations, the machine learning diffusion model 314 may be, or otherwise includes, a vocoder model capable of processing the spectrogram to generate higher fidelity audio. In some implementations, the machine learning diffusion model 314 may specifically process the spectrogram (e.g., the intermediate representation 312) to generate audio data 316, rather than conditionally processing the intermediate representation 312 based on a corpus 304 of sentence 308B or text data.

[0069] In some implementations, both the generative model 310 and the diffusion model 314 may or otherwise include a diffusion model, a diffusion layer, or a similar layer conventionally included in a diffusion model or the like.

[0070] The computational system can use loss function 318 to evaluate the difference between audio data 316 and audio recordings 308A of training data 308. Based on this difference, the computational system can adjust the values ​​of one or more parameters of the machine learning diffusion model 314. Furthermore, in some implementations, the computational system can also adjust the values ​​of one or more parameters of the machine learning generative model 310 in an end-to-end manner. In this way, the computational system can train the machine learning generative model 310 and the machine learning diffusion model 314 to jointly generate high-quality audio data 316 from text content (e.g., sentence 308B).

[0071] Figure 4 A data flow diagram 400 depicts the use of a machine learning generative model and a machine learning diffusion model, jointly trained to generate high-quality audio data, from text prompts according to an example embodiment of this disclosure. Specifically, a computing system (e.g., Figure 1A A server computing system 130, etc., can obtain text content 402. The text content 402 can describe a query 404 indicating the desired type of audio content. Following the depicted example, query 404 can indicate an expectation of “relaxing music without vocals… something with piano”. A machine learning generative model 310 can process the text content 402 to obtain an intermediate representation 406 (e.g., as about…). Figure 3The machine learning diffusion model 314 can process intermediate representations to obtain audio data 408. Audio data 408 may be, for example, 30 seconds of music without vocals, including piano instruments. In this way, the machine learning generative model 310 and the machine learning diffusion model 314 can be combined to generate audio data that semantically matches the request described by the text content 402.

[0072] Figure 5 A flowchart depicts an example method 500 for training a machine learning diffusion model to generate high-quality audio data, according to an example embodiment of this disclosure. Although Figure 5 The steps performed in a particular order are described for illustrative and discussion purposes, but the method disclosed herein is not limited to the particular order or arrangement described. The steps of method 500 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of this disclosure.

[0073] At position 502, the computational system can use a machine learning text generation model to generate a corpus of text data. The corpus of text data can include multiple sentences. Each sentence can describe the type of audio.

[0074] At position 504, the computational system can obtain multiple audio recordings. In some implementations, audio recordings can be sampled from various music recordings. For example, the multiple audio recordings could be clips of 10 seconds, 20 seconds, 30 seconds, etc., from various songs. For each of the multiple audio recordings, the computational system can process the audio recording using a machine learning audio classification model to obtain training data. The training data may include the audio recordings and one or more sentences within the joint audio-text embedding space of the machine learning audio classification model that are closest to that audio recording.

[0075] In some implementations, before processing the audio recordings with a machine learning audio classification model, the computational system may obtain an associated corpus of multiple audio samples and descriptive text data from an audiovisual data hosting entity. For each of the multiple audio samples, the corpus of descriptive text data may include one or more portions of text content describing the audio recording, provided by a user of the audiovisual data hosting entity. For each of the multiple audio samples, the computational system may process the audio sample using the audio embedding portion of the machine learning audio classification model to obtain an audio embedding. The computational system may then process one or more portions of the text content describing the audio recording using the text embedding portion of the machine learning audio classification model to obtain a text embedding, and train the machine learning audio classification model using a contrastive loss function that evaluates the difference between the audio embedding and the text embedding.

[0076] At point 506, the computational system can use a machine learning generative model to process one or more sentences to obtain intermediate representations of those sentences.

[0077] At point 508, the computational system can process the intermediate representation using a machine learning cascaded diffusion model to obtain the audio data. In some implementations, processing the intermediate representation with a machine learning cascaded diffusion model may include applying a Gaussian diffusion process to the audio recording, and processing the audio recording and conditional signal with the machine learning cascaded diffusion model to obtain the audio data.

[0078] In some implementations, the intermediate representation of the text content may be a spectrogram, and the computing system may use a machine learning diffusion model to process the spectrogram to obtain audio data. Alternatively, in some implementations, the intermediate representation of the text content may include a low-fidelity audio signal, and the computing system may use a machine learning diffusion model to process the low-fidelity audio signal and the text content to obtain audio data.

[0079] At point 510, the computational system can train a machine learning cascade diffusion model based on the differences between the audio data and the audio recording. In some implementations, the computational system can train both a machine learning generative model and a machine learning cascade diffusion model based on the differences between the audio data and the audio recording.

[0080] In some implementations, the computational system may further obtain text content describing the query. This query may indicate the desired type of audio content. The computational system may process the text content using a machine learning generative model to generate an intermediate representation of the text content, and then process this intermediate representation using a machine learning diffusion model to obtain audio data. This audio data may include audio of the desired type.

[0081] Additional Public Content

[0082] This paper discusses technologies related to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and divisions of tasks and functions between and within components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0083] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of illustration and not limitation. Modifications, variations, and equivalents to such embodiments will be readily apparent to those skilled in the art upon understanding the foregoing. Therefore, this disclosure does not exclude such modifications, variations, and / or additions to the subject matter that will be readily understood by those of ordinary skill in the art. For example, features shown or described as part of one embodiment may be used with another embodiment to produce yet further embodiments. Therefore, this disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. A computing system, comprising: One or more processors; A machine learning generator model, which is trained to generate intermediate representations of text content; A machine learning diffusion model trained to generate audio data from an intermediate representation of the text content, wherein the audio data responds to a query described by the text content; and One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the participating computing device to perform operations, the operations including: The machine learning generator model processes the text content to generate an intermediate representation of the text content, wherein the text content description indicates a query of the desired type for the audio content, and wherein the intermediate representation includes a low-fidelity audio signal or a spectrogram; and The intermediate representation is processed using the machine learning diffusion model to obtain audio data, wherein the audio data includes audio of the desired type with audio content, and wherein processing the intermediate representation using the machine learning diffusion model includes: The low-fidelity audio signal or at least one of the spectrograms is processed using the machine learning diffusion model to obtain the audio data.

2. The computing system of claim 1, wherein the machine learning diffusion model includes a machine learning cascade diffusion model, and the machine learning cascade diffusion model includes one or more attention mechanisms.

3. The computing system of claim 1, wherein processing the text content with the machine learning generator model further comprises: A Gaussian diffusion process is applied to the intermediate representation.

4. The computing system of claim 1, wherein the method comprises, before processing the text content with the machine learning generator model: A corpus of text data is generated using a machine learning text generation model, wherein the corpus of text data consists of multiple sentences, and each sentence describes the type of audio. For each audio record in a set of multiple audio recordings: The audio record is processed using a machine learning audio classification model to obtain training data, the training data including the audio record and one or more sentences within the joint audio-text embedding space of the machine learning audio classification model that are closest to the audio record; and The training data is used to train at least one of the machine learning generator model or the machine learning diffusion model.

5. The computing system of claim 1, wherein the query described by the text content includes one or more characteristics of music; and The audio data mentioned therein includes music having at least one of the one or more of the aforementioned characteristics.

6. A computer-implemented method, comprising: A corpus of text data generated by a computing system comprising one or more computing devices using a machine learning text generation model, wherein the corpus of text data comprises multiple sentences, and wherein each sentence describes the type of audio. For each audio record in a set of multiple audio recordings: The computing system processes the audio record using a machine learning audio classification model to obtain training data, the training data including the audio record and one or more sentences that are closest to the audio record among the plurality of sentences in the joint audio-text embedding space of the machine learning audio classification model; The computing system processes the one or more sentences using a machine learning generation model to obtain intermediate representations of the one or more sentences, wherein the intermediate representations include low-fidelity audio signals or spectrograms; The intermediate representation is processed by the computing system using a machine learning cascaded diffusion model to obtain audio data, wherein processing the intermediate representation using the machine learning cascaded diffusion model includes: processing at least one of the low-fidelity audio signal or the spectrogram using the machine learning cascaded diffusion model to obtain the audio data; and The computing system trains the machine learning cascade diffusion model based on the difference between the audio data and the audio recording.

7. The computer-implemented method of claim 6, wherein training the machine learning cascade diffusion model comprises training the machine learning generative model and the machine learning cascade diffusion model by the computing system based on the difference between the audio data and the audio recording.

8. The computer-implemented method of claim 6, wherein processing the intermediate representation with the machine learning cascade diffusion model comprises: The computing system applies a Gaussian diffusion process to the audio recording; as well as The audio data is obtained by the computing system processing the audio recording and conditional signal using the machine learning cascade diffusion model.

9. The computer-implemented method of claim 6, wherein before processing the audio recording with a machine learning audio classification model, the method comprises: The computing system obtains an associated corpus of multiple audio samples and descriptive text data from an audiovisual data hosting entity, wherein for each of the multiple audio samples, the corpus of descriptive text data includes one or more portions of text content provided by a user of the audiovisual data hosting entity to describe the audio recording; For each of the plurality of audio samples: The computing system processes the audio samples using the audio embedding part of the machine learning audio classification model to obtain audio embeddings; The computing system processes one or more portions of the text content describing the audio recording using the text embedding part of the machine learning audio classification model to obtain the text embedding; as well as The computing system uses a contrastive loss function to train the machine learning audio classification model, which evaluates the difference between the audio embedding and the text embedding.

10. The computer-implemented method of claim 6, wherein the method further comprises: The computing system obtains text content describing the query, wherein the query indicates the desired type of audio content; The computing system processes the text content using the machine learning generator model to generate an intermediate representation of the text content; as well as The intermediate representation is processed by the computing system using the machine learning cascade diffusion model to obtain audio data, wherein the audio data includes audio of the desired type of audio content.

11. One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the participating computing device to perform operations, the operations including: A machine learning generator model is used to process text content to generate an intermediate representation of the text content, wherein the text content describes a query of a desired type indicating audio content, wherein the intermediate representation includes a low-fidelity audio signal or spectrogram, and wherein the machine learning generator model is trained to generate the intermediate representation of the text content; and The intermediate representation is processed using a machine learning diffusion model to obtain audio data, wherein the audio data includes audio of a desired type of audio content, and wherein the machine learning diffusion model is trained to generate audio data from an intermediate representation of text content, and wherein processing the intermediate representation using the machine learning diffusion model includes: processing at least one of the low-fidelity audio signal or the spectrogram using the machine learning diffusion model to obtain the audio data.

12. The one or more non-transitory computer-readable media of claim 11, wherein the machine learning diffusion model includes a machine learning cascade diffusion model, and the machine learning cascade diffusion model includes one or more attention mechanisms.

13. One or more non-transitory computer-readable media as claimed in claim 11, wherein processing the text content with the machine learning generator model further comprises: A Gaussian diffusion process is applied to the intermediate representation.

14. One or more non-transitory computer-readable media as claimed in claim 11, wherein the method comprises: A corpus of text data is generated using a machine learning text generation model, wherein the corpus of text data consists of multiple sentences, and each sentence describes the type of audio. For each audio record in a set of multiple audio recordings: The audio record is processed using a machine learning audio classification model to obtain training data, the training data including the audio record and one or more sentences within the joint audio-text embedding space of the machine learning audio classification model that are closest to the audio record; and The training data is used to train at least one of the machine learning generator model or the machine learning diffusion model.

Images