A computing system, a computer-implemented method, and one or more non-transient computer-readable means for generating audio data based on descriptive textual prompts.

Description

1 / 22 A COMPUTER SYSTEM, A COMPUTER-IMPLEMENTED METHOD, AND ONE OR MORE NON-TRANSIENT COMPUTER-READABLE MEANS FOR GENERATING AUDIO DATA BASED ON DESCRIPTIVE TEXT PROMPTS PRIORITY CLAIM

[0001] This application is based on and claims priority to U.S. Provisional Application 63 / 481,746, with a filing date of January 26, 2023. The Applicant claims priority and the benefit of the application and incorporates the application herein by reference in its entirety. FIELD

[0002] This disclosure relates generally to generative models. More specifically, this disclosure relates to diffusion models trained to generate audio data based on descriptions of audio types. FUNDAMENTALS

[0003] Generative models have proven increasingly useful in the field of machine learning. Deep generative models are used in a wide range of domains. Some types of generative models, such as diffusion models, can be trained to generate high-quality outputs in a format different from the input format. For example, some diffusion models can be trained to process a textual prompt to generate high-quality image outputs. SUMMARY

[0004] Aspects and advantages of the modalities of this disclosure will be set forth in part in the description below, or may be learned from the description, or may be learned through practice of the modalities.

[0005] An exemplary aspect of the present disclosure relates to a computing system. The computing system includes one or more processors. The computing system includes a machine-learned generative model trained to generate intermediate representations of textual content. The computing system includes a machine-learned broadcast model trained to generate audio data from intermediate representations of textual content, wherein the audio data responds to a request described by the textual content. The computing system includes one or more non-transient computer-readable media that store instructions which, when executed by one or more Petition 870260065899, dated 03 / 07 / 2026, page 20 / 77 2 / 22 processors cause the participating computing device to perform operations. The operations include processing textual content with the machine-learned generator model to generate an intermediate representation of the textual content, where the textual content is descriptive of a request indicating a desired type of audio content. The operations also include processing the intermediate representation with the machine-learned broadcast model to obtain audio data, where the audio data comprises audio of the desired type of audio content.

[0006] Another example aspect of the present disclosure relates to a computer-implemented method. The method includes generating, by a computing system comprising one or more computing devices, a corpus of textual data with a machine-learned text generation model, wherein the corpus of textual data comprises a plurality of sentences and wherein each sentence is descriptive of a type of audio. The method includes, for each of a plurality of audio recordings, processing, by the computing system, the audio recording with a machine-learned audio classification model to obtain training data comprising the audio recording and one or more sentences from the plurality of sentences closest to the audio recording within an audio-text embedding space together with the machine-learned audio classification model.The method includes processing, by the computing system, one or more sentences with a machine-learned generation model to obtain an intermediate representation of one or more sentences. The method includes processing, by the computing system, the intermediate representation with a machine-learned cascade diffusion model to obtain audio data. The method includes training, by the computing system, the machine-learned cascade diffusion model based on the difference between the audio data and the audio recording.

[0007] Another example aspect of the present disclosure relates to one or more non-transient computer-readable media that store instructions which, when executed by one or more processors, cause the participating computing device to perform operations. The operations include processing textual content with a machine-learned generator model to generate an intermediate representation of the textual content, wherein the textual content is Petition 870260065899, dated 03 / 07 / 2026, page 21 / 77 3 / 22 describes a request that indicates a desired type of audio content and in which the machine-learned generator model is trained to generate intermediate representations of textual content. The operations include processing the intermediate representation with a machine-learned diffusion model to obtain audio data, where the audio data comprises audio of the desired type of audio content and where the machine-learned diffusion model is trained to generate audio data from intermediate representations of textual content.

[0008] Other aspects of this disclosure are directed to various systems, apparatus, non-transient computer-readable media, user interfaces and electronic devices.

[0009] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated into and form a part of this descriptive report, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the related principles. BRIEF DESCRIPTION OF THE FIGURES

[0010] A detailed discussion of the modalities directed to someone skilled in the art is set out in the descriptive report, which refers to the attached figures, in which:

[0011] Figure 1A depicts a block diagram of an example computing system that performs generative model training for generating audio data from textual content according to example modalities of this disclosure.

[0012] Figure 1B depicts a block diagram of an example computing device that performs high-quality audio data generation conditioned on a text prompt according to example embodiments of the present disclosure.

[0013] Figure 1C depicts a block diagram of an example computing device that performs machine learning model training for generating high-quality audio data according to example modalities of the present disclosure. Petition 870260065899, dated 03 / 07 / 2026, page 22 / 77 4 / 22

[0014] Figure 2 depicts a data flow diagram for training an audio-text embedding model for audio classification according to example modalities of the present disclosure.

[0015] Figure 3 depicts a data flow diagram for training a machine-learned generation model and a machine-learned diffusion model for generating high-quality audio data conditioned on textual prompts according to example modalities of the present disclosure.

[0016] Figure 4 depicts a data flow diagram for generating high-quality audio from a textual prompt using a machine-learned generation model and a machine-learned diffusion model collectively trained for generating high-quality audio data according to example embodiments of the present disclosure.

[0017] Figure 5 depicts a flow diagram of an example method for training a machine-learned diffusion model to generate high-quality audio data according to example modalities of the present disclosure.

[0018] Reference numbers that are repeated in plural figures are intended to identify the same features in multiple implementations. DETAILED DESCRIPTION Overview

[0019] Generally, this disclosure is directed toward diffusion models trained to generate audio data based on descriptions of audio types. More specifically, generative models with the ability to produce high-quality audio from a different input type, such as a textual prompt, are highly desirable. However, there are a number of roadblocks to implementing such models with a sufficient degree of accuracy. As one example, training a model to generate audio data from a textual prompt requires large amounts of textual descriptions of audio clips, which can be prohibitively difficult to extract (or generate). As another example, the architectures conventionally used for such models have generally failed to produce audio with a sufficient degree of accuracy.

[0020] Therefore, the implementations of this disclosure propose diffusion models for generating audio data based on textual prompts. Petition 870260065899, dated 03 / 07 / 2026, page 23 / 77 5 / 22 Descriptive. More specifically, a computing system can obtain a large number of audio samples and an associated corpus of descriptive textual data. For example, the computing system can extract the audio samples from the audio data of videos hosted by an audiovisual data hosting entity, and the computing system can extract the corresponding corpus of descriptive textual data from the textual content provided by users to describe the respective audio samples (e.g., a music video and the comments provided for the music video by users). The computing system can use the audio samples and the corpus of descriptive textual data to train a machine-learned audio classification model using a contrastive loss function (e.g., a model embedding audio-text together, etc.).

[0021] The computing system can drive the machine-learned audio classification model to train and optimize a model for audio generation. More specifically, the computing system can first generate a corpus of textual data with a machine-learned text generation model (e.g., a large language model (LLM), etc.). The textual data corpus can include a large number of sentences, each describing a type of audio (e.g., “A light EDM beat has a bass, strings, and a simple piano”). The computing system can also obtain a large number of audio recordings. The computing system can evaluate the audio recordings and the textual data corpus with the machine-learned audio classification model to select the most accurate pairs of audio recordings and descriptive sentences for training data.

[0022] For each of the pairs of audio recordings and textual content (i.e., sentences), the computing system can first process the textual content with a machine-learned generation model to obtain an intermediate representation of the textual content (e.g., a latent representation, a low-fidelity audio sample, a spectrogram, etc.). The computing system can then process the intermediate representation with a machine-learned diffusion model to obtain audio data. The computing system can train the machine-learned diffusion model and / or the machine-learned generation model with a loss function that evaluates the difference between the Petition 870260065899, dated 03 / 07 / 2026, page 24 / 77 6 / 22 audio data and the audio recording that corresponds to the textual content. In this way, the implementations of this disclosure can efficiently and effectively train a series of models for generating high-quality audio samples conditioned to the textual content.

[0023] The aspects of the present disclosure provide a number of technical effects and benefits. As an example of a technical effect and benefit, conventional generative models generally produce low-quality audio data from textual prompts or have no capability to produce audio data from textual content. Furthermore, training such models requires substantial amounts of computing resources (e.g., power, memory, energy, computing cycles, bandwidth, etc.). However, the implementations of the present disclosure provide the ability to generate training data specifically tailored to more effectively and efficiently train a new architecture for generating high-quality audio data, thereby reducing the amount of computing resources required for training, while simultaneously providing the ability to generate high-quality audio from textual prompts.

[0024] With reference now to the Figures, example forms of the present disclosure will be discussed in more detail. Example Devices and Systems

[0025] Figure 1A depicts a block diagram of an example computing system 100 that performs generative model training for generating audio data from textual content according to example embodiments of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled via a network 180.

[0026] User computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device. Petition 870260065899, dated 03 / 07 / 2026, page 25 / 77 7 / 22

[0027] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or a plurality of processors that are operatively connected. The memory 114 may include one or more non-transient computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0028] In some implementations, the user computing device 102 may store or include one or more models 120. For example, the models 120 may be or may otherwise include various machine-learned models, such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, broadcast networks, or other forms of neural networks. Some examples of machine-learned models may drive an attention mechanism, such as self-attention. For example, some example machine-learned models may include multi-headed self-attention models (e.g., transformer models).

[0029] Specifically, in some implementations, 120 models may be, or otherwise include, diffusion models. As described in this document, diffusion models generally refer to machine-learned generative models that function to generate audio by iterative noise removal of random noise. For example, the input to a diffusion model might be a conditioning signal c, a randomly sampled time step t, and a sample xt obtained by corrupting the original sample x through a normal or Gaussian diffusion process with a noise schedule parameterized by the standard deviation σί of the noise at time t. The time interval t can be defined as [0,1] from which uniform sampling can occur during training. Petition 870260065899, dated 03 / 07 / 2026, page 26 / 77 8 / 22

[0030] The dext distribution can be parameterized by a single noise vector belonging to a standard normal distribution, as xc can be written as a function of the original sample, the deterministic noise schedule, and the noise vector, so that xt(XY). The loss function used to train the 120 models can be defined as leti(xtYY)- elI2], where I represents a fixed weight function that can serve as, or be viewed as, a hyperparameter. The training of the 120 and / or 140 models will be discussed in more detail with respect to the 160 model trainer.

[0031] In some implementations, the user computing device 102 may utilize the models 120 by taking random noise at time t = 1 (e.g., through an audio sampler capable of sampling random audio, etc.) and removing the random noise based on the noise predictions provided by the model 120. For example, ancestral sampling can be used to control the quality of the generated audio data. For example, the degree of stochasticity of the noise removal process can be controlled by adjusting a stochasticity parameter Y of the audio sampler. Other examples include the size of the noise removal step (e.g., how often noise removal occurs), variance schedule, loss weight, etc.

[0032] In some implementations, one or more models 120 may be received from the server computing system 130 via the network 180, stored in the memory of the user computing device 114, and then used or otherwise implemented by one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single model 120 or a single set of models 120 (for example, to perform parallel generation of audio data from textual prompts via multiple model instances or sets of models).

[0033] More specifically, in some implementations, 120 models may include a large language model (LLM). The LLM can be used to generate descriptive phrases of music (e.g., generic descriptive phrases of many different types of audio data). 120 models may include a machine-learned audio classification model. 120 models may include a machine-learned generation model (e.g., a diffusion model, etc.). Petition 870260065899, dated 03 / 07 / 2026, page 27 / 77 9 / 22 and a machine-learned diffusion model. The machine-learned generation model and the machine-learned diffusion model can be used together to generate high-quality audio data based on textual content. For example, a user of user computing device 102 can provide textual content that includes a request for a particular type of music (e.g., with user input component 122 etc.). The user computing device can process the textual content with the machine-learned generative model of models 120 to obtain an intermediate representation of the textual content. The user computing device 102 can then process the intermediate representation with the machine-learned diffusion model of models 120 to obtain high-quality audio that matches the request.

[0034] Additionally or alternatively, one or more models 140 may be included or otherwise stored and implemented by the server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the models 140 may be implemented by the server computing system 140 as a portion of a web service (e.g., an audio generation service). Thus, one or more models 120 may be stored and implemented in the user computing device 102 and / or one or more models 140 may be stored and implemented in the server computing system 130.

[0035] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a pen). The touch-sensitive component may serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user may provide user input.

[0036] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (for example, a core of Petition 870260065899, dated 03 / 07 / 2026, page 28 / 77 10 / 22 processor, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or a plurality of processors that are operatively connected. Memory 134 may include one or more non-transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0037] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In cases where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0038] As described above, the server computing system 130 may store or otherwise include one or more models 140. For example, the models 140 may be or may otherwise include various machine-learned models. Examples of machine-learned models include neural networks or other multi-layered nonlinear models. Examples of neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some examples of machine-learned models may drive an attention mechanism, such as self-attention. For example, some example machine-learned models may include multi-headed self-attention models (e.g., transformer models).

[0039] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 and interact with the training computing system 150, which is communicatively coupled via the network 180. The training computing system 150 can be separate from the server computing system 130 or it can be a portion of the server computing system 130.

[0040] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (for example, a core of Petition 870260065899, dated 03 / 07 / 2026, page 29 / 77 11 / 22 processor, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or a plurality of processors that are operatively connected. Memory 154 may include one or more non-transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0041] The training computing system 150 may include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored on the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation of errors. For example, a loss function may be backpropagated through the models to update one or more model parameters (e.g., based on a gradient of the loss function). Various loss functions may be used, such as mean squared error, probability loss, cross-entropy loss, joint loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters over a series of training iterations.

[0042] In some implementations, performing backpropagation of errors may include performing truncated backpropagation over time. The 160 model trainer can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization ability of the models to be trained.

[0043] In particular, model trainer 160 can train models 120 and / or 140 based on a training dataset 162. The training dataset 162 can include, for example, audio samples, a corpus of textual content, audio recordings, etc. More specifically, for example, the training dataset can include audio samples and corresponding descriptive text from an audiovisual hosting entity (e.g., a video hosting website, etc.). For example, the audio samples can be Petition 870260065899, dated 03 / 07 / 2026, page 30 / 77 12 / 22 extracted from video clips, and descriptive text can be extracted from user-provided text content for the video clip. Model trainer 160 can train a machine-learned audio classification model from models 120 / 140 using the audio samples and descriptive text content.

[0044] Training data 162 may also include a corpus of textual data generated using an LLM of models 120 / 140. Training data 162 may include a large number of audio recordings. Model trainer 160 may train the machine-learned generation model and the machine-learned diffusion model of models 120 / 140 with the audio recordings and the corpus of textual data.

[0045] In some implementations, if the user has provided consent, the training examples may be provided by the user's computing device 102. Thus, in such implementations, the model 120 provided to the user's computing device 102 may be trained by the training computing system 150 on user-specific data received from the user's computing device 102. In some cases, this process may be referred to as model customization.

[0046] The Model 160 trainer includes computer logic used to provide the desired functionality. The Model 160 trainer can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, the Model 160 trainer includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, the Model 160 trainer includes one or more sets of computer-executable instructions that are stored on a tangible, computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.

[0047] A 180 network can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., internet), or some combination thereof, and can include any number of wired or wireless links. In general, communication over the 180 network can be carried out through any type of wired and / or wireless connection, using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., by Petition 870260065899, dated 03 / 07 / 2026, page 31 / 77 13 / 22 example, VPN, secure HTTP, SSL).

[0048] The machine learning models described in this descriptive report can be used in a variety of tasks, applications and / or use cases.

[0049] In some implementations, the input to the machine-learned models of this disclosure may be text or natural language data. The machine-learned models may process the text or natural language data to generate an output. As an example, the machine-learned models may process the natural language data to generate a language encoding output. As another example, the machine-learned models may process the text or natural language data to generate a latent text embedding output. As another example, the machine-learned models may process the text or natural language data to generate a translation output. As another example, the machine-learned models may process the text or natural language data to generate a classification output.As another example, machine-learned models can process text or natural language data to generate text segmentation output. As another example, machine-learned models can process text or natural language data to generate semantic intent output. As another example, machine-learned models can process text or natural language data to generate expanded text or natural language output (e.g., text or natural language data that is of higher quality than the input text or natural language, etc.). As another example, machine-learned models can process text or natural language data to generate predictive output.

[0050] In some implementations, the input to the machine-learned models of this disclosure may be latent encoding data (e.g., a latent space representation of an input, etc.). The machine-learned models may process the latent encoding data to generate an output. As an example, the machine-learned models may process the latent encoding data to generate a recognition output. As another example, the machine-learned models may process the latent encoding data to generate a reconstruction output. As yet another example, the models Petition 870260065899, dated 03 / 07 / 2026, page 32 / 77 14 / 22 machine-learned models can process latent encoding data to generate a search output. As another example, machine-learned models can process latent encoding data to generate a regrouping output. As yet another example, machine-learned models can process latent encoding data to generate a prediction output.

[0051] Figure 1A illustrates an example computing system that can be used to implement the present disclosure. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include the model trainer 160 and the training dataset 162. In such implementations, the models 120 may be trained and used locally on the user computing device 102. In some of these implementations, the user computing device 102 may implement the model trainer 160 to customize the models 120 based on specific user data.

[0052] Figure 1B depicts a block diagram of an example computing device 10 that performs high-quality audio data generation conditioned on a text prompt according to example embodiments of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[0053] Computing device 10 includes a series of applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine-learned models. For example, each application might include one machine-learned model. Examples of applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0054] As illustrated in Figure 1B, each application can communicate with a number of other computing device components, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0055] Figure 1C depicts a block diagram of a device. Petition 870260065899, dated 03 / 07 / 2026, page 33 / 77 Example 15 / 22 computing device 50 that performs machine learning model training for generating high-quality audio data according to example embodiments of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0056] Computing device 50 includes a series of applications (e.g., applications 1 to N). Each application is in communication with a central intelligence layer. Examples of applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and models stored therein) using an API (e.g., a common API across all applications).

[0057] The core intelligence layer includes a series of machine-learned models. For example, as illustrated in Figure 1C, a respective machine-learned model may be provided for each application and managed by the core intelligence layer. In other implementations, two or more applications may share a single machine-learned model. For example, in some implementations, the core intelligence layer may provide a single model for all applications. In some implementations, the core intelligence layer is included within or otherwise implemented by a computing device operating system 50.

[0058] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized data repository for the computing device 50. As illustrated in Figure 1C, the central device data layer can communicate with a number of other computing device components, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0059] Figure 2 depicts a 200 data flow diagram for training an audio-text embedding model for audio classification according to example modalities of the present disclosure. More specifically, Petition 870260065899, dated 03 / 07 / 2026, page 34 / 77 16 / 22 A computing system (e.g., training computing system 150 of Figure 1A, etc.) can obtain audio samples 202 and corresponding sets of descriptive textual content 204 from an audiovisual hosting entity 205 (e.g., a video hosting website, etc.). For example, the audio samples 202 can be sampled from video clips hosted by the audiovisual hosting entity, and the corresponding sets of descriptive textual content 204 can be comments, titles, descriptions, etc., provided for the corresponding video clip by users of the audiovisual hosting entity.

[0060] The computing system can train a machine-learned audio classification model 206 (e.g., an audio-text embedding model) with audio samples 202 and corresponding sets of textual content 204. More specifically, the computing system can process audio samples 202 with an audio embedding portion 208 of the machine-learned audio classification model 206 to obtain audio embeddings 212. The computing system can process descriptive textual content sets 204 with a text embedding portion 210 of the machine-learned audio classification model 206 to obtain text embeddings 214. The computing system can evaluate audio embeddings 212 and text embeddings 214 within an embedding space 216 using a contrastive loss function 218.Based on the differences, the computer system can train the machine-learned audio classification model 206.

[0061] Figure 3 depicts a data flow diagram 300 for training a machine-learned generation model and a machine-learned diffusion model for generating high-quality audio data conditioned on textual prompts according to example embodiments of the present disclosure. More specifically, a computing system (e.g., the training computing system 150 of Figure 1A, etc.) can utilize a machine-learned text generation model 302 (e.g., a conventional big language model) to generate a textual data corpus 304. The textual data corpus 304 can include a plurality of sentences. The sentences can generically describe various types of audio data, audio data characteristics, etc. For example, if the desired type of audio data is music, the data corpus Petition 870260065899, dated 03 / 07 / 2026, page 35 / 77 Textual 17 / 22 304 may include generically descriptive phrases about music (for example, some types of music use synthesizers, violins, piano, drums, and vocals!). The computer system may also obtain a series of audio recordings 306. Audio recordings 306 can be any type or method of audio recording desired for training the models. For example, audio recordings 306 could be 30-second music clips.

[0062] Audio recordings 306 and textual data corpus 304 can be processed using the machine-learned audio classification model 206 (for example, as described in relation to Figure 2) to obtain training data 308. The training data 308 can include a series of training pairs, each including an audio recording 308A and descriptive phrases 308B that most accurately describe audio recording 308A. More specifically, the machine-learned audio classification model 206 can be used to evaluate the textual data corpus 304 against audio recordings 306 within an embedding space of the machine-learned audio classification model 206. By doing so, the computing system can determine the phrases that most accurately describe each audio recording 306.For example, for a training pair of training data 308, the phrases 308B of the training pair would be those closest to audio recording 308A within the embedding space of the machine-learned audio classification model 206.

[0063] The computing system can process the 308B phrases with a machine-learned generation model 310. The machine-learned generation model 310 can be any type or manner of generative model. For example, the machine-learned generation model 310 can be a diffusion model or a cascade diffusion model. The machine-learned generation model 310 can output an intermediate representation 312 of the 308B phrases. The intermediate representation 312 can be any type of intermediate representation (e.g., a low-fidelity audio sample, a spectrogram, an embedding, a latent representation, etc.). For example, the 308B phrases can be vectorized and fed to the machine-learned generation model 310 as a cross-attention sequence. As an output, the machine-learned generation model 310 can output the intermediate representation 312, which can be or otherwise Petition 870260065899, dated 03 / 07 / 2026, page 36 / 77 18 / 22 include a low-fidelity audio signal (e.g., ~3kHz).

[0064] The computing system can process the intermediate representation 312 with a machine-learned diffusion model 314. The machine-learned diffusion model 314 (e.g., a cascaded diffusion model) can output audio data 316. It should be noted that the audio data 316 can be any type or manner of audio data (e.g., an .mp3 file, raw audio data, 16kHz audio, etc.). To follow the previous example, the machine-learned diffusion model can process the intermediate representation 312 (e.g., low-fidelity audio) and can output audio data 316 that includes audio that is of higher fidelity (e.g., ~16kHz) than the low-fidelity audio of the intermediate representation 312. In some implementations, the machine-learned diffusion model 314 can convert the intermediate representation 312 to generate the audio data 316.For example, the machine-learned diffusion model 314 can convert the 3kHz audio included in the intermediate representation 312 by applying Fast Fourier Transforms (FFTs) to the audio sequence and then applying inverse FFT to obtain higher fidelity audio based on the lower fidelity Fourier coefficients.

[0065] Alternatively, in some implementations, the machine-learned generation model 310 may be a spectrogram model trained to generate an audio spectrogram representation based on the input data (e.g., vectorized representations of phrases 308B etc.). For example, the intermediate representation 312 may be a spectrogram that includes ~80 channels with a frequency of ~100 features per second. In some implementations, the pixel values ​​of the spectrogram may be normalized to lie within [-1,1]. Similarly, in some implementations, the machine-learned diffusion model 314 may be or otherwise include a vocoder model that can process the spectrogram to generate the highest fidelity audio.In some implementations, instead of processing the intermediate representation 312 conditioned on the sentences 308B or the textual data corpus 304, the machine-learned diffusion model 314 may process only the spectrogram (e.g., the intermediate representation 312) to generate the audio data 316.

[0066] In some implementations, both the 310 generation model and the Petition 870260065899, dated 03 / 07 / 2026, page 37 / 77 19 / 22 diffusion model 314 may be, or otherwise include, diffusion models, diffusion layers, or analogous layers conventionally included in diffusion models or similar.

[0067] The computing system can evaluate the difference between the audio data 316 and the audio recording 308A from the training data 308 using the loss function 318. Based on the difference, the computing system can adjust the values ​​of one or more parameters of the machine-learned diffusion model 314. Additionally, in some implementations, the computing system can also adjust the values ​​of one or more parameters of the machine-learned generation model 310 in an end-to-end manner. In such a way, the computing system can train the machine-learned generation model 310 and the machine-learned diffusion model 314 to collectively generate high-quality audio data 316 from textual content (e.g., sentences 308B).

[0068] Figure 4 depicts a 400 data flow diagram for generating high-quality audio from a textual prompt using a machine-learned generation model and a machine-learned diffusion model collectively trained for generating high-quality audio data according to example embodiments of the present disclosure. Specifically, a computing system (e.g., server computing system 130 of Figure 1A, etc.) can obtain 402 textual content. The 402 textual content can describe a 404 request indicating a desired type of audio content. To follow the depicted example, the 404 request might indicate a desire for relaxing music without vocals... something with piano.The machine-learned generation model 310 can process the textual content 402 to obtain an intermediate representation 406 (for example, as described in relation to Figure 3), and the machine-learned diffusion model 314 can process the intermediate representation to obtain audio data 408. The audio data 408 could be, for example, 30 seconds of music without vocals that includes piano instrumentals. In this way, the machine-learned generation model 310 and the machine-learned diffusion model 314 can, together, generate audio data that semantically corresponds to the requests described by the textual content 402.

[0069] Figure 5 depicts a flow diagram of an example method 500 for training a machine-learned diffusion model for Petition 870260065899, dated 03 / 07 / 2026, page 38 / 77 20 / 22 generation of high-quality audio data according to example embodiments of the present disclosure. Although Figure 5 depicts steps performed in a specific order for illustrative and discussion purposes, the methods of the present disclosure are not limited to the order or arrangement particularly illustrated. The various steps of method 500 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0070] In 502, a computer system can generate a corpus of textual data with a machine-learned text generation model. The textual data corpus can include a plurality of sentences. Each sentence can be descriptive of a type of audio.

[0071] In 504, the computing system can obtain a plurality of audio recordings. In some implementations, the audio recordings can be sampled from multiple music recordings. For example, the plurality of audio recordings might be 10-second, 20-second, 30-second, etc. clips from various songs. The computing system can, for each of the plurality of audio recordings, process the audio recording with a machine-learned audio classification model to obtain training data. The training data might include the audio recording and one or more phrases from the plurality of phrases closest to the audio recording within an audio-text embedding space of the machine-learned audio classification model.

[0072] In some implementations, before processing the audio recording with a machine-learned audio classification model, the computing system may obtain a plurality of audio samples and an associated corpus of descriptive textual data from an audiovisual data hosting entity. The corpus of descriptive textual data may include, for each of the plurality of audio samples, one or more portions of textual content provided by users of the audiovisual data hosting entity to describe the audio recording. The computing system may, for each of the plurality of audio samples, process the audio sample with an audio embedding portion of the machine-learned audio classification model to obtain an audio embedding. The computing system may then process the one or more portions of textual content that describe the audio recording with a portion of Petition 870260065899, dated 03 / 07 / 2026, page 39 / 77 21 / 22 text embedding of the machine-learned audio classification model to obtain a text embedding and train the machine-learned audio classification model using a contrastive loss function that evaluates a difference between the audio embedding and the text embedding.

[0073] In 506, the computing system can process one or more sentences with a machine-learned generation model to obtain an intermediate representation of one or more sentences.

[0074] In 508, the computing system can process the intermediate representation with a machine-learned cascade diffusion model to obtain audio data. In some implementations, processing the intermediate representation with the machine-learned cascade diffusion model may include applying a Gaussian diffusion process to the audio recording and processing the audio recording and a conditioning signal with the machine-learned cascade diffusion model to obtain the audio data.

[0075] In some implementations, the intermediate representation of the textual content may be a spectrogram, and the computing system may process the spectrogram and the machine-learned diffusion model to obtain the audio data. Alternatively, in some implementations, the intermediate representation of the textual content comprises a low-fidelity audio signal, and the computing system may process the low-fidelity audio signal and the textual content with the machine-learned diffusion model to obtain the audio data.

[0076] In 510, the computing system can train the machine-learned cascade diffusion model based on a difference between the audio data and the audio recording. In some implementations, the computing system can train the machine-learned generation model and the machine-learned cascade diffusion model based on the difference between the audio data and the audio recording.

[0077] In some implementations, the computing system can still obtain, from the textual content that describes a request. The request may indicate a desired type of audio content. The computing system can process the textual content with the machine-learned generator model to generate a Petition 870260065899, dated 03 / 07 / 2026, page 40 / 77 22 / 22 intermediate representation of the textual content and process the intermediate representation with the machine-learned diffusion model to obtain audio data. The audio data may include audio of the desired type of audio content. Additional Disclosure

[0078] The technology discussed in this document refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionalities among and within components. For example, the processes discussed in this document can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0079] Although the present matter has been described in detail with respect to several specific embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon an understanding of what has been set forth, can easily produce alterations, variations and equivalents of such embodiments. Consequently, the disclosure in question does not preclude the inclusion of such modifications, variations and / or additions to the present matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment may be used with another embodiment to produce yet another embodiment. Thus, it is intended that the present disclosure encompass such alterations, variations and equivalents. Petition 870260065899, dated 03 / 07 / 2026, p. 41 / 77

Claims

1 / 6 CLAIMS 1. A computing system, characterized in that it comprises: one or more processors; a machine-learned generative model trained to generate intermediate representations of textual content; a machine-learned broadcast model trained to generate audio data from intermediate representations of textual content, wherein the audio data responds to a request described by the textual content; and one or more non-transient computer-readable media that store instructions which, when executed by one or more processors, cause the participating computing device to perform operations, the operations comprising: processing textual content with the machine-learned generative model to generate an intermediate representation of the textual content, wherein the textual content is descriptive of a request indicating a desired type of audio content;and processing of the intermediate representation with the machine-learned diffusion model to obtain audio data, where the audio data comprises audio of the desired type of audio content.

2. A computing system according to claim 1, characterized in that the machine-learned diffusion model comprises a machine-learned cascade diffusion model comprising one or more attention mechanisms.

3. A computing system according to claim 1, characterized in that the intermediate representation of the textual content comprises a low-fidelity audio signal; and in which the processing of the intermediate representation with the machine-learned diffusion model comprises processing the low-fidelity audio signal and the textual content with the machine-learned diffusion model to obtain the audio data.

4. A computing system, according to claim 1, characterized in that the intermediate representation of the textual content comprises a spectrogram; and Petition 870260065899, dated 03 / 07 / 2026, page 42 / 77 2 / 6, in which the processing of the intermediate representation with the machine-learned diffusion model comprises the processing of the spectrogram and with the machine-learned diffusion model to obtain the audio data.

5. A computing system according to claim 1, characterized in that processing textual content with the machine-learned generator model further comprises: applying a Gaussian diffusion process to the intermediate representation.

6. A computing system according to claim 1, characterized in that, before processing the textual content with the machine-learned generator model, the system comprises: generating a corpus of textual data with a machine-learned text generation model, wherein the corpus of textual data comprises a plurality of sentences and wherein each sentence is descriptive of a type of audio; for each of a plurality of audio recordings: processing the audio recording with a machine-learned audio classification model to obtain training data comprising the audio recording and one or more of the plurality of sentences closest to the audio recording within an audio-text embedding space together with the machine-learned audio classification model; and training at least one of the machine-learned generator model or the machine-learned diffusion model with the training data.

7. A computer system according to claim 1, characterized in that the request described by the textual content comprises one or more musical characteristics; and in that the audio data comprises music with at least one of the one or more characteristics.

8. A computer-implemented method, characterized in that it comprises: generating, by a computing system comprising one or more computing devices, a corpus of textual data with a machine-learned text generation model, wherein the corpus of textual data comprises a plurality of sentences and wherein each sentence is descriptive of a type of audio; for each of a plurality of audio recordings: Petition 870260065899, dated 03 / 07 / 2026, p.43 / 77 3 / 6 process, by the computing system, the audio recording with a machine-learned audio classification model to obtain training data comprising the audio recording and one or more phrases from the plurality of phrases closest to the audio recording within an audio-text embedding space together with the machine-learned audio classification model; process, by the computing system, the one or more phrases with a machine-learned generation model to obtain an intermediate representation of one or more phrases; process, by the computing system, the intermediate representation with a machine-learned cascade diffusion model to obtain audio data; and train, by the computing system, the machine-learned cascade diffusion model based on the difference between the audio data and the audio recording.

9. A computer-implemented method according to claim 8, characterized in that the training of the machine-learned cascade diffusion model comprises the training, by the computing system, of the machine-learned generation model and the machine-learned cascade diffusion model based on the difference between the audio data and the audio recording.

10. A computer-implemented method according to claim 8, characterized in that the processing of the intermediate representation with the machine-learned cascade diffusion model comprises: applying, by the computer system, a Gaussian diffusion process to the audio recording; and processing, by the computer system, the audio recording and a conditioning signal with the machine-learned cascade diffusion model to obtain the audio data.

11. A computer-implemented method according to claim 8, characterized in that, before processing the audio recording with a machine-learned audio classification model, the method comprises: obtaining, by the computing system, a plurality of audio samples and an associated corpus of descriptive textual data from an audiovisual data hosting entity, wherein the corpus of descriptive textual data comprises, for each of the plurality of audio samples, one or more portions of textual content provided by users of the audiovisual data hosting entity to describe the audio recording; for each of the plurality of audio samples: processing, by the computing system, the audio sample with an audio embedding portion of the machine-learned audio classification model to obtain an audio embedding;The computer system processes one or more portions of textual content that describe the audio recording with a text embedding portion of the machine-learned audio classification model to obtain a text embedding; and the computer system trains the machine-learned audio classification model using a contrastive loss function that evaluates the difference between the audio embedding and the text embedding.

12. A computer-implemented method according to claim 8, characterized in that the method further comprises: obtaining, by the computer system, textual content describing a request, wherein the request indicates a desired type of audio content; processing, by the computer system, the textual content with the machine-learned generator model to generate an intermediate representation of the textual content; and processing, by the computer system, the intermediate representation with the machine-learned diffusion model to obtain audio data, wherein the audio data comprises audio of the desired type of audio content.

13. A computer-implemented method according to claim 12, characterized in that the intermediate representation of the textual content comprises a spectrogram; and in that the processing of the intermediate representation with the machine-learned diffusion model comprises the processing, by the computer system, of the spectrogram and with the machine-learned diffusion model to obtain the audio data. Petition 870260065899, dated 03 / 07 / 2026, p. 45 / 77 5 / 6 14. A computer-implemented method according to claim 12, characterized in that the intermediate representation of the textual content comprises a low-fidelity audio signal; and in which the processing of the intermediate representation with the machine-learned diffusion model comprises the processing, by the computer system, of the low-fidelity audio signal and the textual content with the machine-learned diffusion model to obtain the audio data.

15. One or more non-transient computer-readable media, characterized in that they store instructions which, when executed by one or more processors, cause the participating computing device to perform operations, the operations comprising: processing textual content with a machine-learned generator model to generate an intermediate representation of the textual content, wherein the textual content is descriptive of a request indicating a desired type of audio content and wherein the machine-learned generator model is trained to generate intermediate representations of textual content; and processing the intermediate representation with a machine-learned broadcast model to obtain audio data, wherein the audio data comprises audio of the desired type of audio content and wherein the machine-learned broadcast model is trained to generate audio data from intermediate representations of textual content.

16. One or more non-transient computer-readable means according to claim 15, characterized in that the machine-learned diffusion model comprises a machine-learned cascade diffusion model comprising one or more attention mechanisms.

17. One or more non-transient computer-readable media, according to claim 15, characterized in that the intermediate representation of the textual content comprises a low-fidelity audio signal; and wherein the processing of the intermediate representation with the machine-learned diffusion model comprises the processing, by the computing system, of the low-fidelity audio signal and the textual content with the machine-learned diffusion model to obtain the audio data.

18. One or more non-transient computer-readable media, according to Petition 870260065899, dated 03 / 07 / 2026, page 46 / 77 6 / 6, claim 15, characterized in that the intermediate representation of the textual content comprises a spectrogram; and in that the processing of the intermediate representation with the machine-learned diffusion model comprises the processing of the spectrogram and with the machine-learned diffusion model to obtain the audio data.

19. One or more non-transient computer-readable media, according to claim 15, characterized in that processing the textual content with the machine-learned generator model further comprises: applying a Gaussian diffusion process to the intermediate representation. 20.One or more non-transient computer-readable means, according to claim 15, characterized in that, before processing the textual content with the machine-learned generator model, the method comprises: generating a corpus of textual data with a machine-learned text generation model, wherein the corpus of textual data comprises a plurality of sentences and wherein each sentence is descriptive of a type of audio; for each of a plurality of audio recordings: processing the audio recording with a machine-learned audio classification model to obtain training data comprising the audio recording and one or more of the plurality of sentences closest to the audio recording within an audio-text embedding space together with the machine-learned audio classification model; and training at least one of the machine-learned generator model or the machine-learned diffusion model with the training data.Petition 870260065899, dated 03 / 07 / 2026, p. 47 / 77.

Images