Generative music creation platform

Abstract

Described are music creation platforms, pipelines, methods, systems, media, and applications enabling users to generate personalized music which reflects the style of a chosen artist based on user audio input.

Description

WSGR Docket No. 68499-701.601GENERATIVE MUSIC CREATION PLATFORMCROSS-REFERENCE

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 813,357, filed May 28, 2025 and U.S. Provisional Patent Application No. 63 / 727,089, filed December 2, 2024, which are incorporated herein by this reference in its entirety.BACKGROUND

[0002] Generative audio refers to the creation of audio files from databases of audio clips. In some cases, generative audio works by using neural networks to learn the statistical properties of an audio source, then reproduces those properties.SUMMARY

[0003] What is needed are Al agents for artists, scaling their creativity and allowing superfans to co-create with them by providing only fun and simple fan inputs. A basic process may include: 1) the fan picks an artist to make music with, 2) the fan sings or hums a melody and adds a few lyrics, and 3) the platform applies a custom Al model built to add the artist’s sound.

[0004] Accordingly, in one aspect, disclosed herein are computer-implemented methods of generative music creation comprising: maintaining a library of music packs, each music pack comprising elements based on the musical style of an artist; receiving a selection of an artist from a user; receiving an audio input comprising a melody from the user; extracting Musical Instrument Digital Interface (MIDI) data from the audio input; generating lead music from the MIDI data; extracting features from the MIDI data; applying a search algorithm using the extracted features to match the audio input to mostly closely related pre-generated lead music from a database of pre-generated lead music; and creating music in the artist's style based at least in part on the generated lead music and the matched pre-generated lead music. In some embodiments, the method further comprises receiving a selection of a beat from a user. In some embodiments, the music in the artist's style is further based on the selected beat. In some embodiments, one or more of the music packs are generated by a machine learning model from a music collection for an artist. In some embodiments, the audio input comprises humming. In some embodiments, the audio input comprises speaking, singing, or rapping. In some embodiments, the method further comprises receiving an additional audio input comprising lyrics from the user. In some embodiments, the additional audio input comprises speaking, singing, or rapping. In further embodiments, the method further comprises applying a model to extract MIDI data from the additional audio input. In further embodiments, the music in the artist's style is further based on the MIDI data extracted from the additional audio input. In someWSGR Attorney Docket No. 68499-701.601 embodiments, the method further comprises continuously training a foundational model to generate lead music based on the voice audio input. In further embodiments, the foundational model is trained using the music packs to inform one or more audio matching processes. In some embodiments, the selected artist's style is implemented via a control net for a diffusion model.

[0005] In another aspect, disclosed herein are computer-implemented systems comprising a computing device comprising at least one processor and instructions executable by the at least one processor to provide a generative music creation application comprising: a library of music packs, each music pack comprising elements based on the musical style of an artist; a user interface receiving a selection of an artist from a user and an audio input comprising a melody from the user; a software module extracting Musical Instrument Digital Interface (MIDI) data from the audio input; a software module generating lead music from the MIDI data; a software module extracting features from the MIDI data; a software module applying a search algorithm using the extracted features to match the audio input to mostly closely related pre-generated lead music from a database of pre-generated lead music; and a software module creating music in the artist's style based at least in part on the generated lead music and the matched pre-generated lead music.

[0006] In yet another aspect, disclosed herein are one or more non-transitory computer-readable storage media encoded with instructions executable to cause the one or more processors to perform generative music creation operations comprising: maintaining a library of music packs, each music pack comprising elements based on the musical style of an artist; receiving a selection of an artist from a user; receiving an audio input comprising a melody from the user; extracting Musical Instrument Digital Interface (MIDI) data from the audio input; generating lead music from the MIDI data; extracting features from the MIDI data; applying a search algorithm using the extracted features to match the audio input to mostly closely related pregenerated lead music from a database of pre-generated lead music; and creating music in the artist's style based at least in part on the generated lead music and the matched pre-generated lead music.

[0007] Accordingly, in one aspect, disclosed herein are computer-implemented methods of generative music creation comprising: maintaining a library of music packs, each music pack comprising elements based on the musical style of an artist; receiving a selection of an artist from a user; receiving a first voice audio input comprising a melody from the user; extracting Musical Instrument Digital Interface (MIDI) data from the first voice audio input; generating lead music from the MIDI data; extracting features from the MIDI data; applying a searchWSGR Attorney Docket No. 68499-701.601 algorithm using the extracted features to match the first voice audio input to mostly closely related pre-generated lead music from a database of pre-generated lead music; and creating music in the artist's style based at least in part on the generated lead music and the matched pregenerated lead music. In some embodiments, the method further comprises receiving a selection of a beat from a user. In some embodiments, the music in the artist's style is further based on the selected beat. In some embodiments, one or more of the music packs are generated by a machine learning model from a music collection for an artist. In some embodiments, the first voice audio input comprises humming. In some embodiments, the first voice audio input comprises speaking, singing, or rapping. In some embodiments, the method further comprises receiving a second voice audio input comprising lyrics from the user. In some embodiments, the second voice audio input comprises speaking, singing, or rapping. In further embodiments, the method further comprises applying a model to extract MIDI data from the second voice audio input. In further embodiments, the music in the artist's style is further based on the MIDI data extracted from the second voice audio input. In some embodiments, the method further comprises continuously training a foundational model to generate lead music based on the voice audio input. In further embodiments, the foundational model is trained using the music packs to inform one or more audio matching processes. In some embodiments, the selected artist's style is implemented via a control net for a diffusion model.

[0008] In another aspect, disclosed herein are computer-implemented systems comprising a computing device comprising at least one processor and instructions executable by the at least one processor to provide a generative music creation application comprising: a library of music packs, each music pack comprising elements based on the musical style of an artist; a user interface receiving a selection of an artist from a user and a first voice audio input comprising a melody from the user; a software module extracting Musical Instrument Digital Interface (MIDI) data from the first voice audio input; a software module generating lead music from the MIDI data; a software module extracting features from the MIDI data; a software module applying a search algorithm using the extracted features to match the first voice audio input to mostly closely related pre-generated lead music from a database of pre-generated lead music; and a software module creating music in the artist's style based at least in part on the generated lead music and the matched pre-generated lead music. In some embodiments, the library of music packs are generated by a machine learning model from a music collection for an artist. In some embodiments, the selection of the artist is based on the audio input. In some embodiments, the audio input from the user comprises humming. In some embodiments, the audio input from the user comprises speaking, singing, or rapping. In some embodiments, the system may furtherWSGR Attorney Docket No. 68499-701.601 comprise an additional audio input. In some embodiments, the additional audio input comprises lyrics from the user. In some embodiments, the lyrics are conveyed by speaking, singing, or rapping. In some embodiments, the software module is applied to the additional audio input, extracting MIDI data. In some embodiments, the music created in the artists’ style is altered by the MIDI data of the additional audio input. In some embodiments, the module is continuously training to generate music based on an audio input. In some embodiments, the continuous training are trained by the library of music packs to inform one or more audio matching processes. In some embodiments, the artist’s style is implemented via a control net for a diffusion model.

[0009] In yet another aspect, disclosed herein are one or more non-transitory computer-readable storage media encoded with instructions executable to cause the one or more processors to perform generative music creation operations comprising: maintaining a library of music packs, each music pack comprising elements based on the musical style of an artist; receiving a selection of an artist from a user; receiving a first voice audio input comprising a melody from the user; extracting Musical Instrument Digital Interface (MIDI) data from the first voice audio input; generating lead music from the MIDI data; extracting features from the MIDI data; applying a search algorithm using the extracted features to match the first voice audio input to mostly closely related pre-generated lead music from a database of pre-generated lead music; and creating music in the artist's style based at least in part on the generated lead music and the matched pre-generated lead music.

[0010] In some embodiments, the library of music packs are generated by a machine learning model from a music collection. In some embodiments, the music in the artist’s style is based on the audio input. In some embodiments, the audio input comprises humming. In some embodiments, the audio input comprises speaking, singing, or rapping. In some embodiments, the storage media further comprises an additional audio input. In some embodiments, the additional audio input comprises lyrics provided by the user. In some embodiments, the lyrics are conveyed by speaking, singing, or rapping. In some embodiments, the additional audio input comprises MIDI data, where the MIDI data is extracted. In some embodiments, the MIDI data modulates the music in the artist’s style. In some embodiments, the storage media is continuously training to generate music based on the audio input. In some embodiments, the continuous training for music generation is based on the additional audio input. In some embodiments, the continuous training for music generation is further trained by the library of music packs to inform one or more audio matching processes. In some embodiments, the artist’s style is implemented via a control net for a diffusion model.WSGR Attorney Docket No. 68499-701.601BRIEF DESCRIPTION OF THE DRAWINGS[Oil] A better understanding of the features and advantages of the present subject matter will be obtained by reference to the following detailed description that sets forth illustrative embodiments and the accompanying drawings of which:

[0012] FIG. 1 shows a non-liming example of a music generation framework.

[0013] FIG. 2 shows a non-limiting example overview of a multi-modal feature extraction pipeline.

[0014] FIG. 3 shows a non-limiting example overview of a hybrid encoder and upsampler architecture.

[0015] FIG. 4 illustrates a non-limiting example of an objective of the subject matter described herein;

[0016] FIG. 5 shows a non-limiting example of a platform architecture;

[0017] FIG. 6 shows a non-limiting example of a Neural Synth architecture;

[0018] FIG. 7 shows a non-limiting example of a Neural Synth training pipeline;

[0019] FIG. 8 shows a non-limiting example of an instrumental generation framework without timbre training;

[0020] FIG. 9 shows a non-limiting example of a multi-stream conditioning framework;

[0021] FIGS. 10A-10X show non-limiting examples of graphic user interfaces (GUIs) illustrating a first user experience (UX);

[0022] FIGS. 11A-11S show non-limiting examples of graphic user interfaces (GUIs) illustrating a second user experience (UX);

[0023] FIGS. 12A-12F show non-limiting examples of graphic user interfaces (GUIs) illustrating a third user experience (UX);

[0024] FIGS. 13A-13B show non-limiting examples of graphic user interfaces (GUIs) illustrating a fourth user experience (UX) allowing a user to play games based on generated music;

[0025] FIGS. 14A-14C show non-limiting examples of graphic user interfaces (GUIs) illustrating a user experience allowing for remixing and rearranging sections of a completed mix;

[0026] FIGS. 15A-15B shows a non-limiting example of musical scale distributions of anWSGR Attorney Docket No. 68499-701.601 example dataset;

[0027] FIG. 16 shows a non-limiting example of a melodic pattern visualization;

[0028] FIG. 17 shows example performance metric data;

[0029] FIG. 18 shows a non-limiting example of a computing device; in this case, a device with one or more processors, memory, storage, and a network interface;

[0030] FIG. 19 shows a non-limiting example of a web / mobile application provision system; in this case, a system providing browser-based and / or native mobile user interfaces; and

[0031] FIG. 20 shows a non-limiting example of a cloud-based web / mobile application provision system; in this case, a system comprising an elastically load balanced, auto-scaling web server and application server resources as well synchronously replicated databases.DETAILED DESCRIPTION

[0032] Recent advances in Al-generated content (AIGC) have made strides in various creative domains, with models capable of producing high-quality images, videos, and text that are increasingly indistinguishable from human-created content. Music generation, however, presents unique challenges due to its temporal nature, structural complexity, and the emotional depth it conveys. While previous research has explored symbolic music generation through various prompting mechanisms, and others have focused on humming transcription, fewer works have addressed the specific challenge of transforming humming into instrument-specific melodies with high fidelity and controllability.

[0033] A crucial ability for a creating authentic, expressive piece of generated music is the ability to control the timbral characteristics. Different instruments not only produce different sounds but also follow different idiomatic patterns of playing that affect how melodies are realized. For instance, a melody that sounds natural on a piano might be awkward or technically challenging on a violin due to the different physical constraints and expressive capabilities of these instruments. A system that can adapt hummed melodies to suit specific instrumental characteristics could significantly enhance the utility and quality of Al music generation.

[0034] Al-generated sounds specifically for electronic music — particularly lead instruments — as an alternative to traditional software-based virtual synthesizers may be a place for updates. Running full virtual synthesizers on backend servers is often impractical due to licensing, performance, and scalability constraints. Moreover, each synthesizer has its own sonic "character," which, while distinctive, ultimately limits the variety of sounds available to creators.WSGR Attorney Docket No. 68499-701.601These practical limitations in existing music production workflows highlight the need for more flexible, Al-driven approaches to instrumental sound generation.

[0035] As disclosed herein, a music creation framework designed to bridge this gap by transforming simple MIDI sequences extracted from humming into polished, instrument-specific melodies, would be beneficial. This music creation framework leverages a transformer-based architecture that learns the relationship between hummed melodies and their professional instrumental counterparts, guided by explicit control signals that specify the target instrument’s timbral characteristics. Unlike recent text-based generative methods, which typically lack precise control and can be difficult to integrate seamlessly into compositions, the frameworks, as disclosed herein, may directly render high-fidelity (44.1 kHz) stereo audio from MIDI input without altering the fundamental melodic intent, making it more suitable for practical music production scenarios.

[0036] In order to maintain the core musical intent of the hummed melody, a specialized architecture, as disclosed herein, may allow for the intent to be maintained while adapting it to instrument-specific idioms and expressivity patterns. The specialized architecture may comprise an explicit control mechanism that allows for targeted generation toward specific instrumental timbres by using audio samples to control the desired output characteristics. The specialized architecture may further comprise a lightweight yet effective feature representation derived from discrete audio codec compression, enabling efficient training while preserving essential musical information.

[0037] Additionally, by capturing each synthesizer’s unique timbral qualities, the framework may produce lead sounds akin to those users already have in mind, while offering higher quality and more precise control than typical 16 kHz text-to-audio approaches. This approach provides creators with the familiar workflow of using reference sounds to guide their production, but with the expanded flexibility and quality that neural synthesis enables.

[0038] The framework, disclosed herein, can effectively transform simple humming-derived MIDI sequences into sophisticated instrumental melodies that respect both the original melodic intent and the target instrument’s characteristics. Through distributed training, high computational efficiency while maintaining model quality may be achieved, enabling practical deployment of this technology.

[0039] By democratizing the process of creating instrument-specific melodies from simple humming, the framework disclosed herein represents an important step toward making music creation more accessible while maintaining high standards of musical quality and expressivity.WSGR Attorney Docket No. 68499-701.601The disclosed contributes to the broader goal of developing Al systems that enhance human creativity rather than simply automating creative tasks.

[0040] Neural audio synthesis has emerged as a promising paradigm for music production, yet significant challenges remain in developing systems that can accurately render symbolic music with specific timbral characteristics. The disclosed method further addresses the fundamental problem of timbre-conditioned melody synthesis: generating high-fidelity audio waveforms that follow precise melodic instructions while faithfully reproducing the timbral qualities of a reference audio sample.Certain Definitions

[0041] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present subject matter belongs.

[0042] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Any reference to “or” herein is intended to encompass “and / or” unless otherwise stated.

[0043] Reference throughout this specification to “some embodiments,” “further embodiments,” or “a particular embodiment,” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in some embodiments,” or “in further embodiments,” or “in a particular embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0044] As herein disclosed, the fundamental problem of timbre-controlled melody synthesis is addressed by a mapping function fe that transforms a MIDI melody m and a reference audio snippet c (serving as a timbre control signal) into a synthesized audio waveform w of Equation 1 w = f0(m, c)Equation 1

[0045] To successfully address the fundamental problem as herein described, the generated waveform must simultaneously satisfy melodic accuracy (correctly realizing the notes, timing, and articulation specified in m) and timbral fidelity (exhibiting the sonic characteristics presentWSGR Attorney Docket No. 68499-701.601 in c).

[0046] The primary technical challenge stems from the inherent complexity of disentangling timbre from melodic content in audio signals. This difficulty is compounded when working with realistic control signals that may be brief, incomplete, or contain different melodic content than the target. The method disclosed herein bridges this methodological gap through a architecture designed specifically for timbre-conditioned neural audio rendering.

[0047] Referring to FIG. 1, an example music generation framework 100 is provided. Here, an individual user provides a musical hum 110. The musical hum is converted into a MIDI representation 120, which is input into the Descript Audio Codec (DAC encoder) 130. The DAC encoder produces a series of features, or codebooks 140, based on the input. Next, the AR transformer 150 utilizes the series of features to transform those features into a predicted melody from an initial codebook of a series of codebooks. The Hybrid Music Encoder and Upsampler 160 processes the predicted melody and to predict outcomes from additional codebooks. The DAC decoder 170 then processes the series of codebook outputs, producing a final musical output 180.

[0048] In some cases, the music generation framework may be intended for personal use. In some cases, the music generation framework may be intended for professional use. In some cases, the music generation framework may be intended for creative use. In some cases, the music generation framework may be intended for educational use. In some cases, the music generation framework may be intended for use without sharing. In some cases, the music generation framework may be intended for sharing with others. In some cases, the music generation framework may be intended for use by one individual without collaboration. In some cases, the music generation framework may be intended for collaborative use.

[0049] In some embodiments, an auditory input 110 is provided. In some cases, the auditory input may be created by a user. In some cases, the auditory input may be created by a single user. In some cases, the auditory input may be created by a plurality of users. In some cases, the auditory input may be created by a living user. In some cases, the auditory input may be created by an artificial user. In some cases, the auditory input may be vocals.

[0050] In some cases, the auditory input 110 may be melodic sounds. In some cases, the auditory input may be percussive sounds. In some cases, the auditory input may be harmonic sounds. In some cases, the auditory input may be rhythmic sounds. In some cases, the auditory input may be made by a user. In some cases, the auditory input may be singing. In some cases, the auditory input may be whistling. In some cases, the auditory input may be blowing. In someWSGR Attorney Docket No. 68499-701.601 cases, the auditory input may be humming. In some cases, the auditory input may be made by an electronic source, such as a computer or automated device. In some cases, the auditory input may be made by an instrumental source.

[0051] In some cases, the auditory input 110 may comprise music features. In some cases, the auditory input may comprise traditional music elements. In some cases, the auditory input may comprise a rhythmic pattern. In some cases, the auditory input may comprise dynamics. In some cases, the auditory input may comprise harmony. In some cases, the auditory input may comprise melody. In some cases, the auditory input may comprise pitch. In some cases, the auditory input may comprise tempo. In some cases, the auditory input may comprise musical texture. In some cases, the auditory input may comprise timbre. In some cases, the auditory input may comprise emotional intent.

[0052] In some cases, a data preprocessing may be implemented. The dataset consists of a number of melodies (mi, m2, ... , mp, where / is the total number of melodies present in the dataset, and synthesizer presets (pi, p2, ... ,p , where C'is the total number of melodies present in the dataset. In some cases, the melodies may be MIDI files. For each melody m. and preset p a rendered waveform is generated Wid, representing the full-length audio rendering of the melody using the given preset. This creates a complete set of rendered waveforms W containing unique audio samples.

[0053] In some cases, the dataset may comprise a number of melodies. In some cases, the dataset may comprise from about 5 melodies to about 5,000 melodies. In some cases, the dataset may comprise from about 5 melodies to about 100 melodies. In some cases, the dataset may comprise from about 100 melodies to about 500 melodies. In some cases, the dataset may comprise from about 500 melodies to about 1000 melodies. In some cases, the dataset may comprise from about 1000 melodies to about 2500 melodies. In some cases, the dataset may comprise from about 2500 melodies to about 5000 melodies. In some cases, the dataset may comprise 1,250 melodies.

[0054] In some cases, the melodies may have a maximum duration. In some cases, the maximum duration may be from about 1 second to about 60 seconds. In some cases, the maximum duration may be from about 1 second to about 10 seconds. In some cases, the maximum duration may be from about 10 seconds to about 20 seconds. In some cases, the maximum duration may be from about 20 seconds to about 30 seconds. In some cases, the maximum duration may be from about 30 seconds to about 40 seconds. In some cases, the maximum duration may be from about 40 seconds to about 50 seconds. In some cases, theWSGR Attorney Docket No. 68499-701.601 maximum duration may be from about 50 seconds to about 60 seconds. In some cases, the maximum duration may be 5 seconds.

[0055] In some cases, the melodies may adhere to a pitch range. In some cases, the pitch range may be from Cl to C8, where C4 is middle C. In some cases, the pitch range may be from Cl to C4. In some cases, the pitch range may be from C2 to C5. In some cases, the pitch range may be from C3 to C6. In some cases, the pitch range may be from C4 to C7. In some cases, the pitch range may be from C5 to C8. In some cases, the pitch range may be from C3 to G5.

[0056] In some cases, the pitch range may cover one or more octaves. In some cases, the pitch range may be a vocal range. In some cases, the pitch range may be a soprano voice range, from about C4 to C6. In some cases, the pitch range may be a mezzo-soprano voice range, from about A3 to A5. In some cases, the pitch range may be an alto voice range, from about F3 to E5. In some cases, the pitch range may be a tenor voice range, from about C3 to C5. In some cases, the pitch range may be a baritone voice range, from about A2 to A4. In some cases, the pitch range may be a bass voice range, from about E2 to E4. In some cases, the pitch range may be an instrumental range. In some cases, the pitch range may be a piano range, wherein the range is about 7 octaves. In some cases, the pitch range may be a guitar range, wherein the range is usually 3.5 octaves. In some cases, the pitch range may be a flute range, wherein the range is about 3 octaves. In some cases, the pitch range may be a violin range, wherein the range is about 4 octaves. In some cases, the pitch range may be a trumpet range, wherein the range is about 2.5 octaves.

[0057] In some cases, the melodies may adhere to a musical mode. In some cases, the melodies may adhere to a Mixolydian mode. In some cases, the melodies may adhere to a Lydian mode. In some cases, the melodies may adhere to a Phrygian mode. In some cases, the melodies may adhere to a Dorian mode. In some cases, the melodies may adhere to a Hypolydian mode. In some cases, the melodies may adhere to a Hypophrygian mode. In some cases, the melodies may adhere to a Common mode. In some cases, the melodies may adhere to a Locrian mode. In some cases, the melodies may adhere to a Hypodorian mode. In some cases, the melodies may adhere to a Hypomixolydian mode. In some cases, the melodies may adhere to an Ionian mode. In some cases, the melodies may adhere to an Aeolian mode. In some cases, the melodies may adhere to a Greek mode. In some cases, the melodies may adhere to a Western Church mode. In some cases, the melodies may adhere to a Modem Mode.

[0058] In some cases, the melodies may adhere to a musical scale. In some cases, the melodies may adhere to a major scale. In some cases, the melodies may adhere to a minor scale. In someWSGR Attorney Docket No. 68499-701.601 cases, the melodies may adhere to a chromatic scale. In some cases, the melodies may adhere to a diatonic scale. In some cases, the melodies may adhere to a harmonic scale. In some cases, the melodies may adhere to a melodic scale. In some cases, the melodies may adhere to a harmonic minor scale. In some cases, the melodies may adhere to a double harmonic scale. In some cases, the melodies may adhere to a Phrygian dominant scale. In some cases, the melodies may adhere to a minor blues scale.

[0059] In some cases, the melodies may adhere to a tempo. In some cases, the tempo may be from about 40 beat per minute (bpm) to about 220 bpm. In some cases, the tempo may be from 40 bpm to about 80 bpm. In some cases, the tempo may be from 80 bpm to about 120 bpm. In some cases, the tempo may be from 120 bpm to about 160 bpm. In some cases, the tempo may be from 160 bpm to about 200 bpm. In some cases, the tempo may be from 200 bpm to about 220 bpm. In some cases, the tempo may be from 60 bpm to about 200 bpm. In some cases, the melodies may adhere to an adagio tempo. In some cases, the melodies may adhere to a largo tempo. In some cases, the melodies may adhere to an andante tempo. In some cases, the melodies may adhere to a moderato tempo. In some cases, the melodies may adhere to an allegretto tempo. In some cases, the melodies may adhere to an allegro tempo. In some cases, the melodies may adhere to a vivace tempo. In some cases, the melodies may adhere to a presto tempo. In some cases, the melodies may adhere to a prestissimo tempo.

[0060] In some cases, the dataset may comprise a number of synthesizer presets. In some cases, the dataset may comprise from about 5 synthesizer presets to about 500 synthesizer presets. In some cases, the dataset may comprise from about 5 synthesizer presets to about 100 synthesizer presets. In some cases, the dataset may comprise from about 100 synthesizer presets to about 200 synthesizer presets. In some cases, the dataset may comprise from about 200 synthesizer presets to about 300 synthesizer presets. In some cases, the dataset may comprise from about 300 synthesizer presets to about 400 synthesizer presets. In some cases, the dataset may comprise from about 400 synthesizer presets to about 500 synthesizer presets. In some cases, the dataset may comprise 50 synthesizer presets.

[0061] In some cases, the audio rendering may have a full-length. In some cases, the full-length may be from about 1 second to about 60 seconds. In some cases, the full-length may be from about 1 second to about 10 seconds. In some cases, the full-length may be from about 10 seconds to about 20 seconds. In some cases, the full-length may be from about 20 seconds to about 30 seconds. In some cases, the full-length may be from about 30 seconds to about 40 seconds. In some cases, the full-length may be from about 40 seconds to about 50 seconds. InWSGR Attorney Docket No. 68499-701.601 some cases, the full-length may be from about 50 seconds to about 60 seconds. In some cases, the full-length may be 5 seconds.

[0062] In some cases, the waveform may comprise a number of unique audio samples. In some cases, the waveform may comprise from about 100 unique audio samples to about 1,000,000 unique audio samples. In some cases, the waveform may comprise from about 100 unique audio samples to about 1,000 unique audio samples. In some cases, the waveform may comprise from about 1,000 unique audio samples to about 10,000 unique audio samples. In some cases, the waveform may comprise from about 10,000 unique audio samples to about 100,000 unique audio samples. In some cases, the waveform may comprise from about 100,000 unique audio samples to about 500,000 unique audio samples. In some cases, the waveform may comprise from about 500,000 unique audio samples to about 1,000,000 unique audio samples. In some cases, the waveform may comprise 62,500 unique audio samples.

[0063] In some cases, the model may be trained. In some cases, the model may be trained by control signals. In some cases, the model may be trained by generated control signals. In some cases, the generated control signals may be variations of each waveform W. In some cases, the waveform variations may comprise multiple crops of varying durations. A crop d,j, << / represents the A111crop of seconds of duration d (where d G 1, 2, 3, 4) from waveform WirJ. Each crop begins at a random start position Sij,k, wherein the selected start time ensures that the entire crop fits within the original full-length of the waveform. For example, if the full length is 5 seconds, the start time is selected to ensure that the time between 5 and d is at least 5 seconds. Since these crops have different lengths, a tiling function r is applied to normalize their duration to match the target waveform length. The resulting function T(cy,*,rf) concatenates the crop with itself repeatedly until reaching the desired length.

[0064] In some cases, the model may be trained by a triplet. In some cases, the model may be trained by a triplet comprising an input melody (m;), a tiled control signal (T), and a target waveform (w). In some cases, the input melody is a MIDI melody. In some cases, the tiled control signal may potentially be from any melody (ma). In some cases, the tiled control signal may potentially be from any melody (ni) using preset ( / y>), resulting in a tiled control signal T(ca,b,k,d). In some cases, the target waveform may represent a melody (m;). In some cases, the target waveform may represent a melody (m;) rendered with a preset (pb), resulting in a target waveform (wi,b). In some cases, the triplet may be represented by (mt, T. ca,b,k,d), Wi,b). In some cases, a critical property is maintained in this triplet. In some cases, the maintained critical property is that the preset indices match between control and target, ensuring consistent timbreWSGR Attorney Docket No. 68499-701.601 transfer while allowing cross-melody augmentation.

[0065] In some cases, the implemented data preprocessing creates a rich training environment that enables the model to robustly learn timbre characteristics from arbitrary control signals and appropriately apply them to diverse melodic inputs.

[0066] In some cases, the crops may be relative to the full waveform. In some cases, the crops may be distinct from the full waveform. In some cases, the crops may be identical to the full waveform. In some cases, when the crops are identical to the full waveform, they would allow the model to simply copy the control signal rather than learning the timbre transformation. Therefore, using a 5-second full length as an examples, for crops where d= 5, ca,b,k,5 may only be utilized when a j. This utilization strategy ensures that the control signal comes from a different melody than the target. This forces the model to learn proper timbre extraction and application across varied melodic content rather than memorizing specific wave patterns.Input Conversion

[0067] After receiving an initial auditory input from a user, such as humming, the input is converted to an electronic representation 120. The input conversion, completed by a neural model, employs an instrument-agnostic approach, wherein the approach generalizes the input across diverse sound sources while maintaining efficiency.

[0068] The input conversion is completed through a multiple step framework. First, Constant-Q Transform (CQT) is applied to the input audio, wherein the transform applies 3 bins per semitone and an 11 ms hop size to the input audio. This is calculated as below in Equation 2XCQT= CQT(xEquation 2

[0069] Second, a Harmonic Stacking Layer is applied to the CQT, which aligns harmonically- related frequencies along a third dimension. This calculation is provided in Equation 3, where Shis a shift operation that vertically shifts the CQT by the number of frequency bins corresponding to harmonic A, and H is the set of harmonics. In some embodiments, the set comprises 7 harmonics and 1 sub-harmonic.XH=S XCQTEHEquation 3

[0070] The stacked representation XH) is then processed by Convolutional Layers to simultaneously predict at least one posteriorgram, or a categorical distribution. In some cases,WSGR Attorney Docket No. 68499-701.601 three posteriorgrams are predicted, as shown in Equations 4-6. In the below, where Ypis the multipitch posteriorgram (Equation 4), which is related to the 3 bins per semitone, Ynis the note activation posteriorgram (Equation 5), which is based off of 1 bin per semitone, Yois the onset detection posteriorgram (Equation 6), based off of 1 bin per semitone, fp / fn / fo / fc are convolutional mappings, and o is the sigmoid activation function.

[0071] During training of the neural model through the above, a combined loss function is also employed, where Lpand Lnare binary cross-entropy losses for multipitch and note activation targets respectively, and Lo+ and Lo- are class-balanced binary cross-entropy losses for onset detection with weights to address class imbalance. In some cases, the weights may be equal. In some cases, the weights may be w+ = 0.95 and w- = 0.05. In some cases, the weights may be from 0 to 1. Combined loss function calculation is shown in Equation 7.L = Lp + Ln+ w+x LQ++ w_ x Lo_Equation 7

[0072] Specific note events are then generated after initial input processing. The further note event generation comprises: 1) peak-picking the onset posteriorgram Yoto identify candidate onset times to, ft),' 2) tracing forward through Ynuntil the likelihood falls below T„ threshold; and 3) creating additional note events from remaining high-likelihood regions in Yn.

[0073] Through this process of input conversion, processing of various instrumental and noninstrumental sounds may be processed with low resource requirements. In one example, a 7 minute recording may be converted in 24 seconds using only 951 MB of memory. The resulting conversion and extraction results in three distinct input modalities: MIDI Files, Complete Audio Recordings, and Cropped Audio Segments.

[0074] Each MIDI representation of the converted audio is then converted into a sine wave representation. The converted sine wave is further padded or cropped to achieve a uniform target length, or size. This resulting normalized sine wave representation preserves the melodic and harmonic content of the original MIDI file in a consistent format. Here, the normalized sine wave representation is characterized as: size = 1 x 9 x T, where T is the time dimension).WSGR Attorney Docket No. 68499-701.601

[0075] In parallel, the input audio files, as .wav files, and cropped audio segments were passed through a similar conversion. Each audio source is first standardized to the target length through padding or cropping operations. The audio is then split into left and right stereo channels. In some embodiments, the complete audio recordings may be Full L and Full R, where L and R represent left and right. In some embodiments, the cropped segments may be Crop L and Crop R, where L and R represent left and right.

[0076] The sine representation, complete audio recordings, and cropped segments are grouped together, or concatenated. The concatenation is configured to form a comprehensive multimodal representation, including aspects from all three input modalities, providing different complimentary representations of the musical content. Each concatenated grouping is processed by the Descript Audio Codec (DAC) Encoder, to extract features and generate a set of audio features.

[0077] In some cases, the electronic representation may be a music information file. In some cases, the electronic representation may be an MP3 file. In some cases, the electronic representation may be a MIDI (Musical Instrument Digital Interface) file. In some cases, the electronic representation may not be an audio file. In some cases, the electronic representation may be selected due to its file size. In some cases, the electronic representation may be selected due to its versatility.

[0078] In some cases, the set of harmonics H may comprise a combination of harmonics and sub-harmonics. In some cases, the set of harmonics may comprise at least one harmonic. In some cases, the set of harmonics may comprise 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10 harmonics. In some cases, the set of harmonics may comprise at least one sub-harmonic. In some cases, the set of harmonics may comprise 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10 sub-harmonics. In some cases, the set of harmonics may comprise a plurality of harmonics. In some cases, the set of harmonics may comprise a plurality of sub-harmonics. In some cases, the set of harmonics may comprise at least one harmonic and at least one sub-harmonic. In some cases, the set of harmonics may comprise a plurality of harmonics and at least one sub harmonic.

[0079] In some cases, the input audio may be non-instrum ental. In some cases, the input audio may be created by a user. In some cases, the input audio may be percussive sounds. In some cases, the input audio may be harmonic sounds. In some cases, the input audio may be rhythmic sounds. In some cases, the input audio may be instrumental. In some cases, the input audio may comprise a percussion instrument. In some cases, the input audio may comprise a woodwind instrument. In some cases, the input audio may comprise a string instrument. In some cases, theWSGR Attorney Docket No. 68499-701.601 input audio may comprise a brass instrument. In some cases, the input audio may comprise an electronic instrument.

[0080] In some cases, the input audio may have a recording length. In some cases, the audio input may be from about 5 seconds to about 60 minutes. In some cases, the audio input may be from about 5 seconds to about 60 seconds, or 1 minute. In some cases, the audio input may be from about 1 minute to about 10 minutes. In some cases, the audio input may be from about 10 minute to about 20 minutes. In some cases, the audio input may be from about 20 minutes to about 30 minutes. In some cases, the audio input may be from about 30 minutes to about 40 minutes. In some cases, the audio input may be from about 40 minutes to about 50 minutes. In some cases, the audio input may be from about 50 minutes to about 60 minutes. In some cases, the audio input may be from about 1 minute to 5 minutes. In some cases, the audio input may be from about 5 minutes to 10 minutes. In some cases, the audio input is about 7 minutes.

[0081] In some cases, the audio conversion may have a conversion time. In some cases, the conversion time may be from about 1 second to about 240 seconds. In some cases, the conversion time may be from about 1 second to about 30 seconds. In some cases, the conversion time may be from about 30 seconds to about 60 seconds. In some cases, the conversion time may be from about 60 seconds to about 120 seconds. In some cases, the conversion time may be from about 120 seconds to about 180 seconds. In some cases, the conversion time may be from about 180 seconds to about 240 seconds. In some cases, the conversion time may be about 24 seconds.

[0082] In some cases, the audio conversion time may be a percentage of the audio input length. In some cases, the audio conversion time may be from about 1 % to about 25 % of the audio input length. In some cases, the audio conversion time may be from about 1 % to about 5 % of the audio input length. In some cases, the audio conversion time may be from about 5 % to about 10 % of the audio input length. In some cases, the audio conversion time may be from about 10 % to about 15 % of the audio input length. In some cases, the audio conversion time may be from about 15 % to about 20 % of the audio input length. In some cases, the audio conversion time may be from about 20 % to about 25 % of the audio input length. In some cases, the audio conversion time may be from about 3 % to about 8% of the audio input length. In some cases, the audio conversion time may be about 6 % of the audio input length.

[0083] In some cases, the converted audio may be stored on memory. In some cases, the converted audio size may be from about 100 megabytes (MB) to about 100 gigabytes (GB). In some cases, the audio size may be from about 100 MB to about 500 MB. In some cases, theWSGR Attorney Docket No. 68499-701.601 audio size may be from about 500 MB to about 1000 MB, or 1 GB. In some cases, the audio size may be from about 1 GB to about 50 GB. In some cases, the audio size may be from about 50 GB to about 100 GB. In some cases, the converted audio may be stored on about 900 MB of storage. In some cases, the converted audio may be stored on about 951 MB of storage.

[0084] In some cases, the audio file may be split. In some cases, the audio file may be mono. In some cases, the audio file may be split along stereo channels. In some cases, the audio file may be split into a left channel and a right channel. In some cases, the audio file may be a quadraphonic sound. In some cases, the audio file may be split into four channels.

[0085] In some cases, at least one input modality may be utilized. In some cases, at least two input modalities may be utilized. In some cases, at least three input modalities may be utilized. In some cases, at least four input modalities may be utilized. In some cases, at least five input modalities may be utilized. In some cases, at least ten input modalities may be utilized. In some cases, one, two, three, four, or five input modalities may be utilized. In some cases, each of the input modalities are split. In some cases, at least one of the input modalities are split. In some cases, the input modalities are split into two parts. In some cases, the input modalities are split into three parts. In some cases, the input modalities are split into four parts. In some cases, the input modalities are split into more than four parts.

[0086] In some cases, the concatenated group may comprise a combination of input modalities. In some cases, the concatenated group may comprise a sine wave representation. In some cases, the concatenated group may comprise a full audio channel. In some cases, the concatenated group may comprise more than one full audio channel. In some cases, the concatenated group may comprise a cropped audio channel. In some cases, the concatenated group may comprise more than one cropped audio channel. In some cases, the concatenated group may comprise a sine wave representation and a full audio channel. In some cases, the concatenated group may comprise a sine wave representation and more than one full audio channel. In some cases, the concatenated group may comprise a sine wave representation and a cropped audio channel. In some cases, the concatenated group may comprise a sine wave representation and more than one cropped audio channel. In some cases, the concatenated group may comprise a full audio channel and a cropped audio channel. In some cases, the concatenated group may comprise more than one full audio channel and a cropped audio channel. In some cases, the concatenated group may comprise a full audio channel and more than one cropped audio channel. In some cases, the concatenated group may comprise more than one full audio channel and more than one cropped audio channel. In some cases, the concatenated group may comprise a sine waveWSGR Attorney Docket No. 68499-701.601 representation, at least one full audio channel, and at least one cropped audio channel. In some cases, the concatenated group may comprise a sine wave representation, two full audio channels, and two cropped audio channels.Descript Audio Codec (DAC) Encoder

[0087] The converted audio may then be encoded by an audio codec method utilizing a neural audio codec approach. Early AE-based codecs trained models to reconstruct handcrafted acoustic features using independent quantization modules that weren’t globally optimized, requiring extensive signal-specific assumptions and separate synthesizers. Later waveformdomain AE speech codec lacked joint quantizer training. Subsequent developments introduced end-to-end training with straight-through gradient estimation and softmax quantization, though these systems faced challenges with either slow autoregressive inference or quality limitations in non-autoregressive (NAR) approaches.

[0088] Recent neural audio codecs have been driven by Generative Adversarial Network (GAN)-based NAR architectures that achieve fast encoding / decoding with superior quality and efficient bitrates. Parallel innovations include neural vocoder approaches that reconstruct audio waveforms from quantized handcrafted features, conventional codec codes, or neural AE embeddings. Additionally, postfiltering techniques have simplified code-to-waveform mapping by enhancing outputs from pre-trained codec decoders. Therefore, a Descript Audio Codec (DAC) encoder may be used, which offers superior performance across diverse audio types. DAC’s architecture integrates a discriminator networks that effectively address phase reconstruction challenges and better preserve harmonic structures.

[0089] The concatenated grouping is processed by the DAC Encoder 130, where the acoustic characteristics across all input modalities. The resulting features are then split based on the input modality, thereby ungrouping the concatenated group, producing separate feature sets. The feature sets comprise: Sine features (1 x 9 x T ), Full_features (2 x 9 xT), representing the stereo channels), and Crop features (2 x 9 x T), where T is the time dimension. FIG. 2 provides an overview of the multi-modal feature extraction pipeline 200. The provided illustrates the input conversion and DAC encoder processes as disclosed herein. First, the input from three sources 210 is received and cropped 220 into audio segments 230. Each input undergoes concatenation 240 and processing by the DAC Encoder 250, which results in the features being split 260.

[0090] The DAC encoder is configured to transform raw audio into a sequence of discrete representations. After receiving an input audio, a high-fidelity neural codec encodes the audio file into a number of tokens per second across parallel quantized streams, or codebooks. ForWSGR Attorney Docket No. 68499-701.601 example, a 44.1 kilohertz (kHz) audio input may be encoded to approximately 86 tokens per second for a 512-sample frame hop across 9 codebooks.

[0091] Each codebook z yields a sequence of tokens z, = [z1,, z2,, ... , zTi] with each token being a 10 - bit index (0 - 1023) into that codebook’s vocabulary. For example, each codebook z e { 1, ... , 9} yields a sequence of tokens z, = [z7 / , z2, . . . , zTi\ with each token being a 10 - bit index (0 - 1023). The first codebook z1captures the most coarse information (broad structure and spectral content), while subsequent codebooks, z2. . . zN, representing residual finer details that refine the audio fidelity. In a case, where N = 9, subsequent codebooks may be represented from Z2.. . Z9. Formally, the DAC encoder EDAC is represented in Equation 8. where / is the number of time frames for the input signal of length x, where N = 9 codebooks are utilized.EDACM = {zT,zT, ...,zlT}Equation 8

[0092] Further, DAC is non-generative in itself, where it requires an original audio input to produce tokens, but it provides an efficient latent representation that Hybrid Encoder and Upsampler (HMEU) can model. The DAC encoder, and later decoder, weights are frozen and are used as an off-the-shelf acoustic tokenizer. This freezing ensures that the token space is rich enough to cover the distribution of real audio, and that decoding the generated tokens yields high-quality audio. In the approach, as herein disclosed, zi is assumed to be given from the decoded input audio and zz, ... , z<? are generated such that the decoder produces a realistic musical audio output.

[0093] In some cases, the input audio may have a frequency. In some cases, the audio frequency may be from about 5 kilohertz (kHz) to about 200 kHz. In some cases, the audio frequency may be from about 5 kHz to about 50 kHz. In some cases, the audio frequency may be from about 50 kHz to about 100 kHz. In some cases, the audio frequency may be from about 100 kHz to about 150 kHz. In some cases, the audio frequency may be from about 150 kHz to about 200 kHz. In some cases, the audio frequency may be about 44.1 kHz.

[0094] In some cases, the audio file may be encoded onto a number of tokens per second. In some cases, the audio file may be encoded onto from about 10 tokens per second to about 1,000 tokens per second. In some cases, the audio file may be encoded onto from about 10 tokens per second to about 100 tokens per second. In some cases, the audio file may be encoded onto from about 100 tokens per second to about 250 tokens per second. In some cases, the audio file may be encoded onto from about 250 tokens per second to about 500 tokens per second. In someWSGR Attorney Docket No. 68499-701.601 cases, the audio file may be encoded onto from about 500 tokens per second to about 750 tokens per second. In some cases, the audio file may be encoded onto from about 750 tokens per second to about 1000 tokens per second. In some cases, the audio file may be encoded onto about 86 tokens per second.

[0095] In some cases, a unified token may be utilized. In some cases, each token may provide for a different characteristic or modality. In some cases, the modality may be humming, MIDI, timbre control, or music output, among other tasks throughout the music generation protocol. In some cases, the characteristic may be style, genre, target instrument, intensity, tempo, pitch, or other musical modulations. In some cases, a unified token may encode more than one modality or characteristic. In some cases, a unified token may encode more than one modality in a sequence. In some cases, the unified token may encode more than one modality in a rigid sequence, wherein every unified token has a set order. In some cases, the unified token may encode more than one modality in a flexible sequence, wherein the sequence may vary across unified tokens. In some cases, a unified token may be processed by a decoder. In some cases, a unified token may be processed by a transformer. In some cases, a unified token may be processed by a decoder-only transformer.

[0096] In some cases, the unified token may be transformed using prefix task tokens. For example, a token may be specialized to specify generative intent (e.g., token as <sin+control2music>). For example, a meta tokens may be intended to provide instructions to guide generation of a genre (e.g., a Punk token may be <STYLE:PUNK>), a task (e.g. a task token may be <TASK:TIMBRE_TRANSFER>), or a target instrument (e.g., a Guitar token may be <INSTR:GUITAR>). In some cases, the prefix task tokens may be configured to perform meta-tasks. In some cases, the prefix task tokens may be configured to perform at least one meta-task. In some cases, the prefix task tokens may be configured to perform multiple meta- tasks. In some cases, the prefix task tokens may be configured to perform generative meta-tasks. In some cases, the prefix task tokens may be configured to perform multiple generative meta- tasks. In some cases, the meta-tasks may result in a single instruction for further music generation. In some cases, the single instruction may be a single semantic instruction. In some cases, the single instruction may guide the next task or stage for music generation.

[0097] In some cases, the encoded file may have a sample frame hop. In some cases, the encoded file may have from about 10 sample frame hops to about 1,000 sample frame hops. In some cases, the encoded file may have from about 10 sample frame hops to about 100 sample frame hops. In some cases, the encoded file may have from about 100 sample frame hops toWSGR Attorney Docket No. 68499-701.601 about 250 sample frame hops. In some cases, the encoded file may have from about 250 sample frame hops to about 500 sample frame hops. In some cases, the encoded file may have from about 500 sample frame hops to about 750 sample frame hops. In some cases, the encoded file may have from about 750 sample frame hops to about 1000 sample frame hops. In some cases, the encoded file may have about 512 sample frame hops.

[0098] In some cases, parallel quantized streams, or codebooks, are present. In some cases, from about 2 codebooks to about 50 codebooks may be present. In some cases, from about 2 codebooks to about 10 codebooks may be present. In some cases, from about 10 codebooks to about 20 codebooks may be present. In some cases, from about 20 codebooks to about 30 codebooks may be present. In some cases, from about 30 codebooks to about 40 codebooks may be present. In some cases, from about 40 codebooks to about 50 codebooks may be present. In some cases, about 9 codebooks may be present. In some cases, the parallel streams may be processed in parallel. In some cases, the parallel streams may be processed together, by a central transformer.

[0099] In some cases, the converted audio may be encoded by a program. In some cases, the converted audio may be encoded by an encoder program that encodes or decodes audio data. In some cases, the encoder program may compress the audio data. In some cases, the encoder program may transmit the audio data. In some cases, the encoder program may store the audio data.

[0100] In some cases, the audio codec method may follow a parametric approach. In some cases, the audio codec method may follow a neural audio codec approach. While parametric methods have historically suffered from bandwidth limitations and quality constraints, neural audio codecs have demonstrated remarkable advancements. In some cases, the neural codec approach may be an auto-encoder (AE). In some cases, the neural codec approach may be a neural vocoder. In some cases, the neural codec approach may be a postfilter design.

[0101] In some cases, the separate feature sets may result from the input modalities. In some cases, the number of separate feature sets may be equal to the number of input modalities. In some cases, the number of separate feature sets may be greater than the number of input modalities. In some cases, the number of separate feature sets may be less than the number of input modalities.AR Transformer

[0102] A generative component of the Hybrid Encoder and Upsampler (HMEU) is aWSGR Attorney Docket No. 68499-701.601 transformer-based sequence model, wherein the model predicts the fine codebook tokens z . . . zg, conditioned from the coarse codebook zi. The model applies an encoder-decoder architecture, similar to the Audio Transformer design for multi-level token modeling disclosed herein. The encoder takes the codebook 1 token sequence z1I:T as input, and the decoder autoregressively outputs the sequence of tokens for codebooks 2 through N.Hybrid Encoder and Upsampler (HMEU)

[0103] HMEU 160 is a hybrid neural audio generation model that marries a pretrained compression-based audio tokenizer with a transformer-driven generative model. The HMEU leverages the Descript Audio Codec (DAC) to encode audio into discrete acoustic tokens, then uses a Transformer to generate new audio by predicting fine-grained token sequences. FIG. 3 provides a brief overview of this process 300. First, the split audio features 310 are processed by the AR transformer 320, resulting in predicted melody features 330, as a result of the first codebook. The HMEU 340 processes these initial melody features to upsample through the remaining codebooks 350. The resultant features are finally processed by the DAC decoder 360, resulting in a predicted melody, or musical output 370.

[0104] As an example, given an input audio signal x(t), the DAC encoder produces a hierarchy of N= 9 parallel codebooks (streams of discrete tokens) zi, Z2, ... , zg. The first codebook zi (the coarsest token stream) is treated as conditioning input to HMEU’s transformer. The transformer then autoregressively predicts the tokens for the remaining 8 codebooks Z2 through zg. Finally, the predicted tokens \z,2, ... , z g] are passed (along with zi) into the DAC decoder to reconstruct a waveform x(t). In effect, HMEU uses the strong prior of a neural codec for representation, while introducing a generative modeling capability that DAC alone lacks. This design ensures that high-level musical structure is captured by the coarse codebook zi, and the transformer focuses on generating the fine audio details consistent with that structure.

[0105] In some cases, a masked acoustic token modeling approach may be utilized for codebook upsampling. Specifically, non-autoregressive architecture may be implemented to efficiently upsample from the first DAC codebook to a complete set of codebooks. These codebooks are then decoded to audio using the DAC decoder, as discussed herein. This method may enable faster generation of codebooks compared to autoregressive alternatives while maintaining high audio fidelity. In some cases, the non-autoregressive architecture may have a bidirectional transformer design to process all tokens in parallel through a variable masking schedule. This method may prove particularly effective for codebook upsampling. Additionally, the hierarchical structure, which separates coarse and fine token generation, allows for generation ofWSGR Attorney Docket No. 68499-701.601 detailed audio representations from more abstract melodic specifications. The non- autoregressive architecture may also provide efficient parallel decoding capabilities while focusing its application specifically on translating between different levels of audio representation in the DAC framework.

[0106] First, audio is first encoded into a sequence of discrete tokens via a hierarchical vector quantization scheme, as shown below, where a is an encoder function, Qi is the 7thvector quantizer, Zi is the quantized approximation of residual Ri-i, with Ro = Z. The scheme below is intended to encode and decode the discrete tokens based on a given measure. First, in Equation 9, the audio is encoded into latent representation. Second, in Equation 10, an approximation of Z is provided with leftover residual. Third, in Equation 11, the leftover residual is enhanced, which leads to creation of a matrix for token sequences.Z = e(x) Equation ^

[0107] The original signal may be reconstructed by in Equation 12.Equation 12

[0108] Next, tokens are randomly masked according to a variable masking schedule. Let Y E RTXNrepresent the matrix of encoded tokens, where T is the sequence length and N is the number of codebooks. Further, a set of masked tokens YM and a set of unmasked tokens Yu are utilized. A representation where the objective mis top maximize the likelihood of predicting the true masked tokens given the unmasked tokens is provided in Equation 13.Equation 13

[0109] Next, during inference, tokens are iteratively predicted in parallel using a confidencebased sampling procedure.

[0110] Step 1 : Estimate: For each masked token y E YM, estimate probability distribution over the vocabulary of codebook values V: p(y |Ku, #).

[0111] Step 2: Sample: Generate token estimates y E V by sampling from these distributions.WSGR Attorney Docket No. 68499-701.601

[0112] Step 3: Rank by Confidence: Compute confidence scores with temperature-annealed Gumbel noise: confidence(yr) = log (p(yt)) + temp • gt. Here, gt ~ Gumbel (0,1) and temp is annealed to 0 over sampling iterations.

[0113] Step 4 : Select: Determine the number of tokens to mask k according to a cosine masking schedule, as in Equation 14, where y is the cosine schedule, t is the current iteration, tr is the total iterations, and D is the total number of tokens. The k lowest-confidence token estimates are re-masked, while the remaining high-confidence estimates are kept.Equation 14

[0114] Step 5: Repeat until the desired number of iterations is reached.

[0115] In some cases, a Two-Level Generation strategy may be employed, utilizing two models:

[0116] Coarse Model: Generates the subset of Nccoarse tokens that capture high-level structure, as shown in Equation 15.Equation 15

[0117] Coarse-to-Fine Model: Generates the remaining Nf fine tokens conditioned on the coarse tokens, as shown in Equation 16.Equation 16AR Transformer

[0118] The coarse token sequence z1I:T is then read by an encoder. First, the discrete tokens are embedded into a continuous space. For example, the vocabulary size of the codebook Vi may be established. In some cases, the vocabulary size may be from about 10 to about 10,000. In some cases, the vocabulary size may be from about 10 to about 100. In some cases, the vocabulary size may be from about 100 to about 1,000. In some cases, the vocabulary size may be from about 1,000 to about 5,000. In some cases, the vocabulary size may be from about 5,000 to about 10,000. In some cases, the vocabulary size may be 1,024 ( \Vi\ = 1024). From the vocabulary size, a learned embedding table for codebook 1 may result (Ei:Vi — > Rrf). The embedded sequence iswhere a positional encoding is addedWSGR Attorney Docket No. 68499-701.601 to inject temporal order information. A sequence of hidden states are produced from processing of the embedded sequence with self-attention layer (Ze), represented as h1:T=Trans f ormer Enc(eT), where these encoder states ( / i1:T)serve as conditioning context, summarizing the coarse audio structure into a form that can be attended to by the decoder in future steps. The encoder in this step acts as an analysis transform 150 that computes high-level features from the coarse tokens (such as harmony, rhythm, or timbre cues present in zi) that will guide the generation of detailed content.DAC Decoder

[0119] The DAC decoder 170 then can invert the DAC encoding process, decoding a set of token sequences [z1I.-T, ... , z^z.r] back to a waveform DAC decoder (DDAC) (zi, ... , ZN) ~ X, where N is the number of codebooks. Further, the decoder is a causal transformer that generates a sequence of fine tokensin an autoregressive manner.

[0120] A single flattened target sequence representing all fine codebook tokens in a chose order is constructed. These tokens are generated frame-by-frame in chronological order. For each time step t = 1 to T, the tokens for each of the codebooks are output for time t before generating and outputting tokens for time t + 1. Within each time Z, tokens z ... , zNtare output in an order of ascending codebook index so that the model can capture any dependency of higher-index codebooks on lower-index ones at the same frame. Therefore, a total output token sequence yields a length = T x 8. Within this sequence, a generic target token is «, where n indexes over the sequence of all (t,i) pairs with z {2, ... ,9}. The autoregressive factorization of the generation process is represented as in Equation 17, wheredenotes all previously generated tokens up to codebook z-1 at time t and all tokens at earlier time frame < Z., and N is the number of codebooks.Equation 17

[0121] When predicting the next token in the sequence, the decoder attends to three sources of information: (1) the entire coarse token sequence z .Tvia encoder-decoder cross-attention, (2) any already generated fine tokens at the same time frame Z (for codebooks < z), and (3) all generated fine tokens at earlier time frames < Z (for all codebooks 2-N). This ensures that generation is coherent both temporally (depending on past frames’ tokens) and across codebook levels (each fine token can depend on the coarser tokens of that frame).WSGR Attorney Docket No. 68499-701.601

[0122] The decoder is implemented as La layers of transformer blocks, each consisting of selfattention (masked to ensure causality in the flattened sequence order) followed by cross-attention over the encoder outputs hi.-r, and finally position-wise feedforward networks.

[0123] The final output 180 from the decoder, once finally compiled, produces a musical output.Platform architecture

[0124] The subject matter described herein involves the use of deep learning and artificial intelligence to analyze, compress, and generate music in the style of a particular artist. Provided are music creation platforms, pipelines, methods, systems, media, and applications that facilitate development of games and tools to power a new generation of artists who can create music, perform for live audiences, and interact with fans as well as fan creators who can co-create music with their favorite artists and perform those creations for live audiences and other fans. In particular, Splash Music has developed a cutting-edge generative music platform, specializing in music for the digital generation (i.e., genres like hyperpop, EDM, Glitch, Phonk, Trap, Lo-Fi, Hip-hop, and others) as well as tools to give users the option to search and combine samples based on text prompts or use a powerful generative model trained on this dataset to create unique new production-quality music.

[0125] An objective of components of the music creation platforms, pipelines, methods, systems, media, and applications described herein comprises allowing users to generate a personalized melody which reflects the style of a chosen artist based on audio input, e.g., humming, rapping, singing, speaking, playing an instrument, etc. For example, referring to FIG. 4, in some embodiments, the subject matter described herein 400 enables a user to input audio by humming 402 and generate music in the style of a selected artist 404.

[0126] Components of the platform architecture, in various embodiments, include:

[0127] Basic Pitch: A pre-trained Basic Pitch model to extract MIDI from humming, which is then refined using a proprietary algorithm.

[0128] Lead Music Generator: A surge synthesizer to generate lead music from MIDI. In particular, a machine learning model called “Neural Synth.”

[0129] Pre-generated Music Packs: First, music packs include data manually based on an artist’s music and may include a library of music packs, each based on the music of a particular artist. In some embodiments, a machine learning model automates the generation of music packs from the artist’s music collections, or a subset thereof. In further embodiments, music packs are utilized to inform music matching processes used in music synthesis. Second, music packsWSGR Attorney Docket No. 68499-701.601 include a proprietary database of ethically trained music.

[0130] Splash Music Algorithm 1: Takes music packs and lead music from the surge synthesizer, then combines them to create music in the artist’s style based on the user’s humming melody.

[0131] Splash Music Algorithm 2: Takes music packs and high-quality lead music from the pre generated database, then combines them to create music in the artist’s style based on the user’s humming melody.

[0132] Feature Extractor: Matches the user’s humming with the existing music database by extracting features using pre-trained machine learning models.

[0133] MIDI Search Algorithm: A proprietary search algorithm to find the lead music most closely related to the user’s humming.

[0134] Fine-tuned Basic Pitch: Saves the user’s humming along with the MIDI extracted by the Basic Pitch model. The extracted MIDI is then manually refined. This process helps build a large dataset of MIDI and humming pairs, which is used to fine-tune the Basic Pitch model.

[0135] Splash MIDI to Lead Music Transformer (SplashMT): A foundational model which is continuously trained to generate lead music directly from the user’s humming. In various embodiments, this model can fill-in gaps and / or provide structure.

[0136] Referring to FIG. 5, in a particular embodiment, a music creation platform architecture 500 described herein includes a Basic Pitch model 504, which receives user audio input in the form of, by way of example, a user hum 502. The Basic Pitch model 504 and extracts MIDI 506 from the user hum 502 data, which is then refined using a proprietary algorithm. In this embodiment, the MIDI 506 is utilized by the platform in multiple ways. First, the MIDI 506, along with the user’s audio input, is saved to a Fixed MIDI Database 508 as a MIDI / input pair and used to manually and / or automatically fine tune 510 the Basic Pitch model 504. Second, the MIDI 506 is provided to a Lead Music Generator 525, e.g., a surge synthesizer such as “Neural Synth,” to generate lead music from the MIDI. Generated lead music is used by a First Music Algorithm 530 in combination with a Pre-Generated Music Pack from a Library of Pre- Generated Music Packs 528. Specifically, the First Music Algorithm 530 takes music packs and lead music from the surge synthesizer and utilizes them to create music in the artist’s style based on the user’s humming melody 534 (see also 536). Third, the MIDI 506 provided to a Feature Extractor 522, which matches the user’s humming with existing music by extracting features using pre-trained machine learning models. The extracted features are used to by a MIDI SearchWSGR Attorney Docket No. 68499-701.601Algorithm 520 useful to find pre-generated Lead Music 524 most closely related to the user’s audio input for use by a Second Music Algorithm 532 in combination with a Pre-Generated Music Pack from a Library of Pre-Generated Music Packs 528. The Second Music Algorithm 532 takes music packs and high-quality lead music from the pre-generated database and utilizes them to create music in the artist’s style based on the user’s humming melody 536 (see also 534). Further, in this embodiment, the Fixed MIDI Database 508 is used to train a MIDI to Lead Music Transformer 512, a foundational model which generates lead music directly from the user’s humming. The MIDI to Lead Music Transformer 512 populates a Pre-Generated Lead Music Database 514. A Feature Extractor 516 maintains a Pre-Generated MIDI / Music Feature Vectors Database 518, which also powers the MIDI Search Algorithm 520.

[0137] Components of the NeuralSynth architecture, in various embodiments, include a Basic Pitch model which gives melody lines from music in MIDI format which is then synthesized in audio. The music created based on humming is tailored to the style of the artist chosen by the user.

[0138] Referring to FIG. 6, in a particular embodiment, a surge synthesizer architecture, such as a NeuralSynth architecture 600, described herein includes a Basic Pitch model 604 which provides melody lines from user audio input (music) 602 and extracts MIDI format 606 which is then used to synthesize audio 908. In this embodiment, both the synthesized audio 608, as well as the music 602, are provided to DAC Encoders, 610 and 614, respectively, which generate a series of codes (e.g., code 1, code 2, ... , code ri) for each, 612 and 616, respectively. Further, in this embodiment, the codes are processed by an AR Transformer 618, VampNet620 (a masked acoustic token modeling approach to music synthesis and compression), and by a DAC Encoder 622 to revert the codes back into audio (music) 624 and a Control Net 626 for the particular artist, which controls creation of music in the style of the particular artist 628 by, for example, a diffusion model.

[0139] Two models in particular require training: AR Transformer 1 and AR Transformer 2.

[0140] Referring to FIG. 7, in a particular embodiment, a surge synthesizer training pipeline, such as a NeuralSynth training pipeline, 700 described herein extracts “style” from music to inform audio matching processes in a surge synthesizer. In this embodiment, the music 702 is first processed with, for example, Demucs 704 (a deep learning model for music source separation that works on waveforms) to separate the music into, for example, bass, guitar, vocal, and lead data (collectively 706). By way of example, Lead MIDI 708 is processed by a First AR Transformer 710 to generate First Lead Codes 712, subsequently by a Second AR TransformerWSGR Attorney Docket No. 68499-701.601714 to generate First Music Codes 716, and finally by VampNet 718 to generate a series of Music Codes 720 (e.g., code 1, code 2, ... , code ri). The Music Codes 720 are provided to a DAC Encoder 722 to reverts the codes back into audio (music) 724 and a Control Net 726 for the particular artist, which controls creation of music in the style of the particular artist 728 by, for example, a diffusion model.

[0141] In some cases, the architecture may be configured to generate specific instrumental sounds. In some cases, the instrumental sounds may derive from a percussion instrument, like, for example, a drum or mallet instrument. In some cases, the instrumental sounds may derive from a woodwind instrument, like, for example, a flute or reed instrument . In some cases, the instrumental sounds may derive from a string instrument, like, for example, a guitar, violin, or piano. In some cases, the input audio may comprise a brass instrument, like, for example, a trumpet or trombone. In some cases, the instrumental sounds may derive from an electronic instrument, like, for example, a keyboard or synthesizer.

[0142] In some cases, the architecture may be configured for instrumental sound generation with timbre training, such as that associated with the timbre cues of the AR transformer. In some cases, the architecture may be configured for instrumental sound generation without explicit timbre training. In some cases, timbre training may be replaced by an encoder. In some cases, timbre training may be replaced by a timbre control signal. In some cases, the timbre control signal may be representative of the target instrument in question. In some cases, the timbre control signal may be representative of any instrument where audio generation is of interest. In some cases, the timbre control signal may not be representative of the instrument of interest. In some cases, the timbre control signal may be any suitable input. In some cases, the timbre control signal may be processed by an encoder, producing a timbre-encoded output. In some cases, the timbre-encoded output may be processed. In some cases, the timbre-encoded output may be conditioned along with a hummed input. In some cases, the timbre-encoded output may be conditioned with a processed hummed input. In some cases, the timbre-encoded output may be conditioned with a MIDI-extracted hummed input. In some cases, the timbre-encoded output may be conditioned to preserve the initial hummed melody. In some cases, the timbre-encoded output may be conditioned to adapt the timbre. In some cases, the timbre-encoded output may be conditioned to adapt the timbre dynamically. In some cases, the conditioned output may be an output instrument audio.

[0143] In some cases, the control signal may inform the model of the target instrument’ s timbre profile. In some cases, the control signal may inform the model of a string instrument timbreWSGR Attorney Docket No. 68499-701.601 profile. In some cases, the control signal may inform the model of a woodwind instrument timbre profile. In some cases, the control signal may inform the model of a brass instrument timbre profile. In some cases, the control signal may inform the model of an electronic instrument timbre profile. In some cases, the control signal may inform the model of a percussive instrument timbre profile. In some cases, the timbre profile may provide for, for example, a guitar, flute, saxophone, trumpet, trombone, synthesizer, drums, shakers, or xylophone.

[0144] In some cases, the control signal may have a source to provide the target instrument’s timbre characteristics. In some cases, the source of the control signal may be from the target instrument. In some cases, the source of the control signal may be a sourced recording of the target instrument. In some cases, the source of the control signal may be a user recording of the target instrument. In some cases, the source of the control signal may be user performance of the target instrument. In some cases, the source of the control signal may be a clip from a preset library. In some cases, the source of the control signal may be a clip from a synthesizer preset library. In some cases, the source of the control signal may be a user-provided reference sample. In some cases, the control signal may be a combination of more than one control signal source.

[0145] In some cases, the control signal may have a duration. In some cases, the control signal duration may be dependent on the required timbre. In some cases, the control signal may be from about 1 second to about 30 seconds. In some cases, the control signal may be from about 1 second to about 10 seconds. In some cases, the control signal may be from about 10 seconds to about 20 seconds. In some cases, the control signal may be from about 20 seconds to about 30 seconds. In some cases, the control signal may be from about 1 second to about 5 seconds.

[0146] Referring to FIG. 8, in a particular embodiment, an exemplary instrumental generation framework without timbre training 800, as described herein. In this embodiment, two parallel inputs are introduced: a hummed input 810 and a control signal 820. The hummed input originates from a user and is provided for pitch and rhythm. The hummed input 810 is processed by a MIDI extractor 830. The control signal is intended as the source for timbre. The control signal 820 is processed by a timbre feature encoder 840. Each of these processed inputs (processed hummed input and processed control signal) are then refined together by a timbre conditioning transformer 850. The result from the transformer 850 is an output instrument audio 860User Experience

[0147] Referring to FIGS. 10A-10X, in a particular embodiment, the platforms, pipelines,WSGR Attorney Docket No. 68499-701.601 methods, systems, media, and applications described herein present GUIs offering a first exemplary UX allowing a user to generate personalized music, which reflects the style of a chosen artist, based on user audio input, e.g., humming, rapping, and / or singing, etc. In this embodiment, a user accesses an application, for example, a web application, a mobile web application, or a mobile native application to interact with the user interface. Shown in FIG. 10A, a user accesses previously created music (e.g., songs) or can optionally create a new song. FIG. 10B shows a landing / home screen for a particular artist, in this case a simulated artist. In some cases, the simulated artist can be linked to a particular genre, such as pop, rock, techno, electronica, trance, etc. In some cases, the simulated artist can be linked to a real-life artist. In some cases, the platform may allow for selection of an artist from a list of artists. In some cases, the list of artists may comprise more than one artist. In some cases, the list of artists may comprise more than two artists. In some cases, the list of artists may comprise more than three artists.

[0148] The remainder of this exemplary UX, as shown in FIGS. 10C-10X, is organized via an ongoing interactive chat with the artist. FIG. 10C shows initiation of the chat session and FIG. 10D shows the artist providing some instruction. FIGS. 10E-10G show the user selecting a beat for the new song. FIGS. 10H-100 show the user creating a melody for the new song by, for example, humming. FIGS. 10P-10U show the user creating lyrics for the new song by, for example, singing, rapping, or speaking. FIGS. 10V-10X show finalization of the song and the user’s option to listen to and share the new track.

[0149] Referring to FIGS. 11A-11S, in a particular embodiment, the platforms, pipelines, methods, systems, media, and applications described herein present GUIs offering a second exemplary UX allowing a user to generate personalized music, which reflects the style of a chosen artist, based on user audio input, e.g., humming, rapping, and / or singing, etc. In this embodiment, a user accesses an application, for example, a web application, a mobile web application, or a mobile native application to interact with the user interface. FIG. 11A shows a landing / home screen for a particular artist, in this case a simulated artist. FIG. 11B shows an interface providing user access to music (e.g., songs) previously created with the artist and elements allowing the user to optionally create a new song. The user can listen to the previously created songs, as shown in FIG. 11C. The remainder of this exemplary UX, as shown in FIGS. 11D-11S, is organized via an application interface organized by the user created elements of the music, for example, the beat, the lyrics, and the melody (see FIGS. 11G, 11L, and IIP). FIGS. 11D-11F show screens providing instruction. FIGS. 11G-11H show the user selecting a beat for the new song. FIGS. 11I-11K show the user creating a melody for the new song by, forWSGR Attorney Docket No. 68499-701.601 example, singing, rapping, or speaking. FIGS. 11L-11O show the user creating lyrics for the new song by, for example, singing, rapping, or speaking. FIGS. 11Q-11R show finalization of the song and the user’s option to listen to and share the new track and FIG. IIS shows a list of songs created by the user.

[0150] Referring to FIGS. 12A-12F, in a particular embodiment, the platforms, pipelines, methods, systems, media, and applications described herein present GUIs offering a third exemplary UX allowing a user to generate personalized music, which reflects the style of a chosen artist, based on user audio input, e.g., humming, rapping, and / or singing, etc. In this embodiment, a user accesses an application, for example, a web application, a mobile web application, or a mobile native application to interact with the user interface. FIG. 12A shows an artist selection screen, where a user may scroll through various artists to generate music in their style. FIG. 12B shows a beat selection screen, where a user may select a beat from a series of provided beats. Here, 5 beats are provided: Chum, Stomp, Witness, Hype, and Total. FIG. 12C shows a creation selection screen, where a user may select either a melody or a vocal option. Once selected, the user may record the melody or vocals, depending on the selection. FIG. 12D shows a recording screen, where a user is counted down from three to indicate when the system is recording to collect a melody and / or vocals. FIG. 12E shows a finalization screen, wherein selection of a mixer button allows for system operation to allow for generation of the output composition. FIG. 12F shows a user review screen, where a user can review the final mix, title the mix, and share the mix after the final song has been generated based on the inputs.

[0151] Referring to FIGS. 13A-13B, in a particular embodiment, the platforms, pipelines, methods, systems, media, and applications described herein present GUIs offering a fourth exemplary UX allowing a user to play games based on generated personalized music. FIGS. 13A-13B show screens displaying mini-games, where a user may tap / click the screen to match the rhythm of the user-generated song.

[0152] In some cases, the user may select a beat. In some cases, the user may select a beat from a number of beat options. In some cases, the number of beat options may be from about 1 beat to about 1000 beats. In some cases, the number of beat options may be from about 1 beat to about 250 beats. In some cases, the number of beat options may be from about 250 beats to about 500 beats. In some cases, the number of beat options may be from about 500 beats to about 750 beats. In some cases, the number of beat options may be from about 750 beats to about 1000 beats.

[0153] In some cases, the beat may have the style of the selected artist. In some cases, the styleWSGR Attorney Docket No. 68499-701.601 may be any known musical style, such as rock, pop, R&B, electronic, trance, dubstep, etc. In some cases, the beat may have a temp, wherein the tempo may be from about 40 bpm to about 220 bpm.

[0154] In some cases, the user experience may comprise further product features. In some cases, the user experience may not be connected to other users. In some cases, the user experience may be connected to other users. In some cases, the user experience may comprise reactions, such as likes, hearts, thumbs up, etc. In some cases, the user experience may comprise streamcounts. In some cases, the user experience may index and analyze streamcounts. In some cases, the user experience may allow for personal streamcounts. In some cases, the user experience may allow for multiple users to create music together. In some cases, the user experience may allow for multiple users to create music in parallel. In some cases, the user experience may allow for a user to remix a previous users song. In some cases, the user experience may allow for a user to remix their own previous song. In some cases, the user experience may allow for a user to remix a professionally created song. In some cases, the user experience may allow for a user to remix a professionally released song.

[0155] In some cases, the user experience may comprise post-generation editing. In some cases, the user may rearrange sections of a completed mix. In some cases, the user may mix sections of the completed mix. In some cases, the user may adjust the tempo of the completed mix. In some cases, the user may adjust the style of the completed mix. In some cases, the user may re-record the melody for a completed mix. In some cases, the user may re-record the vocals for a completed mix. FIGS. 14A-14C show screens displaying a screen allowing for remixing of a completed mix, where a user may tap / click the screen to match the rhythm of the user-generated song and rearrange sections of a completed mix.

[0156] In some cases, the user experience may comprise a gaming feature. In some cases, the gaming feature may be a mini-game. In some cases, the mini-games may be based on musical metadata, such as tempo, bpm, etc. In some cases, the gaming features may iterate before the game has been selected by a user. In some cases, the gaming features may be improved after gameplay begins but before a round ends. In some cases, the gaming feature may be based on the rhythm, wherein the rhythm is matched by the user player.

[0157] In some cases, the platform may be integrated into another program. In some cases, the platform may be integrated into a gaming program. In some cases, the platform may be integrated into a sandbox gaming program. In some cases, the platform may be integrated into a social media program. In some cases, the platform may be integrated into a music sharingWSGR Attorney Docket No. 68499-701.601 program. In some cases, the platform may be integrated into another music creation program. In some cases, the platform may be a standalone program. In some cases, the stand-alone program may be a computer application. In some cases, the stand-alone program may be a console-based application. In some cases, the stand-alone program may be a mobile application, wherein the mobile application is lOS / Android compatible. In some cases, the stand-alone program may be a mobile native application.Computing System

[0158] Referring to FIG. 18, a block diagram is shown depicting an exemplary machine that includes a computer system 1800 (e.g., a processing or computing system) within which a set of instructions can execute for causing a device to perform or execute any one or more of the aspects and / or methodologies for static code scheduling of the present disclosure. The components in FIG. 18 are examples only and do not limit the scope of use or functionality of any hardware, software, embedded logic component, or a combination of two or more such components implementing particular embodiments.

[0159] Computer system 1800 may include one or more processors 1801, a memory 1803, and a storage 1808 that communicate with each other, and with other components, via a bus 1840. The bus 1840 may also link a display 1832, one or more input devices 1833 (which may, for example, include a keypad, a keyboard, a mouse, a stylus, etc.), one or more output devices 1834, one or more storage devices 1835, and various tangible storage media 1836. All of these elements may interface directly or via one or more interfaces or adaptors to the bus 1840. For instance, the various tangible storage media 1836 can interface with the bus 1840 via storage medium interface 1826. Computer system 1800 may have any suitable physical form, including but not limited to one or more integrated circuits (ICs), printed circuit boards (PCBs), mobile handheld devices (such as mobile telephones or PDAs), laptop or notebook computers, distributed computer systems, computing grids, or servers.

[0160] Computer system 1800 includes one or more processor(s) 1801 (e.g., central processing units (CPUs), general purpose graphics processing units (GPGPUs), or quantum processing units (QPUs)) that carry out functions. Processor(s) 1801 optionally contains a cache memory unit 1802 for temporary local storage of instructions, data, or computer addresses. Processor(s) 1801 are configured to assist in execution of computer readable instructions. Computer system 1800 may provide functionality for the components depicted in FIG. 18 as a result of the processor(s) 1801 executing non-transitory, processor-executable instructions embodied in one or more tangible computer-readable storage media, such as memory 1803, storage 1808, storage devicesWSGR Attorney Docket No. 68499-701.6011835, and / or storage medium 1836. The computer-readable media may store software that implements particular embodiments, and processor(s) 1801 may execute the software. Memory 1803 may read the software from one or more other computer-readable media (such as mass storage device(s) 1835, 1836) or from one or more other sources through a suitable interface, such as network interface 1820. The software may cause processor(s) 1801 to carry out one or more processes or one or more steps of one or more processes described or illustrated herein. Carrying out such processes or steps may include defining data structures stored in memory 1803 and modifying the data structures as directed by the software.

[0161] The memory 1803 may include various components (e.g., machine readable media) including, but not limited to, a random access memory component (e.g., RAM 1804) (e.g., static RAM (SRAM), dynamic RAM (DRAM), ferroelectric random access memory (FRAM), phasechange random access memory (PRAM), etc.), a read-only memory component (e.g., ROM 1605), and any combinations thereof. ROM 1805 may act to communicate data and instructions unidirectionally to processor(s) 1801, and RAM 1804 may act to communicate data and instructions bidirectionally with processor(s) 1801. ROM 1805 and RAM 1804 may include any suitable tangible computer-readable media described below. In one example, a basic input / output system 1806 (BIOS), including basic routines that help to transfer information between elements within computer system 1800, such as during start-up, may be stored in the memory 1803.

[0162] Fixed storage 1808 is connected bidirectionally to processor(s) 1801, optionally through storage control unit 1807. Fixed storage 1808 provides additional data storage capacity and may also include any suitable tangible computer-readable media described herein. Storage 1808 may be used to store operating system 1809, executable(s) 1810, data 1811, applications 1812 (application programs), and the like. Storage 1808 can also include an optical disk drive, a solid- state memory device (e.g., flash-based systems), or a combination of any of the above. Information in storage 1808 may, in appropriate cases, be incorporated as virtual memory in memory 1803.

[0163] In one example, storage device(s) 1835 may be removably interfaced with computer system 1800 (e.g., via an external port connector (not shown)) via a storage device interface 1825. Particularly, storage device(s) 1835 and an associated machine-readable medium may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 1800. In one example, software may reside, completely or partially, within a machine-readable medium on storage device(s)WSGR Attorney Docket No. 68499-701.6011835. In another example, software may reside, completely or partially, within processor(s) 1801

[0164] Bus 1840 connects a wide variety of subsystems. Herein, reference to a bus may encompass one or more digital signal lines serving a common function, where appropriate. Bus 1840 may be any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures. As an example and not by way of limitation, such architectures include an Industry Standard Architecture (ISA) bus, an Enhanced ISA (EISA) bus, a Micro Channel Architecture (MCA) bus, a Video Electronics Standards Association local bus (VLB), a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, an Accelerated Graphics Port (AGP) bus, HyperTransport (HTX) bus, serial advanced technology attachment (SATA) bus, and any combinations thereof.

[0165] Computer system 1800 may also include an input device 1833. In one example, a user of computer system 1800 may enter commands and / or other information into computer system 1800 via input device(s) 1833. Examples of an input device(s) 1833 include, but are not limited to, an alpha-numeric input device (e.g., a keyboard), a pointing device (e.g., a mouse or touchpad), a touchpad, a touch screen, a multi-touch screen, a joystick, a stylus, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), an optical scanner, a video or still image capture device (e.g., a camera), and any combinations thereof. In some embodiments, the input device is a Kinect, Leap Motion, or the like. Input device(s) 1833 may be interfaced to bus 1840 via any of a variety of input interfaces 1823 (e.g., input interface 1823) including, but not limited to, serial, parallel, game port, USB, FIREWIRE, THUNDERBOLT, or any combination of the above.

[0166] In particular embodiments, when computer system 1800 is connected to network 1830, computer system 1800 may communicate with other devices, specifically mobile devices and enterprise systems, distributed computing systems, cloud storage systems, cloud computing systems, and the like, connected to network 1830. Communications to and from computer system 1800 may be sent through network interface 1820. For example, network interface 1820 may receive incoming communications (such as requests or responses from other devices) in the form of one or more packets (such as Internet Protocol (IP) packets) from network 1830, and computer system 1800 may store the incoming communications in memory 1803 for processing. Computer system 1800 may similarly store outgoing communications (such as requests or responses to other devices) in the form of one or more packets in memory 1803 andWSGR Attorney Docket No. 68499-701.601 communicated to network 1830 from network interface 1820. Processor(s) 1801 may access these communication packets stored in memory 1803 for processing.

[0167] Examples of the network interface 1820 include, but are not limited to, a network interface card, a modem, and any combination thereof. Examples of a network 1830 or network segment 1830 include, but are not limited to, a distributed computing system, a cloud computing system, a wide area network (WAN) (e.g., the Internet, an enterprise network), a local area network (LAN) (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a direct connection between two computing devices, a peer-to-peer network, and any combinations thereof. A network, such as network 1830, may employ a wired and / or a wireless mode of communication. In general, any network topology may be used.

[0168] Information and data can be displayed through a display 1832. Examples of a display 1832 include, but are not limited to, a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic liquid crystal display (OLED) such as a passive-matrix OLED (PMOLED) or active-matrix OLED (AMOLED) display, a plasma display, and any combinations thereof. The display 1832 can interface to the processor(s) 1801, memory 1803, and fixed storage 1808, as well as other devices, such as input device(s) 1833, via the bus 1840. The display 1832 is linked to the bus 1840 via a video interface 1822, and transport of data between the display 1832 and the bus 1840 can be controlled via the graphics control 1821. In some embodiments, the display is a video projector. In some embodiments, the display is a headmounted display (HMD) such as a VR headset. In further embodiments, suitable VR headsets include, by way of non-limiting examples, HTC Vive, Oculus Rift, Samsung Gear VR, Microsoft HoloLens, Razer OSVR, FOVE VR, Zeiss VR One, Avegant Glyph, Freefly VR headset, and the like. In still further embodiments, the display is a combination of devices such as those disclosed herein.

[0169] In addition to a display 1832, computer system 1800 may include one or more other peripheral output devices 1834 including, but not limited to, an audio speaker, a printer, a storage device, and any combinations thereof. Such peripheral output devices may be connected to the bus 1840 via an output interface 1824. Examples of an output interface 1824 include, but are not limited to, a serial port, a parallel connection, a USB port, a FIREWIRE port, a THUNDERBOLT port, and any combinations thereof.

[0170] In addition or as an alternative, computer system 1800 may provide functionality as a result of logic hardwired or otherwise embodied in a circuit, which may operate in place of orWSGR Attorney Docket No. 68499-701.601 together with software to execute one or more processes or one or more steps of one or more processes described or illustrated herein. Reference to software in this disclosure may encompass logic, and reference to logic may encompass software. Moreover, reference to a computer-readable medium may encompass a circuit (such as an IC) storing software for execution, a circuit embodying logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware, software, or both.

[0171] Those of skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality.

[0172] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0173] The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by one or more processor(s), or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.WSGR Attorney Docket No. 68499-701.601

[0174] In accordance with the description herein, suitable computing devices include, by way of non-limiting examples, cloud computing platforms, distributed computing platforms, server clusters, server computers, desktop computers, laptop computers, notebook computers, subnotebook computers, netbook computers, netpad computers, set-top computers, media streaming devices, handheld computers, Internet appliances, mobile smartphones, tablet computers, personal digital assistants, video game consoles, and vehicles. Those of skill in the art will also recognize that select televisions, video players, and digital music players with optional computer network connectivity are suitable for use in the system described herein. Suitable tablet computers, in various embodiments, include those with booklet, slate, and convertible configurations, known to those of skill in the art.

[0175] In some embodiments, the computing device includes an operating system configured to perform executable instructions. The operating system is, for example, software, including programs and data, which manages the device’s hardware and provides services for execution of applications. Those of skill in the art will recognize that suitable server operating systems include, by way of non-limiting examples, FreeBSD, OpenBSD, NetBSD®, Linux, Apple® Mac OS X Server®, Oracle® Solaris®, Windows Server®, and Novell® NetWare®. Those of skill in the art will recognize that suitable personal computer operating systems include, by way of nonlimiting examples, Microsoft® Windows®, Apple® Mac OS X®, UNIX®, and UNIX-like operating systems such as GNU / Linux®. In some embodiments, the operating system is provided by cloud computing. Those of skill in the art will also recognize that suitable mobile smartphone operating systems include, by way of non-limiting examples, Nokia® Symbian® OS, Apple® iOS®, Research In Motion® BlackBerry OS®, Google® Android®, Microsoft® Windows Phone® OS, Microsoft® Windows Mobile® OS, Linux®, and Palm® WebOS®. Those of skill in the art will also recognize that suitable media streaming device operating systems include, by way of non-limiting examples, Apple TV®, Roku®, Boxee®, Google TV®, Google Chromecast®, Amazon Fire®, and Samsung® HomeSync®. Those of skill in the art will also recognize that suitable video game console operating systems include, by way of non-limiting examples, Sony® PS3®, Sony® PS4®, Microsoft® Xbox 360®, Microsoft Xbox One, Nintendo® Wii®, Nintendo® Wii U®, and Ouya®.Non-Transitory Computer Readable Storage Medium

[0176] In some embodiments, the platforms, pipelines, methods, systems, media, and applications described herein include one or more non-transitory computer readable storage media encoded with a program including instructions executable by the operating system of anWSGR Attorney Docket No. 68499-701.601 optionally networked computing device. In further embodiments, a computer readable storage medium is a tangible component of a computing device. In still further embodiments, a computer readable storage medium is optionally removable from a computing device. In some embodiments, a computer readable storage medium includes, by way of non-limiting examples, CD-ROMs, DVDs, flash memory devices, solid state memory, magnetic disk drives, magnetic tape drives, optical disk drives, distributed computing systems including cloud computing systems and services, and the like. In some cases, the program and instructions are permanently, substantially permanently, semi-permanently, or non-transitorily encoded on the media.Computer Program

[0177] In some embodiments, the platforms, pipelines, methods, systems, media, and applications described herein include at least one computer program, or use of the same. A computer program includes a sequence of instructions, executable by one or more processor(s) of the computing device’s CPU, written to perform a specified task. Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), computing data structures, and the like, that perform particular tasks or implement particular abstract data types. In light of the disclosure provided herein, those of skill in the art will recognize that a computer program may be written in various versions of various languages.

[0178] The functionality of the computer readable instructions may be combined or distributed as desired in various environments. In some embodiments, a computer program comprises one sequence of instructions. In some embodiments, a computer program comprises a plurality of sequences of instructions. In some embodiments, a computer program is provided from one location. In other embodiments, a computer program is provided from a plurality of locations. In various embodiments, a computer program includes one or more software modules. In various embodiments, a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plug-ins, extensions, add-ins, or add-ons, or combinations thereof.Web Application

[0179] In some embodiments, a computer program includes a web application. In light of the disclosure provided herein, those of skill in the art will recognize that a web application, in various embodiments, utilizes one or more software frameworks and one or more database systems. In some embodiments, a web application is created upon a software framework such asWSGR Attorney Docket No. 68499-701.601Microsoft® .NET or Ruby on Rails (RoR). In some embodiments, a web application utilizes one or more database systems including, by way of non-limiting examples, relational, non-relational, object oriented, associative, XML, and document oriented database systems. In further embodiments, suitable relational database systems include, by way of non-limiting examples, Microsoft® SQL Server, mySQL™, and Oracle®. Those of skill in the art will also recognize that a web application, in various embodiments, is written in one or more versions of one or more languages. A web application may be written in one or more markup languages, presentation definition languages, client-side scripting languages, server-side coding languages, database query languages, or combinations thereof. In some embodiments, a web application is written to some extent in a markup language such as Hypertext Markup Language (HTML), Extensible Hypertext Markup Language (XHTML), or extensible Markup Language (XML). In some embodiments, a web application is written to some extent in a presentation definition language such as Cascading Style Sheets (CSS). In some embodiments, a web application is written to some extent in a client-side scripting language such as Asynchronous JavaScript and XML (AJAX), Flash® ActionScript, JavaScript, or Silverlight®. In some embodiments, a web application is written to some extent in a server-side coding language such as Active Server Pages (ASP), ColdFusion®, Perl, Java™, JavaServer Pages (JSP), Hypertext Preprocessor (PHP), Python™, Ruby, Tel, Smalltalk, WebDNA®, or Groovy. In some embodiments, a web application is written to some extent in a database query language such as Structured Query Language (SQL). In some embodiments, a web application integrates enterprise server products such as IBM® Lotus Domino®. In some embodiments, a web application includes a media player element. In various further embodiments, a media player element utilizes one or more of many suitable multimedia technologies including, by way of non-limiting examples, Adobe® Flash®, HTML 5, Apple® QuickTime®, Microsoft® Silverlight®, Java™, and Unity®.

[0180] Referring to FIG. 19, in a particular embodiment, an application provision system comprises one or more databases 1900 accessed by a relational database management system (RDBMS) 1910. Suitable RDBMSs include Firebird, MySQL, PostgreSQL, SQLite, Oracle Database, Microsoft SQL Server, IBM DB2, IBM Informix, SAP Sybase, Teradata, and the like. In this embodiment, the application provision system further comprises one or more application severs 1920 (such as Java servers, .NET servers, PHP servers, and the like) and one or more web servers 1930 (such as Apache, IIS, GWS and the like). The web server(s) optionally expose one or more web services via app application programming interfaces (APIs) 1940. Via a network, such as the Internet, the system provides browser-based and / or mobile native user interfaces.

[0181] Referring to FIG. 20, in a particular embodiment, an application provision systemWSGR Attorney Docket No. 68499-701.601 alternatively has a distributed, cloud-based architecture 2000 and comprises elastically load balanced, auto-scaling web server resources 2010 and application server resources 2020 as well synchronously replicated databases 2030.Mobile Application

[0182] In some embodiments, a computer program includes a mobile application provided to a mobile computing device. In some embodiments, the mobile application is provided to a mobile computing device at the time it is manufactured. In other embodiments, the mobile application is provided to a mobile computing device via the computer network described herein.

[0183] In view of the disclosure provided herein, a mobile application is created by techniques known to those of skill in the art using hardware, languages, and development environments known to the art. Those of skill in the art will recognize that mobile applications are written in several languages. Suitable programming languages include, by way of non-limiting examples, C, C++, C#, Objective-C, Java™, JavaScript, Pascal, Object Pascal, Python™, Ruby, VB.NET, WML, and XHTML / HTML with or without CSS, or combinations thereof.

[0184] Suitable mobile application development environments are available from several sources. Commercially available development environments include, by way of non-limiting examples, AirplaySDK, alcheMo, Appcelerator®, Celsius, Bedrock, Flash Lite, .NET Compact Framework, Rhomobile, and WorkLight Mobile Platform. Other development environments are available without cost including, by way of non-limiting examples, Lazarus, MobiFlex, MoSync, and PhoneGap. Also, mobile device manufacturers distribute software developer kits including, by way of non-limiting examples, iPhone and iPad (iOS) SDK, Android™ SDK, BlackBerry® SDK, BREW SDK, Palm® OS SDK, Symbian SDK, webOS SDK, and Windows® Mobile SDK.

[0185] Those of skill in the art will recognize that several commercial forums are available for distribution of mobile applications including, by way of non-limiting examples, Apple® App Store, Google® Play, Chrome WebStore, BlackBerry® App World, App Store for Palm devices, App Catalog for webOS, Windows® Marketplace for Mobile, Ovi Store for Nokia® devices, Samsung® Apps, and Nintendo® DSi Shop.Standalone Application

[0186] In some embodiments, a computer program includes a standalone application, which is a program that is run as an independent computer process, not an add-on to an existing process, e.g., not a plug-in. Those of skill in the art will recognize that standalone applications are oftenWSGR Attorney Docket No. 68499-701.601 compiled. A compiler is a computer program(s) that transforms source code written in a programming language into binary object code such as assembly language or machine code. Suitable compiled programming languages include, by way of non-limiting examples, C, C++, Objective-C, COBOL, Delphi, Eiffel, Java™, Lisp, Python™, Visual Basic, and VB .NET, or combinations thereof. Compilation is often performed, at least in part, to create an executable program. In some embodiments, a computer program includes one or more executable complied applications.Web Browser Plug-In

[0187] In some embodiments, the computer program includes a web browser plug-in (e.g., extension, etc.). In computing, a plug-in is one or more software components that add specific functionality to a larger software application. Makers of software applications support plug-ins to enable third-party developers to create abilities which extend an application, to support easily adding new features, and to reduce the size of an application. When supported, plug-ins enable customizing the functionality of a software application. For example, plug-ins are commonly used in web browsers to play video, generate interactivity, scan for viruses, and display particular file types. Those of skill in the art will be familiar with several web browser plug-ins including, Adobe® Flash® Player, Microsoft® Silverlight®, and Apple® QuickTime®. In some embodiments, the toolbar comprises one or more web browser extensions, add-ins, or add-ons. In some embodiments, the toolbar comprises one or more explorer bars, tool bands, or desk bands.

[0188] In view of the disclosure provided herein, those of skill in the art will recognize that several plug-in frameworks are available that enable development of plug-ins in various programming languages, including, by way of non-limiting examples, C++, Delphi, Java™, PHP, Python™, and VB .NET, or combinations thereof.

[0189] Web browsers (also called Internet browsers) are software applications, designed for use with network-connected computing devices, for retrieving, presenting, and traversing information resources on the World Wide Web. Suitable web browsers include, by way of nonlimiting examples, Microsoft® Internet Explorer®, Mozilla® Firefox®, Google® Chrome, Apple® Safari®, Opera Software® Opera®, and KDE Konqueror. In some embodiments, the web browser is a mobile web browser. Mobile web browsers (also called microbrowsers, mini-browsers, and wireless browsers) are designed for use on mobile computing devices including, by way of nonlimiting examples, handheld computers, tablet computers, netbook computers, subnotebook computers, smartphones, music players, personal digital assistants (PDAs), and handheld videoWSGR Attorney Docket No. 68499-701.601 game systems. Suitable mobile web browsers include, by way of non-limiting examples, Google® Android® browser, RIM BlackBerry® Browser, Apple® Safari®, Palm® Blazer, Palm® WebOS® Browser, Mozilla® Firefox® for mobile, Microsoft® Internet Explorer® Mobile, Amazon® Kindle® Basic Web, Nokia® Browser, Opera Software® Opera® Mobile, and Sony® PSP™ browser.Software Modules

[0190] In some embodiments, the platforms, pipelines, methods, systems, media, and applications described herein include software, server, and / or database modules, or use of the same. In view of the disclosure provided herein, software modules are created by techniques known to those of skill in the art using machines, software, and languages known to the art. The software modules disclosed herein are implemented in a multitude of ways. In various embodiments, a software module comprises a file, a section of code, a programming object, a programming structure, a distributed computing resource, a cloud computing resource, or combinations thereof. In further various embodiments, a software module comprises a plurality of files, a plurality of sections of code, a plurality of programming objects, a plurality of programming structures, a plurality of distributed computing resources, a plurality of cloud computing resources, or combinations thereof. In various embodiments, the one or more software modules comprise, by way of non-limiting examples, a web application, a mobile application, a standalone application, and a distributed or cloud computing application. In some embodiments, software modules are in one computer program or application. In other embodiments, software modules are in more than one computer program or application. In some embodiments, software modules are hosted on one machine. In other embodiments, software modules are hosted on more than one machine. In further embodiments, software modules are hosted on a distributed computing platform such as a cloud computing platform. In some embodiments, software modules are hosted on one or more machines in one location. In other embodiments, software modules are hosted on one or more machines in more than one location.Databases

[0191] In some embodiments, the platforms, pipelines, methods, systems, media, and applications described herein include one or more databases, or use of the same. In view of the disclosure provided herein, those of skill in the art will recognize that many databases are suitable for storage and retrieval of, by way of examples, user, artist, music pack, beat, melody, lyric, and model information. In various embodiments, suitable databases include, by way of non-limiting examples, relational databases, non-relational databases, object oriented databases,WSGR Attorney Docket No. 68499-701.601 object databases, entity -relationship model databases, associative databases, XML databases, document oriented databases, and graph databases. Further non-limiting examples include SQL, PostgreSQL, MySQL, Oracle, DB2, Sybase, and MongoDB. In some embodiments, a database is Internet-based. In further embodiments, a database is web-based. In still further embodiments, a database is cloud computing-based. In a particular embodiment, a database is a distributed database. In other embodiments, a database is based on one or more local computer storage devices.EXAMPLES

[0192] The following illustrative examples are representative of experiments conducted with elements of the platforms, pipelines, methods, systems, media, and applications described herein and are not meant to be limiting in any way.Example 1 — Generating Music from Melodic Condition

[0193] Research efforts were conducted to develop a novel architecture to create a Melody Generation model; specifically, to train a neural network model to generate music conditioned on melodic content, such as humming, playing a piano, or a recorded song. The model was constrained by the melodic input, requiring thorough research exploration. Also investigated was the key preprocessing required to achieve this goal with noisy input, such as humming with background noise. This problem, referred to as “audio-to-audio,” remains an open research question, necessitating the generation of new knowledge.Technical Problems Addressed

[0194] Melody Extraction: The ability to accurately extract melodic information from a given piece of music, providing a representation of the melody.

[0195] Model Development: Developing a model that can interpret this melody representation and generate coherent music following the given melody.

[0196] Training vs. Real-Time Inference: Addressing the difference between training data and real-time inference, ensuring the model works effectively with inputs like humming.

[0197] Preprocessing: Learning about the key preprocessing required to achieve this goal with noisy input, such as humming with background noise.WSGR Attorney Docket No. 68499-701.601Technical Improvements Developed

[0198] Enhanced Product Capabilities: Successfully developing a melody-based music generation model will enhance our product, allowing users to hum a tune and have the system generate corresponding music.

[0199] New Dataset Creation: Creating new datasets that follow any given melody, expanding our training resources.

[0200] Innovative Music Generation: Generating new music based on any melody from existing music, opening up new creative possibilities.

[0201] Augmentation of In-House Loop Data: Augmenting our in-house loop data with generated music, providing richer and more diverse training data.

[0202] New knowledge was developed regarding the ability to create a neural network which adheres to a melody input signal (e.g., humming, etc.), extracting the key melody, to then generate music adhering to that melody. For example:

[0203] How to acquire or construct a dataset to train a model to adhere to melodic inputs.

[0204] Techniques required. Specifically, whether audio should be converted to MIDI (Musical Instrument Digital Interface), whether raw pitch content should be extracted in the form of sine waves, or whether raw audio should be used for the model to learn how to align the key melodic content.

[0205] Control mechanisms, such as audio prompts, categorical data like genre, and free text like lyrics, can be included to enhance the generated music.

[0206] Existing solutions are insufficient for our quality standards, as they rarely adhered closely to the input prompt.Input variables included:

[0207] Various types of melody inputs: including raw humming, piano and other instruments, raw sine waves, and other songs with melodic content.

[0208] Songs created from our loops with key melodic content extracted, as per input type above.

[0209] Songs collected from publicly available sources, where we collected almost 100,000 hours of music, with the melodic content extracted.WSGR Attorney Docket No. 68499-701.601

[0210] Use of preexisting models to support our novel architectures, including CLAP, DAC and Encodec, as well as a proprietary audio codec. All models were customized models to fit the specific requirements, thereby altering the original model’s architecture and functionality.

[0211] Proprietary model design.Output variables included:

[0212] Adherence to melodic input, and other input variables such as audio prompts.

[0213] Objective metrics including F rechet Audio Distance to compare generated music with distributions of non-generative music.

[0214] Subjective measures like Mean Opinion Score to measure subjective quality of outputs.

[0215] Inference speed: how rapidly our models can generate music, aiming for real-time or better.

[0216] Model capacity: How long the model can be trained before it starts to perform worse on the test set, as measured by cross-entropy loss.

[0217] The set of input variables were strategically manipulated and the identified output variables consistently measured to achieve greater speed, accuracy, scale, and quality in music generation while adhering to a melodic input. Experiments were conducted to investigate the hypothesis and to extend the current state of knowledge on effective models for music generation. These include:

[0218] Testing various methods of input feature extraction, effectively building different datasets that could be used to train a model to generate new music that adheres to an input melody.

[0219] Different representations of the input melody, including tokenized MIDI, raw sine waves, and unprocessed audio.

[0220] Testing different architectural configurations, trying to find key ones that can adhere to melodic input but also generate high-quality musical content.

[0221] Testing different methods for training models.

[0222] Outcomes were tested using a combination of objective and subjective listening measures. However, adherence to melody content was a key distinguishing factor in these evaluations. That said, the quality of musical output was paramount to getting the best results.WSGR Attorney Docket No. 68499-701.601

[0223] In a first iteration, utilizing a supporting activity development to extract pitch information using Chromagrams for audio, including a loop dataset and utilized a tokenized representation of chroma to audio as our objective. We achieved a key log loss score of 3.2, and in our subjective tests, felt some adherence to melodic content, although insufficient.

[0224] In a second iteration, we aimed to improve the quality of our training data, by first utilizing off-the-shelf stem extraction tooling, like Demucs, to first extract the key melody audio, and then generating Chromagrams. This improved our subjective adherence and did not impact quality. However, Chromagrams did not provide sufficient controls for the input, and were limited in the expressiveness of the melody input provided, particularly it did not allow for editing of the input pre-generative.

[0225] In a third iteration, we experimented with MIDI tokenization, utilizing two off-the-shelf models: Basic Pitch and Essentia to extract key melodic information as represented by MIDI tokens. Here we found that the model had more difficulty adhering to melody input, which we assumed was due to the cardinality of MIDI data. For example, pitch_60 and pitch_61 are totally different tokens to the model, even though they should be closely related.

[0226] In a fourth iteration, we experimented with raw sine waves as input. Instead of feed MIDI tokens, we converted the MIDI to sine waves and used that as audio-to-audio input. This allowed us to reuse the existing preprocessing, but also ensured that the model was not learning a new vocab of tokens, but it was represented as audio. These results provide the best performance of 2.1 on log loss, with some promise with subjective testing.

[0227] Research efforts were conducted to develop a novel architecture to create a Melody Generation model; specifically, to train a neural network model to generate music conditioned on melodic content, such as humming, playing a piano, or a recorded song. The model was constrained by the melodic input, requiring thorough research exploration. Also investigated was the key preprocessing required to achieve this goal with noisy input, such as humming with background noise. This problem, referred to as “audio-to-audio,” remains an open research question, necessitating the generation of new knowledge.Example 2 — Performance Evaluation

[0228] To quantitatively assess the performance of our method, three complementary metrics STFT Distance, MEL Distance and SI-SDR were employed. These complementary metrics evaluated different aspects of audio quality and fidelity. These metrics allow us toWSGR Attorney Docket No. 68499-701.601 comprehensively evaluate both time-frequency representation accuracy and perceptual quality of generated audio.Short-Time Fourier Transform (STFT) Distance

[0229] STFT Distance measures the discrepancy between the time-frequency representations of reference and generated audio signals. This measurement is described in Equation 18, where x represents the reference audio, x represents the generated audio, and T and F are the time and frequency dimensions respectively.Equation 18

[0230] Here, a 1024-point FFT with a hop length of 256 samples and a Hann window was used , averaging the LI distances across all time-frequency bins. This metric is particularly sensitive to phase differences and temporal misalignments, providing insight into the model’s ability to reconstruct precise spectro-temporal patterns.MEL Distance

[0231] The MEL Distance evaluates differences in the mel-frequency domain, which better aligns with human auditory perception than the linear frequency scale used in STFT. The measurement is provided in Equation 19.Equation 19

[0232] Mel-spectrograms using 128 mel bands spanning 20Hz to 20kHz were computed. By emphasizing perceptually relevant frequency bands, MEL Distance provides a more perceptually aligned measure of audio fidelity than raw STFT Distance. Lower MEL Distance values typically correlate with improved perceptual quality, especially for timbral and harmonic content.Scale-Invariant Signal-to-Distortion Ratio (SI-SDR)

[0233] Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) evaluates the overall fidelity of the generated audio while being invariant to arbitrary scaling factors. The measurement is provided in Equation 20, where a is the optimal scaling factor:WSGR Attorney Docket No. 68499-701.601Equation 20

[0234] SI-SDR is measured in decibels (dB) with higher values indicating better performance. Unlike the previous two metrics, SI-SDR operates directly in the time domain, making it complementary to the spectral -based metrics. It is particularly effective at capturing temporal distortions and overall signal fidelity.Example 3 — Example Dataset Development and Performance Testing

[0235] To provide a model with extensive training context, a large-scale synthetic dataset was developed rather than relying on licensed musical material. This approach prioritizes the technical aspects of audio reproduction — specifically the accurate rendering of various pitches, note durations, and transitional characteristics — over musical artistry.

[0236] In this examples, 50 base melodic patterns with 25 variations of each, yielding 1,250 unique melodies were systematically generated. Each melody adheres to specific constraints: a maximum duration of 5 seconds, a pitch range from C3 to G5 (using Roland MIDI frequency standard), and composition within one of eleven musical modes (Ionian, Dorian, Phrygian, Lydian, Mixolydian, Aeolian, Locrian, Harmonic Minor, Double Harmonic, Phrygian Dominant, and Minor Blues). The tempo range varies from 60 to 200 BPM, providing rhythmic diversity across the corpus.

[0237] For the audio rendering phase, a Surge XT, a state-of-the-art open-source wavetable synthesizer, was employed. A total of 50 distinct synthesizer presets (patches) representative of contemporary electronic music lead instruments were carefully designed and stored in the standard .fxp format. Each of the 1,250 MIDI files was then rendered using all 50 presets, resulting in a final corpus of 62,500 high-quality audio samples.

[0238] This comprehensive dataset enabled robust evaluation of neural audio synthesis models across diverse timbral, harmonic, and rhythmic contexts. The systematic variation across musical parameters facilitated the investigation of how synthesis models respond to specific musical features while maintaining consistent experimental conditions. Dataset distribution data can be seen in FIGS. 15A and 15B. FIG. 15A provides a proportional distribution of musical scales showing the prevalence of different scale types in the dataset. FIG. 15B provides box plots showing the tempo distribution (BPM) for each scale type, with median values annotated. Of note, there are some characteristic tempo ranges for different scales, with Minor Blues and Mixolydian scales having higher median tempos compared to Harmonic Minor and Locrian scales.WSGR Attorney Docket No. 68499-701.601

[0239] The dataset was organized hierarchically. Each melodic pattern was represented by a MIDI file (e.g., melody aeolian l 35bpm_000049_var00. mid). For each MIDI file, there existed a corresponding folder with the same base name. Within each folder, 50 WAV files were stored (from 01-surge-lead.wav to 50-surge-lead.wav), each representing the same melodic pattern rendered with a different synthesizer preset.

[0240] The naming convention for the MIDI files follows the pattern melody_[scale]_[bpm]bpm_[id]_var[variation]. Here, [scale] indicates the musical scale (e.g., aeolian, ionian); [bpm] represents the tempo in beats per minute; [id] is a unique numeric identifier for the melodic pattern; and var[variation] denotes different variations of the melodic pattern.

[0241] Each melodic pattern was carefully crafted to represent a variety of musical styles, rhythmic patterns, and melodic contours. By rendering each MIDI file with 50 different synthesizer presets, a wide spectrum of timbral characteristics was captured while preserving the underlying melodic content. This approach allowed the model to learn the relationship between melodic patterns and their various timbral realizations. FIG. 16 provides a visualization of an example melodic pattern. The left side shows the MIDI piano roll representation while the right side displays a x lOgrid of waveform and spectrogram visualizations for all 50 audio variations.

[0242] The synthesizer presets were designed to span multiple categories of sounds typically found in electronic music production. All audio files were rendered at 44.1kHz sample rate with 16-bit depth, providing high-quality audio suitable for detailed timbral analysis. The average duration of each audio sample was approximately 5 seconds.

[0243] For objective evaluation, the chroma cosine similarity metric was utilized. The chroma cosine similarity metric quantifies the spectral similarity between the generated audio samples and their reference counterparts. This metric operates by computing the average cosine similarity between corresponding time frames of quantized chroma representations. The chroma cosine similarity metric is defined in Equation 21, where xt* xtdenotes the dot product between the reference and generated chroma vectors at time step t. |x / |2 and |x / |2 are the L2 norms of the chroma vectors of ground truth and generated signal and 7' is the total number of time frames in the chromagrams.Equation 21WSGR Attorney Docket No. 68499-701.601

[0244] An AWS Trainium infrastructure on SageMaker Hyperpod clusters was employed as the backbone of the training pipeline. A customized Transformer Decoder was used to predict subsequent symbolic music events from preceding token sequences. The model architecture comprises 48 self-attention layers, each featuring 32 attention heads and a hidden size of 1,440. During training, the Adam optimizer was utilized with an initial learning rate of X 10-5. The pre-training procedure spans [X hours / days] with a batch size of [Y] across 4 AWS Trainium nodes. To effectively manage memory consumption and enhance throughput, the pipeline integrates advanced parallelism techniques — including tensor parallelism (TP), data parallelism (DP), Flash Attention, and ZeRO-1 optimization. Real-time monitoring was achieved via the AIM package, TensorBoard visualization, and Slurm -based job scheduling. This optimized framework yielded a 54.4% reduction in training costs, thereby establishing a scalable and robust foundation for the innovative music generation technology, as disclosed herein.

[0245] Performance metric data can be seen in FIG. 17. Notably, phrygian and ionian modes achieved the highest melodic accuracy with chroma similarity scores of 0.8970 and 0.8891 respectively, while locrian mode performed worst (0.8125), suggesting that the model has varying capabilities in handling different musical structures. The SI-SDR metric shows significant variance (ranging from 3.69 to 5.43), with aeolian and ionian modes performing best, indicating that certain scale patterns may be inherently more predictable for the neural network. MEL and STFT loss metrics follow similar patterns across modes, with ionian and phrygian consistently outperforming other scales, demonstrating that these modes likely contain more recognizable patterns that the model can effectively capture. These findings suggest that future model training could benefit from targeted augmentation of underperforming modes or architectural modifications to better handle the complexity of scales like Locrian that show consistently weaker performance.Example 4 — Performance Evaluation

[0246] An ablation study was also completed to evaluate different non-autoregressive architecture configuration for recovering DAC codebooks from the first codebook, with subsequent waveform generation. Here, a first codebook was used to recover Codebooks 2 to 9, wherein 9 total codebooks were utilized. The evaluation shows progressive improvement in the non-autoregressive architecture’s ability to recover DAC codebooks 2-9 from only the first codebook. Evaluation data can be seen in Table 1 below.WSGR Attorney Docket No. 68499-701.601Table 1 - Performance of hybrid non-autoregressive architecture on DAC codebook recovery task

[0247] While preferred embodiments of the present subject matter have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the present subject matter. It should be understood that various alternatives to the embodiments of the present subject matter described herein may be employed in practicing the present subject matter.

Claims

WSGR Attorney Docket No. 68499-701.601CLAIMSWHAT IS CLAIMED IS:

1. A computer-implemented method of generative music creation comprising: a) maintaining a library of music packs, each music pack comprising elements based on the musical style of an artist; b) receiving a selection of an artist from a user; c) receiving an audio input comprising a melody from the user; d) extracting Musical Instrument Digital Interface (MIDI) data from the audio input; e) generating lead music from the MIDI data; f) extracting features from the MIDI data; g) applying a search algorithm using the extracted features to match the audio input to mostly closely related pre-generated lead music from a database of pregenerated lead music; and h) creating music in the artist’s style based at least in part on the generated lead music and the matched pre-generated lead music.

2. The method of claim 1, further comprising receiving a selection of a beat from a user.

3. The method of claim 2, wherein the music in the artist’s style is further based on the selected beat.

4. The method of claim 1, wherein one or more of the music packs are generated by a machine learning model from a music collection for an artist.

5. The method of claim 1, wherein the audio input comprises humming.

6. The method of claim 1, wherein the audio input comprises speaking, singing, or rapping.

7. The method of claim 1, further comprising receiving an additional audio input comprising lyrics from the user.

8. The method of claim 7, wherein the additional audio input comprises speaking, singing, or rapping.

9. The method of claim 7, further comprising applying a model to extract MIDI data from the additional audio input.WSGR Attorney Docket No. 68499-701.60110. The method of claim 9, wherein the music in the artist’s style is further based on the MIDI data extracted from the additional audio input.

11. The method of claim 1, further comprising continuously training a foundational model to generate lead music based on the audio input.

12. The method of claim 11, wherein the foundational model is trained using the music packs to inform one or more audio matching processes.

13. The method of claim 1, wherein the selected artist’s style is implemented via a control net for a diffusion model.

14. A computer-implemented system comprising a computing device comprising at least one processor and instructions executable by the at least one processor to provide a generative music creation application comprising: a) a library of music packs, each music pack comprising elements based on the musical style of an artist; b) a user interface receiving a selection of an artist from a user and an audio input comprising a melody from the user; c) a software module extracting Musical Instrument Digital Interface (MIDI) data from the audio input; d) a software module generating lead music from the MIDI data; e) a software module extracting features from the MIDI data; f) a software module applying a search algorithm using the extracted features to match the audio input to mostly closely related pre-generated lead music from a database of pre-generated lead music; and g) a software module creating music in the artist’s style based at least in part on the generated lead music and the matched pre-generated lead music.

15. The system of claim 14, wherein the library of music packs are generated by a machine learning model from a music collection for an artist.

16. The system of claim 14, wherein the selection of the artist is based on the audio input.

17. The system of claim 14, wherein the audio input from the user comprises humming.

18. The system of claim 14, wherein the audio input from the user comprises speaking, singing, or rapping.WSGR Attorney Docket No. 68499-701.60119. The system of claim 14, wherein the system may further comprise an additional audio input.

20. The system of claim 19, wherein the additional audio input comprises lyrics from the user.

21. The system of claim 20, wherein the lyrics are conveyed by speaking, singing, or rapping.

22. The system of claim 19, wherein the software module of (c) is applied to the additional audio input, extracting MIDI data.

23. The system of claim 22, wherein the music created in the artists’ style is altered by the MIDI data of the additional audio input.

24. The system of claim 14, wherein the module of (g) is continuously training to generate music based on an audio input.

25. The system of claim 24, wherein the continuous training are trained by the library of music packs to inform one or more audio matching processes.

26. The system of claim 14, wherein the artist’s style is implemented via a control net for a diffusion model.

27. A non-transitory computer-readable storage media encoded with instructions executable to cause the one or more processors to perform generative music creation operations comprising: a) maintaining a library of music packs, each music pack comprising elements based on the musical style of an artist; b) receiving a selection of an artist and an audio input comprising a melody from the user; c) extracting Musical Instrument Digital Interface (MIDI) data from the audio input; d) generating lead music from the MIDI data; e) extracting features from the MIDI data; f) applying a search algorithm using the extracted features to match the audio input to mostly closely related pre-generated lead music from a database of pregenerated lead music; andWSGR Attorney Docket No. 68499-701.601 g) creating music in the artist’s style based at least in part on the generated lead music and the matched pre-generated lead music.

28. The storage media of claim 27, wherein the library of music packs are generated by a machine learning model from a music collection.

29. The storage media of claim 27, wherein music in the artist’s style is based on the audio input.

30. The storage media of claim 27, wherein the audio input of (b) comprises humming.

31. The storage media of claim 27, wherein the audio input of (b) comprises speaking, singing, or rapping.

32. The storage media of claim 27, wherein the storage media further comprises an additional audio input.

33. The storage media of claim 32, wherein the additional audio input comprises lyrics provided by the user.

34. The storage media of claim 33, wherein the lyrics are conveyed by speaking, singing, or rapping.

35. The storage media of claim 32, wherein the additional audio input comprises MIDI data, where the MIDI data is extracted as in (c).

36. The storage media of claim 35, wherein the MIDI data modulates the music in the artist’s style.

37. The storage media of claim 27, wherein the storage media is continuously training to generate music based on the audio input.

38. The storage media of claim 37, wherein the continuous training for music generation is based on the additional audio input.

39. The storage media of claim 37, wherein the continuous training for music generation is further trained by the library of music packs to inform one or more audio matching processes.

40. The storage media of claim 27, wherein the artist’s style is implemented via a control net for a diffusion model.

Images