Comparison training for music generator

Abstract

To provide an improved music generation system for automatically generating a new music content based on image representation of an audio file.SOLUTION: A music generation system includes: a music generation subsystem and a music classification subsystem. The music generation subsystem generates an output music content according to a music parameter for defining a policy for generating music. The music classification subsystem is used for classifying whether music is a music content generated by the music generation subsystem or generated by an expert. The music generation subsystem implements an algorithm emphasized by prediction output from the music classification subsystem. The emphasized algorithm includes adjusting the music parameter for generating the music content which is closer to that by humans.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] TECHNICAL FIELD This disclosure relates to audio engineering, and in particular to the creation of musical content. [Background technology]

[0002] Streaming music services typically provide songs to users over the Internet. Users may subscribe to these services and stream music via a web browser or application. Examples of such services include PANDORA, SPOTIFY, GROOVESHARK, etc. Often, users can select a genre of music or a particular artist to stream. Users can usually rate songs (e.g., using a star rating or a like / dislike system), and some music services may tailor which songs to stream to users based on prior ratings. The costs of operating a streaming service (which may include paying royalties for streamed songs) are covered by user subscription fees and / or advertisements played during songs.

[0003] Song selection may be limited by licensing agreements and the number of songs written for a particular genre, users may become bored of hearing the same songs in a particular genre, and these services may not tailor music to a user's tastes, environment, behavior, etc. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 illustrates an example music generator. [Diagram 2] FIG. 2 is a block diagram illustrating an example overview of a system for generating output musical content based on input from multiple different sources, according to some embodiments. [Diagram 3]FIG. 3 is a block diagram illustrating an example music generation system configured for training music generation and classification of music content according to some embodiments. [Figure 4] FIG. 4 is a block diagram illustrating an example music classification subsystem according to some embodiments. [Diagram 5] FIG. 5 is a block diagram illustrating an example music control implementation in a music generation subsystem, according to some embodiments. [Figure 6] FIG. 6 is a block diagram illustrating an example training system for a music classification subsystem, according to some embodiments. [Figure 7] FIG. 7 is a block diagram illustrating an example music generator with a composition subsystem and a performance subsystem according to some embodiments. [Figure 8] FIG. 8 is a block diagram illustrating a more detailed embodiment of a composition and performance subsystem, according to some embodiments. [Figure 9A] FIG. 9A is a block diagram illustrating a graphical user interface according to some embodiments. [Figure 9B] FIG. 9B is a block diagram illustrating a graphical user interface according to some embodiments. [Figure 10] FIG. 10 is a block diagram illustrating an example music generator system including an analysis and composition module according to some embodiments. [Figure 11] FIG. 11 illustrates an example build-up section of music content according to some embodiments. [Figure 12] FIG. 12 illustrates an example technique for arranging selections of music content, according to some embodiments. [Figure 13] FIG. 13 is a flow diagram of a method for training a music generation system according to some embodiments. [Figure 14] FIG. 14 is a flow diagram illustrating an example method according to some embodiments. [Figure 15]FIG. 15 is a flow diagram illustrating an example method according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] U.S. Patent Application No. 13 / 969,372 (now U.S. Patent No. 8,812,144), filed August 16, 2013, which is incorporated herein by reference in its entirety, describes techniques for generating musical content based on one or more musical attributes. To the extent any interpretation is based on a perceived discrepancy between the definitions in U.S. Patent Application No. 13 / 969,372 and the remainder of this disclosure, this disclosure shall govern. The musical attributes may be input by a user or may be determined based on environmental information such as ambient noise, lighting, etc. The disclosure of U.S. Patent Application No. 13 / 969,372 describes techniques for selecting stored loops and / or tracks or generating new loops / tracks and overlaying the selected loops / tracks to generate output musical content.

[0006] U.S. Patent Application No. 16 / 420456 (now U.S. Patent No. 10,679,596), filed May 23, 2019, which is incorporated herein by reference in its entirety, describes techniques for generating music content. To the extent any interpretation is based on a perceived discrepancy between the definitions in U.S. Patent Application No. 16 / 420456 and the remainder of this disclosure, this disclosure shall govern. Music may be generated based on user input or using a computer-implemented method. Various music generator embodiments are described in the disclosure of U.S. Patent Application No. 16 / 420456.

[0007] As used herein, "audio file" refers to audio information for musical content. For example, audio information may include data describing the musical content in a format such as wav, aiff, or FLAC as raw audio. Properties of the musical content may be included in the audio information. The properties may include quantifiable musical properties such as instrument classification, pitch transcription, beat timing, tempo, file length, and audio amplitude at multiple frequency bins. In some embodiments, an audio file includes sound information over a particular time interval. In various embodiments, an audio file includes a loop. The term "loop" as used herein refers to audio information of a single instrument over a particular time interval. Various techniques described with reference to audio files may also be performed with a loop that includes a single instrument. An audio file or loop may be played repeatedly (e.g., a 30-second audio file may be played four times in succession to generate two minutes of musical content), but may also be played only once, for example, without repeats.

[0008] This disclosure first describes an example music generation module and overall system configuration with multiple applications with reference to Figures 1 and 2. Techniques for training the music generation system are described with reference to Figures 3-6. Techniques for music generation are described with reference to Figures 7-8. Figures 9A-9B show example application interfaces.

[0009] In some embodiments, one or more classifiers are trained on a server system (e.g., in the cloud) while reinforcement learning is performed on the user device to improve music generation. The disclosed techniques advantageously enable customized user music generator modules based on a global classifier model used by multiple systems, reducing training time and costs and allowing for centralized training yet personalized results.

[0010] In general, the disclosed music generator includes audio files, metadata (e.g., information describing the audio files), and techniques for combining audio files based on the metadata. The generator may create a music experience using rules that identify audio files based on the target characteristics and metadata of the music experience. It may be configured to extend the set of experiences that can be created by adding or modifying rules, audio files, and / or metadata. Adjustments may be made manually (e.g., an artist adds new metadata) or the music generator may extend the rules / audio files / metadata as it monitors the music experience in a given environment and the desired goals / characteristics. Listener-defined controls may be implemented, for example, to obtain user feedback regarding the goals or characteristics of the music.

[0011] Example Music Generator Overview 1 is a diagram illustrating an example music generator according to some embodiments. In the illustrated embodiment, the music generator module 160 receives various information from a number of different sources and generates the output music content 140.

[0012] In the illustrated embodiment, the module 160 accesses the stored audio files and corresponding attributes 110 for the stored audio files and combines the audio files to generate the output musical content 140. In some embodiments, the music generator module 160 selects the audio files based on their attributes and combines the audio files based on the target musical attributes 130. In some embodiments, the audio files may be selected based on the environmental information 150 in combination with the target musical attributes 130. In some embodiments, the environmental information 150 is used indirectly to determine the target musical attributes 130. In some embodiments, the target musical attributes 130 are explicitly specified by the user, for example, by specifying a desired energy level, mood, multiple parameters, etc. For example, the listener-defined controls described herein may be implemented to specify user preferences that are used as the target musical attributes. Examples of target musical attributes 130 include, for example, energy, complexity, and variety, although more specific attributes (e.g., corresponding to attributes of stored tracks) may also be specified. In general, if higher-level target musical attributes are specified, lower-level specific musical attributes may be determined by the system before generating the output musical content.

[0013] Complexity may refer to the number of audio files, loops, and / or instruments included in a composition. Energy may be related to or orthogonal to other attributes. For example, changing the key or tempo may affect the energy. However, for a given tempo and key, the energy may be changed by adjusting the type of instruments (e.g., by adding hi-hats or white noise), the complexity, the volume, etc. Variety may refer to the amount of change over time in the generated music. Variety may be generated for an otherwise static set of musical attributes (e.g., by selecting different tracks for a given tempo and key) or by varying the musical attributes over time (e.g., by changing tempo and key more frequently if greater variety is desired). In some embodiments, the target musical attributes are considered to exist in a multi-dimensional space, and the music generator module 160 may move slowly through that space, e.g., correcting course as necessary, based on environmental changes and / or user input.

[0014] In some embodiments, the attributes stored with the audio files include information about one or more audio files including tempo, volume, energy, variety, spectrum, envelope, modulation, periodicity, rise time, decay time, noise, artist, instrument, theme, etc. Note that in some embodiments, the audio files are partitioned such that a set of one or more audio files is specific to a particular audio file type (e.g., one instrument or one type of instrument).

[0015] In the illustrated embodiment, the module 160 accesses a stored rule set 120. In some embodiments, the stored rule set 120 specifies rules on how many audio files to layer together to be played simultaneously (which may correspond to the complexity of the output music), which primary / secondary key progressions to use when transitioning between audio files or musical phrases, and which instruments to use together (e.g., instruments that have an affinity with each other) to achieve the target musical attributes. In other words, the music generator module 160 uses the stored rule set 120 to achieve one or more declarative goals defined by the target musical attributes (and / or the target environment information). In some embodiments, the music generator module 160 includes one or more pseudo-random number generators configured to introduce pseudo-randomness to avoid repetitive output music.

[0016] In some embodiments, the environmental information 150 includes one or more of lighting information, ambient noise, user information (facial expressions, body position, activity level, movement, skin temperature, performance of a particular activity, type of clothing, etc.), temperature information, purchasing activity in an area, time of day, day of the week, season, number of people present, weather, etc. In some embodiments, the music generator module 160 does not receive / process the environmental information. In some embodiments, the environmental information 150 is received by another module that determines the target musical attributes 130 based on the environmental information. The target musical attributes 130 may be derived based on other types of content, e.g., video data. In some embodiments, the environmental information is used to adjust one or more stored rule sets 120, e.g., to achieve one or more environmental goals. Similarly, the music generator may use the environmental information to adjust stored attributes for one or more audio files, e.g., to indicate target musical attributes or target audience characteristics with which those audio files are particularly associated.

[0017] The term "module" as used herein means a physical, non-transitory, computer-readable medium that stores information (e.g., program instructions) that instruct a circuit or other circuit (e.g., a processor) configured to perform certain operations to perform certain operations. A module may be implemented in a number of ways, such as as hardwired circuitry or as a memory that stores program instructions executable by one or more processors to perform operations. Hardware circuits may include, for example, custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may be implemented in a programmable hardware device such as a field programmable gate array, programmable array logic, programmable logic device, or the like. A module may be any suitable form of non-transitory computer-readable medium that stores program instructions executable to perform certain operations.

[0018] As used herein, the term "music content" refers to both music itself (a representation of the music) and information that can be used to play music. Thus, a song recorded as a file on a storage medium (such as, but not limited to, a compact disc, a flash drive, etc.) is an example of music content, and sound produced by playing (e.g., via a speaker) this recorded file or other electronic representation is also an example of music content.

[0019] The term "music" includes its well-understood meaning, which includes sounds produced by musical instruments and vocal sounds. Thus, music includes, for example, instrumental performances or recordings, a capella performances or recordings, and performances or recordings that include both instruments and voices. Those skilled in the art will recognize that "music" does not encompass all vocal recordings. For example, works that do not contain musical attributes such as rhythm or rhyme, such as speeches, newscasts, and audiobooks, are not music.

[0020] One musical "content" may be distinguished from another musical content in any suitable way. For example, a digital file corresponding to a first song may represent the first musical content, whereas a digital file corresponding to a second song may represent the second musical content. The term "musical content" may also be used to distinguish specific intervals within a given musical work. Thus, different parts of the same song may be considered as different musical content. Similarly, different tracks (e.g., piano track, guitar track) within a given musical work may correspond to different musical content. In the context of a potentially infinite stream of generated music, the term "musical content" may be used to represent a portion of the stream (e.g., a few bars or minutes).

[0021] The musical content generated by the embodiments of the present disclosure may be "new musical content" that is a combination of musical elements never before generated. The related (and more expansive) concept of "original musical content" is explained in more detail below. To facilitate the explanation of this term, the concept of a "controlling entity" is explained in relation to an instance of musical content generation. Unlike the expression "original musical content", the expression "new musical content" does not imply the concept of a controlling entity. Thus, new musical content refers to musical content that has never before been generated by any entity or computer system.

[0022] Conceptually, this disclosure refers to some "entity" as controlling a particular instance of computer-generated musical content. Such an entity owns legal rights (e.g., copyright) that may correspond to the computer-generated content (to the extent that such rights actually exist). In one embodiment, the individual who creates the computer-implemented musical generator (e.g., codes the various software routines) or operates (e.g., provides inputs) a particular instance of computer-implemented musical generation is the controlling entity. In other embodiments, a computer-implemented musical generator may be created by a legal entity (e.g., a corporation or other business organization) in the form of a software product, computer system, computer device, or the like. In some cases, such a computer-implemented musical generator may be deployed to many clients. Depending on the terms of the license associated with the distribution of this musical generator, the controlling entity may be the creator, distributor, or client in various cases. In the absence of such an explicit legal agreement, the controlling entity of a computer-implemented musical generator is the entity that facilitates (e.g., provides inputs and thereby operates) a particular instance of computer-generated musical content.

[0023] In the sense of this disclosure, computer generation of "original musical content" by a controlled entity means 1) a combination of musical elements not previously generated by either the controlled entity or anyone else, and 2) a combination of musical elements previously generated, but originally generated by the controlled entity. Here, content type 1) means "new musical content," and the definition of "new musical content" is similar to the definition of "new musical content," except that the definition of "new musical content" implies the concept of a "controlled entity," whereas the definition of "new musical content" does not. On the other hand, content type 2) means here "proprietary music content." Note that the term "ownership" in this context does not imply any implied legal rights in the content (although such rights may exist), but is used merely to indicate that the musical content was originally generated by the controlled entity. Thus, a controlled entity's "re-generation" of musical content previously and originally generated by the controlled entity constitutes "generation of original musical content" for the purposes of this disclosure. "Non-original musical content" with respect to a particular controlled entity is musical content that is not "original musical content" to that controlled entity.

[0024] A portion of musical content may include musical components from one or more other musical content. Creating musical content in this manner is called "sampling" the musical content and is common in certain musical works, especially in certain genres. Such musical content is referred to herein as "musical content having a sampled component," "derived musical content," or other similar terms. In contrast, musical content that does not include a sampled component is referred to herein as "musical content without a sampled component," "non-derived musical content," or other similar terms.

[0025] In applying these terms, when a particular musical content is broken down to a sufficient level of granularity, it may be argued that the musical content is derivative (substantially all musical content is derivative). The terms "derivative" and "non-derivative" are not used in this disclosure. With respect to computer-generated musical content, if the computer-generated selects some components from pre-existing musical content of an entity other than the controlling entity (e.g., a computer program selects certain portions of an audio file of a work of a particular artist for inclusion in the generated musical content), such computer-generated is said to be derivative (resulting in derivative musical content). On the other hand, if the computer-generated musical content does not utilize components of such pre-existing content, the computer-generated is said to be non-derivative (resulting in non-derivative musical content). It should be noted that some "original musical content" may be derivative musical content and some may be non-derivative musical content.

[0026] It should be noted that the term "derivative" in this disclosure is intended to have a broader meaning than the term "derivative work" as used in U.S. copyright law. For example, derivative musical content may or may not be a derivative work under U.S. copyright law. The term "derivative" in this disclosure is not intended to convey a negative connotation, but is merely used to imply whether a particular musical content "borrows" portions of content from other works.

[0027] Additionally, the terms "new musical content," "new musical content," and "original musical content" are not intended to encompass only musical content that is trivially different from existing combinations of musical elements. For example, merely changing a few notes of an existing musical work does not constitute new, novel, or original musical content as the terms are used in this disclosure. Similarly, merely changing the key or tempo of an existing musical work or adjusting the relative intensities of frequencies (e.g., using an equalizer interface) does not create new, novel, or original musical content. Furthermore, the terms new and original musical content are not intended to cover musical content that is on the border between original and non-original content. Rather, such terms are intended to cover unquestionably and demonstrably original musical content, including musical content that may be subject to copyright protection for a controlling entity (hereinafter "protectable" musical content). Furthermore, as used herein, "available" musical content refers to musical content that does not infringe the copyrights of any controlled party other than the controlled party. New and / or original musical content is often protected and available. This has advantages in preventing copying of music content and / or paying royalties on music content.

[0028] Although various embodiments described herein use a rule-based engine, various other types of computer-implemented algorithms may be used for any of the computer learning and / or music generation techniques described herein, however, a rule-based approach may be particularly effective in a musical context.

[0029] Overview of Applications, Storage Elements, and Data that May be Used in an Exemplary Music System The music generator module may interact with multiple different applications, modules, storage elements, etc. to generate musical content. For example, an end user may install one of multiple types of applications for different types of computing devices (e.g., mobile devices, desktop computers, DJ equipment, etc.). Similarly, another type of application may be provided for an enterprise user. By interacting with the application while generating musical content, the music generator may receive external information that the music generator may use to identify target musical attributes and / or update one or more rule sets used to generate musical content. In addition to interacting with one or more applications, the music generator module may interact with other modules to receive rule sets, update rule sets, etc. Finally, the music generator module may access one or more rule sets, loops, and / or generated musical content stored in one or more storage elements. Additionally, the music generator module may store any of the above listed items in one or more storage elements that may be local or accessed over a network (e.g., cloud-based).

[0030] 2 is a block diagram illustrating an example overview of a system for generating output musical content based on inputs from multiple disparate sources. In the illustrated embodiment, the system 200 includes a rules module 210, a user application 220, a web application 230, an enterprise application 240, an artist application 250, an artist rule generator module 260, a generated music store 270, and an external input 280.

[0031] In the illustrated embodiment, the user application 220, the web application 230, and the enterprise application 240 receive external inputs 280. In some embodiments, the external inputs 280 include environmental inputs, target music attributes, user inputs, sensor inputs, etc. In some embodiments, the user application 220 is installed on the user's mobile device and includes a graphical user interface that allows the user to interact / communicate with the rules module 210. In some embodiments, the web application 230 is not installed on the user device, but is configured to run within the user device's browser and can be accessed via a website. In some embodiments, the enterprise application 240 is an application used by a large entity to interact with the music generator. In some embodiments, the application 240 is used in combination with the user application 220 and / or the web application 230. In some embodiments, the application 240 communicates with one or more external hardware devices and / or sensors to collect information about the surrounding environment.

[0032] In the illustrated embodiment, the rules module 210 communicates with user applications 220, web applications 230, and enterprise applications 240 to generate the output musical content. In some embodiments, the music generator 160 is included in the rules module 210. Note that the rules module 210 may be included in one of the applications 220, 230, and 240 or may be installed on a server and accessed over a network. In some embodiments, the applications 220, 230, and 240 receive the generated output musical content from the rules module 210 and play the content. In some embodiments, the rules module 210 may request input from the applications 220, 230, and 240 regarding, for example, target musical attributes and environmental information, and use this data to generate the musical content.

[0033] In the illustrated embodiment, the stored rule set 120 is accessed by the rules module 210. In some embodiments, the rules module 210 modifies and / or updates the stored rule set 120 based on communications with applications 220, 230, and 240. In some embodiments, the rules module 210 accesses the stored rule set 120 to generate the output musical content. In the illustrated embodiment, the stored rule set 120 may include rules from an artist rule generator module 260, which is described in more detail below.

[0034] In the illustrated embodiment, the stored rule set 120 is accessed by the rules module 210. In some embodiments, the rules module 210 modifies and / or updates the stored rule set 120 based on communications with applications 220, 230, and 240. In some embodiments, the rules module 210 accesses the stored rule set 120 to generate the output musical content. In the illustrated embodiment, the stored rule set 120 may include rules from an artist rule generator module 260, which is described in more detail below.

[0035] In the illustrated embodiment, the artist application 250 communicates with an artist rule generator module 260 (which may be part of the same application or cloud-based, for example). In some embodiments, the artist application 250 may allow an artist to create a rule set for a particular sound, for example based on a previous composition. This functionality is further described in U.S. Pat. No. 1,067,959. In some embodiments, the artist rule generator module 260 is configured to store the artist rule set for use by the rules module 210. A user may purchase the rule set from a particular artist before using it to generate output music via a particular application. A rule set for a particular artist may be called a signature pack.

[0036] In the illustrated embodiment, the stored audio files and corresponding attributes 110 are accessed by module 210 when applying the rules to select and combine tracks to generate the output musical content. In the illustrated embodiment, rules module 210 stores the generated output musical content in storage element 270.

[0037] 2 are implemented on a server and accessed over a network, sometimes referred to as a cloud-based implementation. For example, stored rule sets 120, audio files / attributes 110, and generated music 270 may all be stored on the cloud and accessed by module 210. In another example, module 210 and / or module 260 may also be implemented in the cloud. In some embodiments, generated music 270 is stored in the cloud and digitally watermarked. This allows, for example, for the mass generation of custom music content in addition to detection of copies of generated music.

[0038] In some embodiments, one or more of the disclosed modules are configured to generate other types of content in addition to musical content. For example, the system may be configured to generate output visual content based on target musical attributes, determined environmental conditions, a currently used rule set, etc. For example, the system may search a database or the Internet based on the current attributes of the music being generated and display a collage of images that dynamically change as the music changes and match the attributes of the music.

[0039] Example machine learning approaches As described herein, the music generator module 160 shown in FIG. 1 may implement various artificial intelligence (AI) techniques (e.g., machine learning techniques) to generate the output musical content 140. In various embodiments, the AI ​​techniques implemented include a combination of deep neural networks (DNNs) and more traditional machine learning techniques and knowledge-based systems. This combination may align the strengths and weaknesses of each of these techniques with the challenges inherent to music composition and personalization systems. Music content has multiple levels of structure. For example, a song has sections, phrases, melodies, notes, and textures. Some machine learning techniques may be effective at analyzing and generating high and low levels of detail in musical content. For example, a DNN may be good at classifying sound textures as belonging to a clarinet or electric guitar at a low level and detecting verses and choruses at a high level. Intermediate levels of detail in musical content, such as melodic construction, orchestration, etc., may be more challenging. DNNs are typically good at capturing a wide range of styles in a single model, so DNNs may be implemented as expressive generative tools.

[0040] In some embodiments, the music generator module 160 leverages expertise by having human-composed audio files (e.g., loops) as the basic unit of musical content used by the music generator module. For example, the social context of expertise can be embedded through rhythmic, melodic, and textural selection to record heuristics at multiple levels of structure. Unlike the separation of DNNs and traditional machine learning based on structural levels, expertise can be applied to any domain where musicality can be enhanced without imposing too strong limitations on the trainability of the music generator module 160.

[0041] In some embodiments, the music generator module 160 uses DNNs to find patterns in how layers of audio are combined vertically by overlapping sounds with each other and horizontally by combining audio files or loops into sequences. For example, the music generator module 160 may implement an LSTM (long short-term memory) recurrent neural network trained on MFCC (Mel Frequency Cepstral Coefficients) audio features of loops used in a multi-track audio recording. In some embodiments, the network is trained to predict and select audio features of a loop for the next beat of music based on knowledge of audio features of the previous beat. For example, the network may be trained to predict audio features of the next 8 beat loop based on knowledge of audio features of the last 128 beats. Thus, the network is trained to utilize low-dimensional feature representations to predict the next beat.

[0042] In certain embodiments, the music generator module 160 uses known machine learning algorithms to assemble sequences of multi-track audio into musical structures with dynamics of intensity and complexity. For example, the music generator module 160 may implement a hierarchical hidden Markov model, which may behave like a state machine with state transitions with probabilities determined by a multi-level hierarchical structure. As an example, a particular type of drop is more likely to occur after a build-up section, and less likely if there are no drums at the end of the build-up. In various embodiments, the probabilities may be trained transparently, as opposed to DNN training, where what is being learned is more opaque.

[0043] Markov models may not be easily trained by presenting example tracks because they may deal with larger temporal structures and the examples may be too long. A feedback control element (such as thumbs up or down on a user interface) may be used to give feedback about the music at any time. The correlation between the music structure and the feedback may then be used to update the structural model used for the composition, such as a transition table or a Markov model. This feedback may be collected directly from measurements of heart rate, sales, or other metrics that allow the system to determine clear classifications. The expert knowledge heuristics described above are also designed to be probabilistic where possible, and are trained in the same way as the Markov models.

[0044] In certain embodiments, the training may be performed by a composer or a DJ. Such training may be separate from the listener training. For example, the training performed by the listener (such as a typical user) may be limited to identifying correct or incorrect classifications based on positive and negative model feedback, respectively. In the case of the composer and the DJ, the training may include hundreds of time steps and may include details about the layers and volume controls used, providing clear details about what drives the changes in the music content. For example, the training performed by the composer and the DJ may include sequence prediction training similar to the global training of the DNN described above.

[0045] In various embodiments, the DNN is trained to predict interactions with the audio interface that a DJ may have made given the most recently played sequence of music. In some embodiments, these interactions may be recorded and used to develop new heuristics that are more transparent. In some embodiments, the DNN receives as input many previous bars of music and utilizes the low-dimensional feature representation described above with additional features that describe changes to the track that the DJ or composer applied. For example, the DNN may receive as input the last 32 bars of music and utilize a low-dimensional feature representation with additional features that describe changes to the track that the DJ or composer applied. These changes may include adjustments to the gain of a particular track, applied filters, delays, etc. For example, a DJ may use the same drum loop that is repeated for 5 minutes during a performance, but gradually increase the gain and delay of the track over time. Thus, the DNN may be trained to predict such gain and delay changes in addition to loop selection. If no loops are played for a particular instrument (e.g., no drum loops are played), then the feature set for that instrument may be all-zero, which may allow the DNN to learn that predicting all-zeros can be a successful strategy, leading to selective stratification.

[0046] In some cases, DJs or composers record live performances using mixers and equipment such as TRAKTOR (Native Instruments GmbH). These recordings are typically captured in high resolution (e.g., 4-track recording or MIDI). In some embodiments, the system breaks down the recording into its constituent loops to generate information about the combination of loops in the composition in addition to the quality of the individual loops. This information is used to train the DNN, giving it the ability to relate both the composition (e.g., ordering, layering, loop timing, etc.) and the quality of the loops to inform the music generator module 160 how to create a musical experience similar to the artist's performance, without using the actual loops that the artist used in the performance.

[0047] Example Music Generator with Comparative Training One technique for generating musical content combines generator and classifier (or "discriminator") subsystems to produce new artifacts in the generated music. One named technique is "generative adversarial networks" (or GANs). GANs utilize an approach in which a music generator is trained to "trick" the classifier into making incorrect classifications, and the classifier uses correct and incorrect classification attempts as new data to train and improve the classifier. Such inputs are provided in the hope that the generator will become better at fooling the classifier, and the classifier will become better at classifying correctly, creating a feedback loop in which the performance of both improves. The process of training these subsystems often relies on the subsystems being constructed as differentiable networks that can be trained through a process of backpropagation.

[0048] However, when complex, non-discriminable processes are part of the music generator, backpropagation alone may not be suitable for training, and slower, more complex reinforcement learning approaches may be required. Unlike GANs, reinforcement learning involves machine learning, the system involves an "agent" and an "environment", and training is not done through a specific technique, although the design of the environment can have a significant impact on the system's outcome. In some cases, reinforcement learning is slow, costly, and produces interesting results, but the results are not necessarily of high quality. To overcome these shortcomings, the inventors of the present application have recognized that improvements can be made by implementing a system that utilizes a reinforcement learning approach in a new music-specific environment designed to enhance the musicality of the system output through the use of human-in-the-loop processes and high-quality audio processing.

[0049] 3 is a block diagram illustrating an example music generator system configured for training music generation and classification of music content, according to some embodiments. In an example embodiment, the system 300 includes a music generation subsystem 310 and a music classification subsystem 320. In a particular embodiment, the music generation subsystem 310 is located on a particular user device (such as a personal computer, laptop, mobile device, or portable device), while the music classification subsystem is located remotely (e.g., in the cloud). Other embodiments of locations for the subsystems are also contemplated. In a particular embodiment, the music generation subsystem 310 includes the music generator module 160 and the parameter generation module 312. The system 300 also includes an enhanced input generation module 330, which may be located separately from the music generation subsystem 310 and the music classification subsystem 320 or may be located within one of these subsystems.

[0050] In various embodiments, the music generator module 160 accesses the stored audio files 302 to generate the output musical content 316. In particular embodiments, the music generator module 160 generates the output musical content 316 from digital musical fragments in the stored audio files 302. In various embodiments, the digital musical fragments are pre-recorded musical fragments. Digital musical fragments may include, for example, samples or portions of pre-recorded digital music (such as partial samples of a song or composition). Digital musical fragments may also include samples of pre-recorded instruments or samples of other pre-recorded music.

[0051] In certain embodiments, the output musical content 316 is a music stream or composition of a certain length (such as 30 seconds or 60 seconds). For example, the output musical content 316 can be a composition of a certain length generated from multiple audio files, portions of multiple audio files, or multiple musical fragments. In various embodiments, the output musical content 316 includes a series of different output musical content. For example, the series of output musical content can include various compositions of a certain length generated from any combination of musical fragments.

[0052] In various embodiments, the set of output musical content in output musical content 316 provides an environment for reinforcement learning of system 300. In particular embodiments, providing the environment includes populating the set of output musical content with any number of possible compositions of a particular length that can be generated from the set of musical fragments obtained by music generator module 160 from stored audio files. In some embodiments, the set of output musical content may be populated with all possible compositions of a particular length that can be generated from the set of musical fragments. Because a large number of musical compositions are possible from the set of musical fragments, in many embodiments, a portion (possibly only a portion) of the compositions generated by music generator module 160 are what a layperson would consider musical.

[0053] In various embodiments, the music generator module 160 generates output musical content 316 from stored audio files 302 (e.g., digital musical fragments) according to one or more music generation parameters 314. As used herein, music generation parameters 314 may include parameters that control the selection and processing of musical fragments or other samples that are combined to generate the output musical content 316. In various embodiments, the music generation parameters 314 define a policy for generating music. For example, the policy may describe how to combine digital musical fragments in sequence and in simultaneous layers, and modify them with music production audio effects (such as via the music controls described herein) including reverb, delay, noise gate, equalization, gain control, and band filters. Such processes and effects are commonly used in modern music production. In some contemplated embodiments, if a set of digital musical fragments is curated by a person with some musical knowledge (e.g., via the music controls), random application of the process in the policy may result in a baseline output of a musical content composition that most people would consider to be music (and is quite likely to be good music).

[0054] In various embodiments, the music generation parameters 314 include parameters 312 previously generated (e.g., by the parameter generation module 312) or parameters created by a user (e.g., using the music control inputs 510 shown in FIG. 5). In certain embodiments, the music generation parameters 314 are generated by the parameter generation module 312. In the illustrated embodiment, the reinforcement learning of the parameter generation module 112 and the music generation subsystem 310 is provided by reinforcement inputs 332 received from a reinforcement input generation module 330. In certain embodiments, the parameter generation module 312 implements an algorithm (e.g., a trained machine learning algorithm) that generates the music generation parameters 314. For example, the parameter generation module 312 may implement an algorithm that adjusts the music generation parameters 314 based on inputs from other systems. Thus, the algorithm is tunable (e.g., trainable) based on the reinforcement inputs 332, and the music generation parameters 314 may be considered tunable parameters in the music generation subsystem 310.

[0055] In some embodiments, the input for adjusting the algorithm includes an output from the music classification subsystem 320. For example, the enrichment input 332 may be based on a prediction of a music source by the music classification subsystem 320 for the output music content generated by the music generator module 160. FIG. 4 is a block diagram illustrating an example music classification subsystem 320 according to some embodiments. In the illustrated embodiment, the music classification subsystem 320 includes a classifier module 400. The classifier module 400 may implement a machine learning algorithm to provide classification using one or more classifiers, for example. In certain embodiments, the classifier module 400 is a trained classifier module that uses one or more trained classifiers 402 to provide a classification of the music content. The training of the classifier module 400 is described in more detail in the embodiment of FIG. 6, described below.

[0056] In certain embodiments, the classifier module 400 receives as input the output musical content 316 (which may include a series of different musical output content). From this input, the classifier module 400 may implement one or more trained classifiers 402 to determine a musical source prediction 410 for the output musical content 316. The musical source prediction 410 may include a prediction as to whether the output musical content 316 has been generated by a human composer or by the music generator module 160 (e.g., whether the musical source is a human composer or a music generator module). In some embodiments, the prediction of the musical source prediction 410 includes a probability that the output musical content 316 was either generated by a human composer or by the music generator module 160. For example, the musical source prediction 410 may be a percentage probability that the output musical content 316 was generated by the music generator module 160. The musical source prediction 410 may then be provided to the enhanced input generation module 330.

[0057] Returning to FIG. 3 , in various embodiments, the music source predictions 410 for the output musical content 316 may not be distinguishable by the parameter generation module 312 and the music generator subsystem 310. Thus, in certain embodiments, the enhanced inputs 332 for adjustment of the algorithms in the parameter generation module 312 may be determined by the enhanced input generation module 330 from the music source predictions 410. In some embodiments, the enhanced input generation module 330 adjusts the music source predictions 410 based on a reward-based system. In such embodiments, the enhanced input generation module 330 may reward the music generation subsystem 310 based on how well the output musical content “fooled” the music classification subsystem 320 into believing that the output musical content was generated by a human composer. For example, the enhanced input generation module 330 may provide a higher reward for a higher probability of the music source predictions 410 that the output musical content 316 was generated by a human composer because the higher the prediction probability, the more likely the music classification subsystem 320 will be fooled by the music generation subsystem 310 into believing that the output musical content was generated by a human composer.

[0058] In some embodiments, the reinforcement input 332 may be a binary input. The binary input may be provided as an alternative to discriminative inputs for adjustment of the algorithms of the parameter generation module 312 to generate the music generation parameters 314. In various embodiments, the binary input includes a binary decision of whether the musical content meets a listening threshold. The listening threshold may be met, for example, if the musical content is determined to be "good" (e.g., musical) versus "bad" (e.g., non-musical). In some embodiments, the binary input decision may be performed by a computer algorithm (e.g., a trained machine learning algorithm). In other embodiments, the binary input may be provided as a human input 318 to the output musical content 316, as described herein.

[0059] In various embodiments, the generation of music generation parameters 314 (e.g., the generation of music generation policies) is human readable and human modifiable. Making the generation parameters 314 human readable or human modifiable may reduce the time and processor cost of producing music. Additionally, facilitating easier manual optimization by a user and enhancing the training of the algorithms in the parameter generation module 312 to generate the music generation parameters 314 may increase the likelihood of generating compositions that are musical in nature. In certain embodiments, making the generation of music generation parameters 314 human readable or human modifiable includes the addition of user-facing music controls in the music generation process.

[0060] 5 is a block diagram illustrating an example music control implementation in the music generation subsystem 310, according to some embodiments. In the illustrated embodiment, the music controls 500 are implemented by the parameter generation module 312 and provided to a user interface. Music control input 510 from a user is then received by the parameter generation module 312 from the user interface. As described above, the music generation parameters 314 may define policies used to control the process of music generation by the music generator module 160. The music controls 500 may allow user input to the music production expressed according to music creation methods familiar to music producers, music artists, or other music-savvy users. Thus, a user may be able to create music content by providing input via the music controls 500 that beneficially impacts the generated output music content 316.

[0061] In some embodiments, the process of music generation between the parameter generation module 312 and the music generator module 160 is distinguishable within the music generation subsystem 310. For example, the music control inputs 510 are distinguishable from the generation of the music generation parameters 314 within the music generation subsystem 310. Thus, the initial settings for the music generation parameters 314 may be found via backpropagation between the music control 510 and the parameter generation module 312. The generation of the initial settings may provide a non-random starting point for the refinement (e.g., adjustment) of the music generation parameters 314 based on the reinforcement inputs 332. For example, the refinement of the music generation parameters 314 based on the reinforcement inputs 332 may start from a more musical "known" position rather than a random unknown musical position.

[0062] The music control 500 may also be implemented to generate music generation parameters 314 based on musical events (such as playing generated musical content), such that a large range (potentially millions or billions) of unique minutes of output musical content may be generated by the music generator module 160 that is continuously varied and not overly repetitive.

[0063] As described above, the music controls 500 may be implemented to define music generation policies (via the music generation parameters 314) designed through the use of common music controls for a musically knowledgeable person, who may then be able to quickly manually adjust the music generation policies to reduce the time and expense of training the music generation subsystem 310. In various embodiments, the music controls 500 may be provided to provide control over the selection and processing of music snippets by the music generator module 160 in generating the output musical content 316. In certain embodiments, the music controls 500 provide control over one or more audio properties of the output musical content 316. Examples of the music controls 500 include, but are not limited to, gain, high pass cutoff, low pass cutoff, reverb send level, reverb length, delay level, delay time, bass, beat, pad, top, melody, backing melody, chords, effects, and combinations thereof. The music controls 500 may be applied to either an audio snippet (e.g., a digital music snippet), a music track (e.g., the output musical content 316), or combinations thereof.

[0064] As described above, application of the music control inputs 510 to the output musical content 316 may modify the output musical content 316 in response to the musical preferences of the user. In various embodiments, the modified output musical content 316 generated by the music generator module 160 is provided as a reinforcement input 332 to the parameter generation module 312. In such embodiments, the modified output musical content 316 may be input to the parameter generation module 312 as a vector or set of vectors. Providing the modified musical content as a vector may facilitate the selection (pair generation) of raw audio content (e.g., digital musical fragments) to play with the output musical content 316 after reinforcement learning by the parameter generation module 312. For example, the music generation parameters determined by the parameter generation module 312 may facilitate searching a library of audio files (e.g., stored audio files 302) for digital musical fragments having characteristics that most closely resemble those described by the policy (described by the music generation parameters 314) and using those selected fragments to generate the output musical content 316. Thus, the music generator module 160 is implemented in a role more similar to that of a music arranger (eg, selecting and arranging musical fragments) than that of a music composer (eg, creating new music from individual notes).

[0065] As an example, a 30 second audio sample of the output music content 316 to which a user has applied music controls 500 to modify the output music content may be used as a vector-formatted enhancement input 332 to the parameter generation module 312. Non-limiting examples of vector content include, for example, {0.1, 0.2, 0.1, 0.4...0.7} (a 64 element vector describing a reduced feature representation of the audio to be used in the bass track for the next beat of music);{0.6} audio signal gain used to play the selected audio;{20.0} a 20 Hz high pass filter should be applied;{16000.0} a 16000 Hz low pass filter should be applied;{0.1} a reverb auxiliary send level of 0.1;{2.0} a 2 beat decay time;{0.0} no delay should be used;{1.0} a 1 beat delay length.

[0066] Vectors such as those described above may be generated via machine learning algorithms or other algorithms that generate floating point vectors through mathematical transformations. For example, the vectors may be generated via feature reduction algorithms (such as variational autoencoders) that normalize the space of reduced features or common audio content analysis vectors. In various embodiments, the history of decisions made on policies (e.g., history of adjustments based on changed content) and selected digital music fragment content may be stored in a ledger or other library. Storing this information may allow the determination of percentages and specific selections of files from a library of content for different artists (e.g., stored audio files 302).

[0067] Returning to Figure 3, as discussed above, human input 318 to the output musical content 316 may be provided by a user (such as an artist or listener). In certain embodiments, implementations of human input 318 include input based on the user's interpretation of the output musical content 316. For example, a human user may provide input as to whether music sounds good or bad to the user. Input may be provided via a user interface of the device (such as a like / dislike selection).

[0068] Other embodiments of human input implementation in the reinforcement input 332 are also contemplated. For example, a classifier module (e.g., classifier module 400 shown in FIG. 4 or another classifier module implemented in the system) may be trained to classify music that the listener likes and dislikes based on a selection made by the listener (e.g., via a like / dislike selection on a user interface). If music that the listener likes is generated by the music generator module 160, the like by the listener may be provided to the reinforcement input generation module 330 and used as reinforcement input 332 (e.g., providing positive feedback) to reinforce the music generation parameters 314 generated by the parameter generation module 312 to generate music that the user likes.

[0069] FIG. 6 is a block diagram illustrating an example training system 600 for the music classification subsystem 320, according to some embodiments. In an example training embodiment, the classifier module 400 receives either the output musical content 316 or the generated musical content 612. Either type of musical content may be received in a particular length (e.g., 30-second or 60-second samples). The generated musical content 612 may include, for example, professionally produced musical content or other musical content produced by a musical expert. In a training embodiment, the output musical content 316 or the generated musical content 612 may be input to the classifier module 400 along with a label that identifies a parameter of the musical content. For example, the label may include an identification of whether the music was generated by the music generator module 160 or whether the music was produced by an expert. Other labels are also contemplated.

[0070] In certain embodiments, the classifier module 400 implements one or more classifiers to determine a music source prediction 614 for the input musical content. For example, the classifier module 400 may determine a prediction that the source of the musical content input to the module is either the music generator module 160 or a human (professional) producer. In various embodiments, the music source prediction 614 includes a probability that the input musical content was generated by either the music generator module 160 or a human composer. For example, the music source prediction 614 may be a percentage probability that the input musical content was generated by the music generator module 160. In some embodiments, the music source prediction 410 (e.g., the music source prediction of the output musical content 316 input to the classifier module 400) is output to the enhanced input generation module 330, as shown in FIG. 4.

[0071] In the illustrated embodiment for training, the training input of the classifier 616, which includes a music source prediction 614 for any type of music content input, is used as the basis for an accuracy measure used for backpropagation in the classifier module 400. Communicating the music source prediction 614 to the classifier module 400 includes providing both a label identifying the source of the music content input and an output prediction (music source prediction 614) for the source of the music content input. The classifier module 400 may then adjust the classifier within the module according to the difference between the label and the music source prediction 614. After training, the classifier module 400 may output a trained classifier 618. The trained classifier 618 may be provided to a device on which the music generation system is installed (e.g., a personal or portable device with a music generation system application installed on the device). In some embodiments, the trained classifier is provided to the device as part of the installation of the music generation system on the device.

[0072] Additional embodiments are contemplated in which the classifier module 400 is trained to perform other classifications for musical content. For example, some embodiments are contemplated in which the classifier module 400 is trained to classify music created by a particular producer. In such embodiments, identifying a particular producer of musical content may enable the generation of a policy that generates musical content that is more likely to be classified (by the parameter generation module 312) as being from the particular producer.

[0073] Returning to FIG. 3, in various embodiments, the music generation subsystem 310 and the music classification subsystem 320 are implemented to generate music that changes based on feedback from a listener (e.g., a user). However, in some embodiments, for a music generation subsystem 310 installed on a certain type of device (e.g., a mobile device), implementing classification based on human input (e.g., likes / dislikes of music) on a large scale may not be desirable or possible due to the storage limitations of such devices and the library of information that needs to be stored for training / learning by the classifier. Thus, in certain embodiments, the music generation subsystem 310 is installed on a device with initial training parameters (e.g., an initially trained classifier), and reinforcement learning is implemented on the device to improve music generation by the system. In such embodiments, input regarding likes / dislikes (or other similar choices) from one or more users may be used to train a classifier module (e.g., a classifier module located in the cloud) located on a remote computing device connected to the device on which the music generation system is installed.

[0074] The classifier module located in the cloud may be trained based on input (feedback) from the user during operation of the music generation system so that it can better predict music content that the user may like or dislike. After some additional training of the classifier module in the cloud, a vector for the user (or a set of vectors for multiple users) may be generated. The vector may be generated to represent the music taste of a particular user. The vector may be, for example, a 64-element vector representing the music taste of a particular user. The vector may then be provided to the device of a particular user for implementation as a reinforcement input (e.g., reinforcement input 332) of the music generation subsystem 310 installed on the device. Thus, there is no need to send classifier updates to the device, and there is no need to store multiple libraries on the device. In addition, a single initial comprehensive training process is implemented for the music generation subsystem 310 before it is installed on a particular device. The music generation subsystem 310 installed on the device may then be updated by reinforcement learning received as input from the cloud to avoid the need to fully train each music generation subsystem on each device. This has the advantage that millions of unique music generation subsystems can be distributed across mobile devices and a single classifier module is trained for all subsystems, reducing training time and costs and allowing for a centralized training environment while still obtaining personalized results for each music generator.

[0075] Example Music Generation Overview Based on User-Defined Control Elements Various techniques are disclosed for generating musical content through interaction between a composition module and a performance module. Generally speaking, the composition subsystem selects and arranges audio files to compose the musical content, while the performance subsystem may filter, add effects, mix, master, etc. to the selected audio files to generate the output musical content that is actually played.

[0076] In some disclosed embodiments, both the composition subsystem and the performance subsystem operate based on user input to music control elements (which may be displayed via one or more user interfaces). As an example, an artist may provide user input to the composition subsystem (which may be implemented, for example, in a server system) and an end user may provide user input to the performance subsystem (which may be implemented, for example, in a user device such as a mobile phone). In other embodiments, the same user may provide input to both modules.

[0077] In some embodiments, the system may allow users to define their own musical controls and may implement machine learning or other algorithms to adjust musical parameters based on adjustments to the user-defined controls, which may be more abstract. For example, a user may An artist may define a "harmony", "happiness" or "mood" control, and the composition subsystem may determine the type of low level parameters to adjust and the amount of adjustment to achieve the artist's vision for this control element. In some embodiments, all or a subset of the user-defined musical controls may also be available to the performance subsystem, for example to allow the user to implement artist-defined controls. In some embodiments, the two subsystems may communicate to provide appropriate performance parameters based on the user-defined musical controls. The disclosed technology is advantageous because it may enable artists to provide customized musical control elements to end users. The disclosed technology also allows end users to customize the real-time performance of generative music while allowing centralized control of certain aspects of the automatic music composition.

[0078] The subsets of musical controls available to the composition subsystem and performance system may or may not overlap, and users of the performance module may specify their own abstract musical controls which, in some embodiments, may be used in conjunction with the abstract musical controls defined by the composition module.

[0079] In general, the disclosed techniques may facilitate the creation of more human-like musical compositions by music generation systems. The inventors recognize that by providing abstract control over music generation, music generation systems can enable musical content to be generated according to a user's stylistic, structural or artistic preferences.

[0080] FIG. 7 is a block diagram illustrating an example music production system 700 including a composition subsystem 710 and a performance subsystem 720 according to some embodiments.

[0081] In the illustrated embodiment, the composition subsystem 710 generates composition control output information based on user input. In some embodiments, the composition control output information identifies selected audio files to be combined and indicates relative timing for inclusion in the output musical content. The composition subsystem 710 may determine how to arrange sections of the musical content and when to transition between sections, as described in more detail below.

[0082] In the illustrated embodiment, the composition subsystem 710 also generates control element setting signals for the performance subsystem 720. In some embodiments, these signals configure the performance subsystem 720 to implement user input to the abstract musical controls in an intended manner. For example, the settings may specify how to adjust various performance actions (filtering, adding effects, mixing, mastering, etc.) based on adjustments to user-defined musical controls.

[0083] As shown, the composition subsystem 710 receives user input. This user input may specify, for example, custom musical control elements. The user input may also provide label information for previous compositions by a particular artist. For example, for the abstract control element "harmony," the artist may label previous compositions with a rating from 0 to N based on the artist's perception of harmony, such that a machine learning model can be trained to adjust the composition parameters to reflect different levels of the harmony control element. Note that the musical control elements may have various encodings, such as accepting user input specifying a specific value, increasing or decreasing, binary input such as good / bad, etc.

[0084] In the illustrated embodiment, the performance subsystem 720 is configured to generate output musical content based on composition control and control element setting information from the composition subsystem 710. As shown, the performance subsystem 720 may also receive user input for, for example, potentially all of a subset of the musical control elements implemented by the composition subsystem 710 (in some embodiments, the performance subsystem 720 may also implement additional musical control elements). In some embodiments, the set of control elements available for user input on the performance subsystem 720 is controlled by user input to the composition subsystem 710.

[0085] In various embodiments, composition control at multiple levels may advantageously provide musical output while allowing user customization for various musical styles. It should be noted that various functions performed by composition subsystem 710 in disclosed embodiments may be moved to performance subsystem 720 in other embodiments, and vice versa. The disclosed separation of functions may provide good results in certain contexts, but is not intended to limit the scope of the present disclosure. Thus, the composition control information of FIG. 7 may, in various embodiments, include various types of information ranging from raw selected audio files to at least partially mixed output content.

[0086] In some embodiments, the composition subsystem 710 and the performance subsystem 720 are implemented on different computer systems and communicate over a network. As described in more detail below, communication between the interfaces may use custom string manipulation and storage techniques to enable a real-time monitoring and control API (e.g., using the Open Sound Control (OSC) protocol). In other embodiments, the subsystems are implemented on the same computer system.

[0087] Additionally, the Performance Subsystem 720 may send control element setting information to the Composition Subsystem 710 (not explicitly shown), for example to allow end user adjustments to affect composition operations such as selecting audio files to be combined. Generally, the disclosed techniques allow user-defined control elements defined on one device / interface to be made available to users on other devices / interfaces, potentially controlling parameter adjustments on other subsystems or devices.

[0088] Detailed Example of a Music Generator with Composition and Performance Subsystems 8 is a block diagram illustrating a more detailed example of a music generator system according to some embodiments. In the illustrated embodiment, the composition subsystem 710 includes a module for customized music control 810A, a composition module 820, and a machine learning module 830. In the illustrated embodiment, the performance subsystem 720 includes a module for customized music control 810B and a performance module 850. Note that more detailed examples of the composition module 820 and the performance module 850 included in different subsystems are described below with reference to FIG. 6.

[0089] Module 810A may communicate with machine learning module 830 based on user input to determine how to implement custom musical controls using low-level parameters. As shown, machine learning module 830 may access previous compositions 840 (whether labeled as a whole or with different part labels by the user according to one or more custom musical controls) to determine how to implement abstract controls using low-level composition parameters (which may include parameters used by both subsystem 710 and subsystem 720). Training may use various suitable machine learning models, e.g., deep neural networks, to determine the purpose of user-specified control elements. As shown, module 810A communicates control element setting information to module 810B, which communicates with performance module 850 to adjust performance parameters based on user input. Similarly, composition module 820 adjusts composition parameters based on signals from module 810A.

[0090] In some embodiments, an artist may explicitly specify the relationship between custom musical controls and lower level parameters. For example, for a "harmony" control, an artist may specify different parameters to create tension (adding layers, riser effects, cutting melodies, etc.) for compositions with stronger harmonic movement than for compositions with weaker harmonic movement. In some embodiments, explicit controls may be used alone or in combination with machine learning analysis to generate the lower level parameters.

[0091] The following description describes various parameters that may be indirectly controlled via custom user interface elements, for example, based on machine learning or on a user indicating a relationship between a control and a parameter, although in some embodiments the music generator system may expose both abstract and lower level controls.

[0092] In some embodiments, the generation of continuous music is achieved through a combination of algorithmically defined musical sections and transitions. Sections may be defined by their musical functions, such as building, maintaining and reducing tension. In some embodiments, sections of tension are built up slowly by repeating musical content, having louder or more layered mixes or playing musical rhythms or pitches that contrast with each other. In contrast, tension may be established quickly by defying the listener's expectations. For example, tension may be established quickly by ending a repeated phrase mid-repetition, suddenly changing the key, introducing a new texture not previously present in the music, suddenly increasing the volume, etc. If the same techniques for creating tension are repeated too often, the listener will expect it and their effectiveness in creating tension will be greatly diminished.

[0093] In some embodiments, tension is released slowly by reducing layers, by aligning the rhythm or pitch of previously contrasting notes, or by ceasing the repetition of a musical phrase. Tension may be released quickly by returning to a previously established expectation, by resuming repetition after a momentary interruption in the previous phrase, by resolving a harmonic dissonance, or by providing a climactic point (e.g., a loud cymbal sound) marking the end of a section.

[0094] The creation of effective continuous music can result from juxtaposing the build-up and release of tension in a variety of ways.

[0095] In some embodiments, the composition subsystem 710 exposes certain controls with default or option values. The controls may include, for example, gain mixes, audio effect chains, arranging music over time, layering musical phrases together, etc. Unlike standard composition tools where only the compositional process added by the user has a musical effect, machine learning algorithms may be used to influence the compositional process if option values ​​are not utilized.

[0096] Examples of parameters controlled by the composition subsystem 710 may include, for example, maximum repetitions of a single loopable audio, use of synthetic noise, and default EQ levels. In some embodiments, the maximum repetitions may be set as a hard limit or left unlimited infinity, but other composition processes are likely to result in departures from repetition without it. The "synthetic noise" control may be used as a binary switch to enable or disable synthetic noise at certain points in the composition, without controlling when it is applied. Timing may be influenced by other controls and machine learning algorithms. The default EQ levels allow for reference levels of frequency, gain, and bandwidth for eight bands, and may be adjusted by other composition processes, including those driven by machine learning algorithms. The composition subsystem 710 may define what types of audio files may be used in different time structures of a composition, without defining exactly when the time structures occur within the composition.

[0097] In general, various categories or "groups" of audio files may be enabled or disabled within different musical arrangement structures. In some embodiments, enabled groups mean that they may be utilized by the subsystem, but it is not guaranteed. Below are some example groups: Beat: defines the rhythm bed, usually a drum kit - unpitched Bass: Bass line - with tune Harmony: Polyphonic midrange tonality for fleshing out harmonies Pad: background soundscape or drone - pitched Top: additional rhythmic content, often percussion, usually syncopated - pitched Melody: Mainly monophonic melody lines - with pitch · Pitched FX: Additional sound effects with strong tonality - pitched · Unpitched FX: Non-tonal additional sound effects (e.g. noise) - unpitched For each of these groups there may be adjustable controls to set these quality benchmarks. ·volume Reverb Amount Delayed Punch Chance Delayed wetness

[0098] In some embodiments, the following sections may be used: build-up, slow increase in tension, sustain, sustain tension level, drop: a climax point followed by low tension, breakdown, slow decrease in tension. Also, the following transitions may be used: add layer, sudden increase in tension, remove layer, sudden decrease in tension, riser effect, quick increase in tension, cut melody, quick increase in tension. The composition subsystem 710 may learn how to apply various sections and transitions based on the compositional preferences of a particular artist of a group of artists, for example.

[0099] In some embodiments, probability-based rules guide the sequence of transitions to encourage variations to slowly build and release tension, and transitions are used to create unexpected turns on these gradual trends. Maximum and minimum frequencies of transitions may be scaled to the length of the section, since overuse of transitions may reduce the overall tension movement of the section it occupies.

[0100] The table below shows examples of transition probabilities for different sections. In some embodiments, a user of the composition subsystem 710 modifies such a table to change the staged musical structure without predetermined absolute timing of the sections.

[0101] [Table 1]

[0102] In this example, the transition from the breakdown section has a 40% chance of moving to a build-up section, a 30% chance of moving to a drop section, and a 30% chance of moving to a sustain section.

[0103] The composition subsystem further breaks down sections to create a hierarchy of sections and subsections. Examples of subsections include: Play only the first beat of the phrase and use delay with feedback to create a new rhythmic phrase for each melody or harmony loop; Specifically removing the drum layer, -Mute harmony loops every 4th or 8th beat, Fade out the current baseline and gradually fade in the new baseline, Muting the rhythm layer for the last 8 beats of a subsection, Low pass filters are tapered from 20,000 to 500Hz across all layers, Examples include:

[0104] Each subsection generates transitions at the beginning, end, and possibly in the middle, so that certain subsections can only occur in certain sections or at certain points in a section. Examples of subsection rules include: The subsection that removes the beat does not occur at the beginning of the drop section, Subsections that change the bass line do not occur during build-up sections. Subsections that do not remove layers may not occur in sections with a total of less than 4 layers played, unless they are breakdown sections. Examples include:

[0105] The number and selection of layers used in each section can be determined to create musical tension. Example layer combinations include: -Intro: Bass and drums throughout Build-up: A gradual fade-in of the bass, drums, percussion, melody, and pads. Drop: Bass, drums, percussion, pads and melody throughout Breakdown: Bass and drums overall, gradually fading out from all other layers · Keep: The same layer used at the end of the previous section Examples include:

[0106] In addition to subsections, musical deviations can be used to create variations and micro-tension that are particularly effective in loop-based repetitive music. In some embodiments, for example: Delayed Punch: The last or first beat of a drum is played again with a delay of 1 or 1 / 2 beat without feedback. High pass filter increases from 0 to 50Hz over 8 beats and instantly returns to 0Hz to give a strong beat pulse; Applying heavy reverb to a single loop, Deviation techniques are used.

[0107] In some embodiments, subsections and deviations may be enabled or disabled in the composition subsystem, and in some cases may be parameterized, so generally speaking the composition subsystem may provide the user (e.g., artist) with more control over composition parameters than the performance subsystem.

[0108] In some embodiments, the performance subsystem 720 is configured to operate based on composition control information (e.g., a composition script) from the composition subsystem to generate the output musical content.

[0109] In some embodiments, the composition subsystem 710 makes decisions by associating feedback data mediated by the performance subsystem 720 with musical controls in the script. In addition to explicit thumb up / down feedback, for example, the act of playing by the performance control subsystem passively suggests some degree of preference. Retrospective analysis of the compositions consumed by the user and the feedback provided helps enable the machine learning system to adjust both the loops and compositional techniques represented on the user device. The performance control subsystem may use the feedback analysis to find similar styles of audio content and attributes from the core set.

[0110] Just as an artist may use rules, heuristics, and training data to shape an artist pack's style, the performance control subsystem may enable the end user to achieve the same with a set of loops that the performance subsystem provides. In this scenario, the performance subsystem may provide a large (and growing) set of loops for playback. Users may have inputs including thumbs up and down that may help them learn what kinds of loops and mixing techniques the performance control subsystem uses when creating a soundscape for them. In addition, UX controls may be provided to the user to allow them more control over how the performance control subsystem mixes the content. This may include mixing levels, section lengths, musical complexity, use of mixing techniques such as builds and drops, etc. These tweaks allow the listener to essentially create their own "style," which the performance control subsystem uses when creating the soundscape. Users may also use their own custom control elements or custom control elements defined using module 810A for higher level control over various compositional aspects described below.

[0111] When the user selects thumbs up / down, various elements of the current composition may be taken into consideration. These include: The current mixing levels, including gain for the various layers, Attributes related to loops near the input, - Input the time, weather, location, etc. Explicit input via the additional UX controls mentioned above, may include.

[0112] These qualities may be captured by the performance subsystem and associated with positive or negative feedback. Over time, this data may help guide the performance control subsystem to train a machine learning module (not explicitly shown in FIG. 8 ) with positive and negative reinforcement, allowing the performance control subsystem to adjust parameters of the music playback (e.g., parameters for the rules engine). The models used to adjust these parameters need not be left to a single user. For example, machine learning may be used to apply changes to the composition and playback based on aggregate feedback for a music app. That is, if a particular user presses thumbs down and the pattern identified at this point matches the pattern of many other users who have thumbs down, the system may adjust the music accordingly in the same way it did for the other users. This allows the performance control subsystem to make personalized changes for the user, but those changes are made based on aggregate data collected across many users.

[0113] The correlation between feedback and musical properties can be adjusted by environmental measurements. In this way, a more nuanced model of the user's preferences is built. For example, rather than learning "this listener likes these types of beats," the performance control subsystem can learn "this listener likes these types of beats at this time of day or when it is raining."

[0114] The success of the end user's personalization may ultimately be measured by positive events associated with the playback. This includes events such as "thumbs up," playback time, etc. In addition to these events, other parameters may be associated with the success of the personalization. For example, a restaurant playing the performance control subsystem may associate an increase in hourly sales as an indicator of success. The performance control subsystem may monitor the POS and associated positive changes in hourly sales as a positive reflection of the composition. In this way, the performance control subsystem may train itself implicitly without requiring explicit input. This training model may be applicable to any set of underlying loops, since the training is just as much about the musical composition as the loops are used.

[0115] Exemplary Communication Techniques Between Composition and Performance Subsystems In some embodiments, communication between subsystems may be real-time monitoring and control, for example using an API over OSC. The disclosed techniques described in detail below may facilitate real-time performance in such contexts. Note that in distributed embodiments, the amount of data sent over the network may be significant when monitoring all parameters and RMS levels, but is still small compared to, for example, a full audio stream. However, these messages may be parsed individually, and the associated string manipulations may be very expensive. Creation of such strings sent from real-time threads may be done carefully to reduce memory allocations and blocking calls.

[0116] OSC uses plain text strings delimited by forward slashes (similar to URLs) to address endpoints, e.g., " / master / compressor / threshold" or " / performance_module / effect_module / parameter_C". These strings may be used, for example, to route various information from the composition module to the appropriate endpoints in the performance module, and in the other direction, e.g., for monitoring purposes. Thus, operations on delimited strings are common operations in some embodiments (notably "split" and "join", which convert between delimited strings and series of strings, plus functions similar to prepend and append to a given delimited string).

[0117] Traditional string manipulation libraries that perform split / join etc. may not be designed with real-time operations in mind. In particular, their storage model typically creates copies of component tokens when splitting a delimited string. To reduce allocations, in some embodiments, the disclosed system uses a specialized DelimitedString container. The DelimitedString class may utilize the actual delimited string as a backing store and expose a list-like API on top of it (essentially a series of pointer offsets to the full string marking the tokens). The backing string may be created with spare capacity on either side to allow for easy prepend / append of additional tokens without requiring costly reallocation.

[0118] The DelimitedString class may be designed to allow efficient "trimming" on both ends. In these embodiments, the recipient of an OSC message (e.g., the playing subsystem 720) treats the OSC address as a hierarchical routing mechanism, typically passing the message through a series of delegates that lop off the first token in the address at each stage. The DelimitedString may manage this efficiently by simply moving a "HEAD" pointer that references an offset into the full string backing store. It may also manage a state stack to remember the state, iteratively lop off parts of the address, and restore the state. This may allow the same DelimitedString to be used again.

[0119] Example User and Enterprise GUI 9A-9B are block diagrams illustrating graphical user interfaces according to some embodiments. In the illustrated embodiment, FIG. 9A includes a GUI displayed by a user application 910, and FIG. 9B includes a GUI displayed by an enterprise application 930. In some embodiments, the GUIs displayed in FIGS. 9A and 9B are not generated by an application, but rather by a website. In various embodiments, any of a variety of suitable elements may be displayed, including one or more of the following elements: dials, buttons, knobs (e.g., to control volume, energy, etc.), display boxes (e.g., to provide updated information to the user), and the like.

[0120] In FIG. 9A , a user application 910 displays a GUI that includes a section 912 for selecting one or more artist packs. Additionally or alternatively, in some embodiments, the packs 914 may include theme packs or packs for a particular occasion (e.g., weddings, birthday parties, graduations, etc.). In some embodiments, the number of packs shown in section 912 is greater than the number that can be displayed in section 912 at one time. Thus, in some embodiments, a user scrolls up or down in section 912 to view one or more packs 914. In some embodiments, a user may select an artist pack 914 upon which the output music content the user wishes to listen to is based. In some embodiments, the artist packs may be purchased and / or downloaded, for example.

[0121] In the illustrated embodiment, the selection element 916 allows the user to adjust one or more music attributes (e.g., energy level). In some embodiments, the selection element 916 allows the user to add / remove / change one or more target music attributes. In various embodiments, the selection element 916 may render one or more UI control elements (e.g., music controls 500).

[0122] In the illustrated embodiment, the selection element 920 allows a user to have a device (e.g., a mobile device) listen to the environment to determine the target music attributes. In some embodiments, the device collects information about the environment using one or more sensors (e.g., a camera, microphone, thermometer, etc.) after the user selects the selection element 920. In some embodiments, the application 910 selects or suggests one or more artist packs based on the environmental information collected by the application when the user selects the element 920.

[0123] In the illustrated embodiment, the selection element 922 allows the user to combine multiple artist packs to generate a new rule set. In some embodiments, the new rule set is based on the user selecting one or more packs for the same artist. In other embodiments, the new rule set is based on the user selecting one or more packs for different artists. The user may indicate weights for the different rule sets, for example, such that a rule set with a higher weighting has more effect on the generated music than a rule set with a lower weighting. The music generator may combine rule sets in a number of different ways, for example, by switching between rules from different rule sets, averaging values ​​for rules from multiple different rule sets, etc.

[0124] In the illustrated embodiment, the selection element 924 allows a user to manually adjust the rules in one or more rule sets. For example, in some embodiments, a user may wish to adjust the generated musical content at a more granular level by adjusting one or more rules in the rule set used to generate the musical content. In some embodiments, this allows a user of the application 910 to become their own disc jockey (DJ) by adjusting the rule set used by the music generator to generate the output musical content using the controls displayed in the GUI of FIG. 9A. These embodiments may also allow for more fine-grained control of target musical attributes.

[0125] In Figure 9B, the enterprise application 930 displays a GUI that also includes an artist pack selection pane 912 with artist packs 914. In the illustrated embodiment, the enterprise GUI displayed by the application 930 also includes elements 916 for adjusting / adding / removing one or more music attributes. In some embodiments, the GUI displayed in Figure 9B is used at work or in a storefront to create a particular environment by creating music content (e.g., to optimize sales). In some embodiments, an employee uses the application 930 to select one or more artist packs that have previously been shown to increase sales (e.g., the metadata for a given rule set may indicate actual experimental results with the rule set in a real-world context).

[0126] In the illustrated embodiment, input hardware 940 sends information to the application or website displaying enterprise application 930. In some embodiments, input hardware 940 is one of a cash register, a heat sensor, a light sensor, a clock, a noise sensor, etc. In some embodiments, information sent from one or more of the above hardware devices is used to adjust target musical attributes and / or rule sets to generate output musical content for a particular environment. In the illustrated embodiment, selection element 938 allows a user of application 930 to select one or more hardware devices to receive environmental input.

[0127] In the illustrated embodiment, the display 934 displays environmental data to a user of the application 930 based on information from the input hardware 940. In the illustrated embodiment, the display 932 shows changes to the rule set based on the environmental data. In some embodiments, the display 932 makes changes made based on the environmental data visible to a user of the application 930.

[0128] 9A and 9B are for theme packs and / or event packs, i.e., in some embodiments, a user or business with a GUI displayed by applications 910 and 930 can select / adjust / modify rule sets to generate musical content for one or more events and / or themes.

[0129] Detailed example of a music generation system 10-12 provide details regarding specific embodiments of the music generator module 160. It should be noted that these specific examples are disclosed for purposes of illustration, and are not intended to limit the scope of the present disclosure. In these embodiments, the construction of music from loops is performed by a client system, such as a personal computer, a mobile device, a media device, or the like. As used in the description of FIGS. 10-12, "loop" may be synonymous with the term "audio file." Generally, loops are contained in audio files, as described herein. Loops may be divided into professionally curated loop packs, which may be called artist packs. Loops may be analyzed for musical properties, and the properties may be stored as loop metadata. Audio within the constructed track may be analyzed (e.g., in real time) and filtered to mix and master the output stream. Various feedback may be sent to the server, including explicit feedback from user interaction with sliders or buttons, and implicit feedback generated by sensors, e.g., based on volume changes, length of listening, environmental information, etc. In some embodiments, the control inputs have known effects (e.g., to directly or indirectly identify target musical attributes) and are used by the composition module.

[0130] The following description introduces various terminology that is used in connection with FIGS. 10-12. In some embodiments, a loop library is a master library of loops that may be stored by a server. Each loop may include audio data and metadata that describes the audio data. In some embodiments, a loop package is a subset of a loop library. A loop package may be a pack for a particular artist, a particular mood, a particular type of event, etc. A client device may download a loop pack for offline listening or may download a portion of a loop pack on demand, for example, for online listening.

[0131] In some embodiments, the generated stream is data that specifies the musical content that a user hears when using the music generator system, although the actual output audio signal may vary slightly for a given generated stream based, for example, on the capabilities of the audio output device.

[0132] In some embodiments, the composition module builds a composition from the loops available in the loop package. The composition module receives the loops, loop metadata and user input as parameters and may be executed by the client device. In some embodiments, the composition module outputs a performance script that is sent to the performance module and one or more machine learning engines. In some embodiments, the performance script summarizes which loops are played on each track of the generated stream and what effects are applied to the stream. The performance script may use beat-relative timing to represent when events occur. The performance script may also encode effect parameters (e.g., for effects such as reverb, delay, compression, equalization, etc.).

[0133] In some embodiments, the performance module receives a performance script as input and renders it into a generated stream. The performance module may generate multiple tracks specified by the performance script and mix the tracks into a stream (e.g., a stereo stream, although the stream may have various encodings in various embodiments, including surround encoding, object-based audio encoding, multi-channel stereo, etc.). In some embodiments, the performance module always generates the same output when provided with a particular performance script.

[0134] In some embodiments, the analytics module is a module executed by a server that receives the feedback information and configures the composition module (e.g., in real-time, periodically, based on administrator command, etc.) In some embodiments, the analytics module uses a combination of machine learning techniques to associate user feedback with performance scripts and loop library metadata.

[0135] FIG. 10 is a block diagram illustrating an example music generator system including an analysis module and a composition module, according to some embodiments. In some embodiments, the system of FIG. 10 is configured to generate a potentially infinite stream of music with direct user control over the mood and style of music. In the illustrated embodiment, the system includes an analysis module 1010, a composition module 1020, a performance module 1030, and audio output devices 1040. In some embodiments, the analysis module 1010 is implemented by a server, and the composition module 1020 and the performance module 1030 are implemented by one or more client devices. In other embodiments, the modules 1010, 1020, and 1030 may all be implemented on the client device or all on the server side.

[0136] In the illustrated embodiment, the analysis module 1010 stores one or more artist packs 1012 and implements a feature extraction module 1014 , a client simulator module 1016 and a deep neural network 1018 .

[0137] In some embodiments, the feature extraction module 1014 adds loops to the loop library after analyzing the looped audio (note that some loops may be received with metadata already generated and may not require analysis). For example, raw audio in formats such as wav, aiff, or FLAC may be analyzed for quantifiable musical characteristics such as instrument classification, pitch transcription, beat timing, tempo, file length, and audio amplitude in multiple frequency bins. The analysis module 1010 may store more abstract musical characteristics or mood descriptions for loops based on manual tagging, for example, by the artist or machine listening. For example, mood may be quantified using multiple discrete categories with a range of values ​​for each category for a given loop.

[0138] For example, consider Loop A which is analyzed to determine that it uses notes G2, Bb2 and D2, the first beat begins 6 milliseconds into the file, the tempo is 122 bpm, the file length is 6483 milliseconds, and the loop has normalized amplitude values ​​of 0.3, 0.5, 0.7, 0.3 and 0.2 across five frequency bins. An artist may classify the loop into the "Funk genre" with mood values ​​as follows:

[0139] [Table 2]

[0140] The analysis module 1010 stores this information in a database, and clients may download subsections of the information, for example, as loop packages. Although an artist pack 1012 is shown for illustrative purposes, the analysis module 1010 may provide various types of loop packages to the composition module 1020.

[0141] In the illustrated embodiment, the client simulator module 1016 analyzes various types of feedback and provides feedback information in a format supported by the deep neural network 1018. In the illustrated embodiment, the deep neural network 1018 receives as input a performance script generated by the composition module. In some embodiments, the deep neural network configures the composition module based on these inputs, for example to improve the correlation between the type of musical output generated and the desired feedback. For example, the deep neural network may periodically push updates to the client device implementing the composition module 1020. It is noted that the deep neural network 1018 is shown for illustrative purposes and may provide powerful machine learning capabilities in the disclosed embodiments, but is not intended to limit the scope of the present disclosure. In various embodiments, various types of machine learning techniques may be implemented alone or in various combinations to perform similar functions. It is noted that the machine learning module may be used to directly implement rule sets (e.g., placement rules or techniques) in some embodiments, or may be used to control modules that implement other types of rule sets, for example, using the deep neural network 1018 of the illustrated embodiment.

[0142] In some embodiments, the analysis module 1010 generates composition parameters for the composition module 1020 to improve the correlation between desired feedback and the use of certain parameters. For example, actual user feedback may be used to adjust composition parameters to attempt to, for example, reduce negative feedback.

[0143] As an example, consider a situation where the module 1010 discovers a correlation between negative feedback (e.g., explicit low rankings, low volume listening, short listening times, etc.) and composing with a large number of layers. In some embodiments, the module 1010 uses techniques such as backpropagation to determine that adjusting the probability parameters used to add more tracks will reduce the frequency of this problem. For example, the module 1010 may predict that reducing the probability parameters by 50% will reduce negative feedback by 8%, and decide to make the reduction and push the updated parameters to the composition module (note that the probability parameters are described in more detail below, but any of the various parameters for the statistical models may be adjusted similarly).

[0144] As another example, consider a situation where the module 1010 finds that negative feedback is correlated with a user setting the mood control to high tension. A correlation may also be found between loops with low tension tags and users requesting high tension. In this case, the module 1010 may increase the parameters to increase the probability of selecting loops with high tension tags when a user requests high tension music. Thus, the machine learning may be based on a variety of information including composition output, feedback information, user control input, etc.

[0145] In the illustrated embodiment, the composition module 1020 includes a section sequencer 1022, a section arranger 1024, a technique implementation module 1026, and a loop selection module 1028. In some embodiments, the composition module 1020 organizes and configures sections of a composition based on loop metadata and user control inputs (e.g., mood controls).

[0146] In some embodiments, the section sequencer 1022 sequences through different types of sections. In some embodiments, the section sequencer 1022 implements a finite state machine that sequentially outputs the next type of section during operation. For example, the composition module 1020 may be configured to use different types of sections, such as intros, build-ups, drops, breakdowns, and bridges, as described in more detail below with reference to FIG. 10. Furthermore, each section may include multiple subsections that define how the music changes throughout the section, including, for example, a transition-in subsection, a main content subsection, and a transition-out subsection.

[0147] In some embodiments, the section arranger 1024 composes the subsections according to arrangement rules. For example, one rule may specify to transition in by gradually adding tracks. Another rule may specify to transition in by gradually increasing gain on a set of tracks. Another rule may specify to chop a vocal loop to create a melody. In some embodiments, the probability of a loop in the loop library being added to a track is a function of the current position in the section or subsection, loops overlaid on other tracks over time, and user input parameters such as mood variables (which may be used to determine target attributes for the generated musical content). The functions may be adjusted by adjusting coefficients, for example, based on machine learning.

[0148] In some embodiments, the technique implementation module 1020 is configured to facilitate section arrangement by, for example, adding rules specified by an artist or determined by analyzing the compositional work of a particular artist. A "technique" may describe how a particular artist implements a compositional rule at a technique level. For example, for an arrangement rule that specifies to transition in by gradually adding tracks, one technique may indicate adding tracks in the following order: drums, bass, pads, then vocals, while another technique may indicate adding tracks in the following order: bass, pads, vocals, then drums. Similarly, for an arrangement rule that specifies to chop a vocal loop to create a melody, a technique may indicate chopping the vocals for every beat per second and repeating the chopped loop section twice before moving to the next chopped section.

[0149] In the illustrated embodiment, the loop selection module 1028 selects loops according to arrangement rules and techniques for inclusion in sections by the section arranger 1024. Once a section is completed, a corresponding performance script may be generated and sent to the performance module 1030. The performance module 1030 may receive portions of the performance script at various levels of granularity. This may include, for example, the entire performance script for a performance of a particular length, a performance script for each section, a performance script for each subsection, etc. In some embodiments, the arrangement rules, techniques, or loop selection are performed statistically, e.g., with different approaches using different percentages of time.

[0150] In the illustrated embodiment, the performance module 1030 includes a filter module 1031, an effects module 1032, a mix module 1033, a master module 1034, and an execute module 1035. In some embodiments, these modules process the performance script and generate music data in a format supported by the audio output device 1010. The performance script may specify which loops to play, when they should be played, which effects should be applied by module 1032 (e.g., per track or per subsection), and which filters should be applied by module 1031.

[0151] For example, a performance script may specify that a low pass filter in the range 1000-20000 Hz should be applied from 0-5000 ms to a particular track. As another example, a performance script may specify that a 0.2 wet reverb set should be applied from 5000-15000 ms to a particular track.

[0152] In some embodiments, the mix module 1033 is configured to perform automatic level control for the combined tracks. In some embodiments, the mix module 1033 uses frequency domain analysis of the combined tracks to determine frequencies with too much or too little energy and applies gain to tracks in different frequency bands to even out the mix. In some embodiments, the master module 1034 is configured to perform multi-band compression, equalization (EQ) or limiting procedures to generate data for final formatting by the execute module 1035. The embodiment of FIG. 10 may automatically generate different output music content according to user input or other feedback information, while machine learning techniques allow for improvement of the user experience over time.

[0153] 11 is a diagram illustrating an example build-up section of musical content, according to some embodiments. The system of FIG. 10 may compose such a section by applying musical arrangement rules and techniques. In the illustrated example, the build-up section includes three sub-sections and separate tracks for vocals, pads, drums, bass, and white noise.

[0154] In the illustrated example, the transition in subsection includes drum loop A, which is also repeated for the main content subsection. The transition in subsection also includes bass loop A. As illustrated, the gain for the section starts low and increases linearly throughout the section (although non-linear increases and decreases are possible). In the illustrated example, the main content and transition out subsections include various vocal, pad, drum and bass loops. As discussed above, the disclosed techniques for automatically sequencing sections, arranging sections and techniques for implementing the techniques may generate a nearly infinite stream of output musical content based on various user adjustable parameters.

[0155] In some embodiments, the computer system displays an interface similar to Figure 11, allowing the artist to specify the techniques to use to compose a section. For example, the artist may create a structure such as that shown in Figure 11, which can be parsed into code for the composition module.

[0156] FIG. 12 illustrates an example technique for arranging sections of musical content according to some embodiments. In the illustrated embodiment, a generated stream 1210 includes multiple sections 1220, each including a start subsection 1222, an expansion subsection 1224, and a transition subsection 1226. In the illustrated example, each section / subsection of multiple types is shown in a table with dotted lines connecting them. In the illustrated embodiment, the circular elements are examples of arrangement tools, which may be further implemented using certain techniques described below. As illustrated, various composition decisions may be made pseudo-randomly according to statistical percentages. For example, the type of subsection, the arrangement tools for a particular type or subsection, or the techniques used to implement the arrangement tools may be statistically determined.

[0157] In the illustrated example, a given section 1220 is one of five types: intro, buildup, drop, breakdown, and bridge, each with a different function of controlling intensity over the section. In this example, state subsections are one of three types: slow build, abrupt shift, or minimal, each with different behavior. The evolution subsections in this example are one of three types: reduce, transform, and aggression. In this example, transition subsections are one of three types: collapse, ramp, and hint. The different types of sections and subsections may be selected based on rules, for example, or may be selected pseudo-randomly.

[0158] In the illustrated examples, the behaviors for the different types of subsections are implemented using one or more arrangement tools. For a slow build, in this example, a low pass filter is applied at a rate of 40% and a layer is added at a rate of 80%. For a transform expand subsection, in this example, a loop is chopped at a rate of 25%. Various additional arrangement tools are shown, including one shots, drop out beats, apply reverb, add pads, add themes, remove layers, and white noise. These examples are included for illustrative purposes and are not intended to limit the scope of the present disclosure. Additionally, for ease of explanation, these examples may not be complete (e.g., real arrangements may typically involve a much larger number of arrangement rules).

[0159] In some embodiments, one or more arrangement tools may be implemented using a particular technique (which may be specified or determined by the artist based on an analysis of the artist's content). For example, one shots may be implemented using sound effects or vocals, loop chopping may be implemented using a stutter or chop half technique, layer removal may be implemented by removing synths or vocals, and white noise may be implemented using a ramp or pulse function. In some embodiments, the particular technique selected for a given arrangement tool may be selected according to a statistical function (e.g., layer removal may remove synths 30% of the time for a given artist and vocals 70% of the time). As mentioned above, arrangement rules or techniques may be determined automatically by analyzing existing compositions, for example using machine learning. Method Example

[0160] 13 is a flow diagram illustrating a method for training a music generation system according to some embodiments. The method illustrated in FIG. 13 may be used with, among others, any of the computer circuits, systems, devices, elements, or components disclosed herein. In various embodiments, some of the illustrated method elements may be performed simultaneously, in a different order than illustrated, or may be omitted. Additional method elements may be performed as desired.

[0161] At 1310, in the illustrated embodiment, a music generator on the computer system generates output musical content of a particular length from a plurality of digital musical fragments, the output musical content including a series of different output musical content generated from the plurality of digital musical fragments, the music generator generating the series of different output musical content in an audio format by sequentially interpreting one or more music generation parameters according to a set of interpretation rules.

[0162] At 1320, in the illustrated embodiment, the music generator receives the enhanced input determined from the generated output music content.

[0163] At 1330, in the illustrated embodiment, music generation parameters for the music generator are adjusted based on the reinforcement input using an algorithm, the music generation parameters including at least one of previously generated music generation parameters or user-created music generation parameters.

[0164] At 1340, in the illustrated embodiment, new output musical content of a particular length is generated from the plurality of digital musical pieces based on the adjusted music generation parameters.

[0165] 14 is a flow diagram of a method for controlling user-specific music control elements, according to some embodiments. The method shown in FIG. 14 may be used with, among others, any of the computer circuits, systems, devices, elements, or components disclosed herein. In various embodiments, some of the illustrated method elements may be performed simultaneously, in a different order than illustrated, or may be omitted. Additional method elements may be performed as desired.

[0166] At 1410, in the illustrated embodiment, the computer system receives user input defining a user-defined musical control element.

[0167] At 1420, in the illustrated embodiment, the computer system receives user input defining labels for user-defined musical control elements for one or more compositions.

[0168] At 1430, in the illustrated embodiment, the computer system trains one or more machine learning models based on the labels to: adjust one or more composition parameters for selecting, based on user input to the user-defined musical control elements, audio tracks to be combined to generate the output musical content; and adjust one or more performance parameters for generating the output musical content based on the selected audio tracks, based on user input to the user-defined musical control elements.

[0169] At 1440, in the illustrated embodiment, the computer system generates output musical content by combining multiple audio tracks according to one or more composition parameters and one or more performance parameters.

[0170] In some embodiments, the computer system transmits configuration information based on one or more machine learning models to the user device, the configuration information indicating how to adjust the performance parameters based on user input to the user-defined musical control elements. In some embodiments, the computer system adjusts one or more composition parameters based on user input to the user-defined musical control elements according to the one or more machine learning models. In some embodiments, the user device of the computer system adjusts one or more composition parameters based on user input to the user-defined musical control elements according to the configuration information.

[0171] In some embodiments, the sending uses a delimited string to specify the data target, and the method further includes forming the string based on a data structure having a pointer to an offset within the string, and forming the string includes prepending or appending to an existing string.

[0172] In some embodiments, the training is further based on user input that explicitly defines relationships between the composition parameters and the user input to the user-defined musical control elements.

[0173] In some embodiments, the one or more composition parameters include parameters for building, maintaining and reducing tension between musical sections, parameters for enabling or disabling audio file categories, parameters for adjusting one or more musical aspects such as volume, amount of reverb, delayed punch probability, delayed wetness, etc. In some embodiments, the one or more performance parameters include at least one of the following types of parameters: filter parameters, effect parameters, and mix parameters.

[0174] 15 is a flow diagram of a method for generating musical content based on user-specific musical control elements, according to some embodiments. The method shown in FIG. 15 may be used with, among others, any of the computer circuits, systems, devices, elements, or components disclosed herein. In various embodiments, some of the illustrated method elements may be performed simultaneously, in a different order than illustrated, or may be omitted. Additional method elements may be performed as desired.

[0175] At 1510, in the illustrated embodiment, the computer system receives configuration information, the configuration information specifying how to adjust one or more performance parameters based on user adjustments to user-defined musical control elements, the one or more performance parameters for generating output musical content based on a selected audio track.

[0176] At 1520, in the illustrated embodiment, the computer system receives user input via a first user interface, the user input defining an adjustment to the user-defined musical control element.

[0177] At 1530, in the illustrated embodiment, the computer system adjusts one or more of the one or more performance parameters based on the user input and the setting information.

[0178] At 1540, in the illustrated embodiment, the computer system generates output musical content by combining the multiple audio files according to the one or more adjusted performance parameters.

[0179] In some embodiments, the computer system receives composition control information (e.g., a composition script) that indicates multiple audio files to be combined and indicates transitions between musical sections. In some embodiments, the composition control information is generated based on user input (e.g., by an artist via an artist user interface or by an end-user via a first user interface) that specifies adjustments to user-defined musical control elements.

[0180] In some embodiments, the user-defined music control elements are defined via a second user interface generated by the server system that sent the configuration information, hi some embodiments, the user-defined music control elements are defined by a user different from the user providing user input via the first user interface.

[0181] In some embodiments, the computer system parses the configuration information using pointers to locations within a delimited string that indicate the target of the configuration information.

[0182] In various embodiments herein, techniques related to automatically generating new musical content are disclosed. In some embodiments, a computer system receives user input specifying user-defined musical control elements. The computer system may train machine learning models to modify both composition and performance parameters based on the user's adjustments to the user-defined musical control elements. In embodiments where the composition and performance subsystems are on different devices, one device may send configuration information to another device, where the configuration information specifies how to adjust the parameters based on the user's input to the user-defined musical control elements. The disclosed techniques may facilitate focused learning for human-like music production while allowing individualized customization for each individual user. The disclosed techniques may also allow artists to define their own abstract musical controls and make those controls available to end users.

[0183] The following numbered paragraphs define various non-limiting embodiments disclosed herein. Set A A1. receiving, by a computer system, a user input defining a user-defined musical control element; receiving, by the computer system, user input defining labels for the user-defined musical control elements for one or more musical compositions; The computer system may then generate one or more machine learning models based on the labels, adjusting, based on user input to the user-defined musical control elements, one or more composition parameters for selecting audio tracks to be combined to generate output musical content; adjusting, based on user input to the user-defined musical control elements, one or more performance parameters for generating output musical content based on a selected audio track; and generating output musical content by combining a plurality of audio tracks in accordance with the one or more composition parameters and the one or more performance parameters; The method includes: A2. the computer system is a server system, and the method comprises: sending configuration information to a user device based on the one or more machine learning models, the configuration information indicating how to adjust performance parameters based on user input to the user-defined musical control elements; The method of the preceding items in Set A, further comprising: A3. The method of any preceding item in set A, further comprising adjusting, by the computer system, one or more composition parameters based on user input to the user-defined musical control elements in accordance with the one or more machine learning models. A4. The method of any preceding claim in set A, further comprising adjusting, by the user device, one or more composition parameters based on user input to the user-defined musical control elements in accordance with the setting information. A5. The method of any preceding item in set A, wherein the sending uses a delimited string to identify a data target, the method further including forming a string based on a data structure having a pointer to an offset within the string, and forming the string includes prepending or appending to an existing string. A6. The method of any preceding claim in set A, wherein said training is further based on user input that explicitly specifies a relationship between a composition parameter and a user input to said user-defined musical control element. A7. The method of any preceding claim in set A, wherein the one or more composition parameters include parameters for building, maintaining, and reducing tension between musical sections. A8. The one or more composition parameters are: A parameter for enabling or disabling the Audio File category; parameters for adjusting one or more musical aspects for one or more audio file categories, the musical aspects including volume, amount of reverb, delayed punch probability, and delayed wetness; The method of any preceding claim in set A, comprising: A9. The method of any preceding claim in set A, wherein the one or more performance parameters include at least one of the following types of parameters: filter parameters, effect parameters, and mix parameters. A10. A non-transitory computer readable medium having instructions stored thereon, the instructions comprising: receiving configuration information that specifies how to adjust one or more performance parameters based on user adjustments to user-defined musical control elements, the one or more performance parameters being for generating output musical content based on a selected audio track; receiving user input via a first user interface, the user input defining an adjustment to the user-defined music control element; adjusting one or more of the one or more performance parameters based on the user input and the setting information; generating output musical content by combining a plurality of audio files in accordance with the one or more adjusted performance parameters; A non-transitory computer-readable medium executable by a computing device to perform operations including: A11. The non-transitory computer-readable medium of any of the preceding clauses in set A, further comprising receiving composition control information indicating the plurality of audio files to be combined and indicating transitions between musical sections. A12. The non-transitory computer-readable medium of any of the preceding clauses in set A, wherein the composition control information is generated based on user inputs defining adjustments to the user-defined musical control elements. A13. The non-transitory computer-readable medium of any of the preceding clauses in set A, wherein the user-defined music control elements are defined via a second user interface generated by a server system that transmitted the setting information. A14. The non-transitory computer-readable medium of any of the preceding clauses in set A, wherein the user-defined music control elements are defined by a user different from the user providing the user input via the first user interface. A15. The non-transitory computer-readable medium of any of the preceding clauses in set A, further comprising parsing the configuration information using a pointer to a location within a delimited string indicating a target of the configuration information. A16. 1. An apparatus including one or more processors, The one or more processors: receiving configuration information that specifies how to adjust one or more performance parameters based on user adjustments to user-defined musical control elements, the one or more performance parameters being for generating output musical content based on a selected audio track; receiving user input via a first user interface, the user input defining an adjustment to the user-defined music control element; adjusting one or more of the one or more performance parameters based on the user input and the setting information; generating output musical content by combining a plurality of audio files in accordance with the one or more adjusted performance parameters; The apparatus is configured to: A17. The one or more processors: receiving composition control information indicating the plurality of audio files to be combined and indicating transitions between musical sections, the composition control information being generated by a machine learning model based on user inputs defining adjustments to the user-defined musical control elements; The apparatus of any preceding item in set A, further configured to: A18. The apparatus of any of the preceding clauses in set A, wherein the user-defined music control elements are defined via a second user interface generated by a server system that transmitted the setting information. A19. The one or more processors: parsing the configuration information using a pointer to a location within a delimited string indicating a target of the configuration information; The apparatus of any preceding item in set A, further configured to: A20. The device of any of the preceding clauses in set A, wherein the device is a mobile device including a display configured to display the first user interface and one or more wireless radios configured to connect to a wireless network. A21. A non-transitory computer readable medium having instructions stored thereon, the instructions being executable by a computing device to perform any combination of the operations performed by the methods described in any of the preceding items in set A. A22. one or more processors; one or more memories having program instructions stored therein; An apparatus comprising: The program instructions are executable by the one or more processors to perform any combination of operations performed by the method described in any of the preceding clauses in set A.

[0184] The present disclosure includes references to "embodiments" or groups of "embodiments" (e.g., "some embodiments" or "various embodiments"). Embodiments are different implementations or instances of the disclosed concepts. References to "an embodiment," "one embodiment," "a particular embodiment," etc. do not necessarily refer to the same embodiment. In addition to those specifically disclosed, numerous possible embodiments are contemplated, including modifications or substitutions that are within the spirit or scope of the present disclosure.

[0185] This disclosure may describe potential advantages that may result from the disclosed embodiments. Not all implementations of these embodiments necessarily manifest any or all of the potential advantages. Whether advantages are realized for a particular implementation depends on many factors, some of which are outside the scope of this disclosure. Indeed, there are many reasons why an implementation within the scope of the claims may not exhibit some or all of the disclosed advantages. For example, a particular implementation may include other circuitry outside the scope of this disclosure that, in combination with one of the disclosed embodiments, negates or reduces one or more of the disclosed advantages. Furthermore, suboptimal design implementations (e.g., implementation techniques or tools) may also negate or reduce the disclosed advantages. Even assuming skilled implementation, realization of advantages may still depend on other factors, such as the environmental conditions in which the implementation is deployed. For example, inputs provided to a particular implementation may prevent one or more of the issues addressed in this disclosure from arising on a particular occasion, such that the benefits of its solution may not be realized. Given the existence of possible factors external to this disclosure, it is expressly intended that the potential advantages described herein should not be construed as limitations on the scope of the claims that must be satisfied in order to prove infringement. Rather, the identification of such potential advantages is intended to illustrate the types of improvements available to a designer having the benefit of this disclosure. The fact that such advantages have been liberally described (e.g., stating that a particular advantage "may result") is not intended to convey any doubt as to whether such advantages can actually be realized, but rather to acknowledge the technological reality that the realization of such advantages often depends on additional factors.

[0186] Unless otherwise specified, the embodiments are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of the claims that are drafted based on this disclosure, even if only a single example of a particular feature is described. The disclosed embodiments are intended to be illustrative, not limiting, unless otherwise stated in the disclosure. As such, it is intended that the claims be able to cover the disclosed embodiments, as well as such alternatives, modifications, and equivalents that would be apparent to one of ordinary skill in the art having the benefit of this disclosure.

[0187] For example, features in this application may be combined in any suitable manner. Accordingly, during prosecution of this application (or an application claiming priority), new claims may be formulated to such combinations of features. In particular, with reference to the appended claims, features from dependent claims may be combined, as appropriate, with features of other dependent claims, including claims that are dependent on other independent claims. Similarly, features from each independent claim may be combined, as appropriate.

[0188] Thus, although the appended dependent claims may be drafted such that each is dependent on one other claim, additional dependencies are also contemplated. Any combination of features in the dependent claims consistent with this disclosure is contemplated and may be claimed in this or another application. In other words, combinations are not limited to those specifically recited in the appended claims.

[0189] Where appropriate, a claim drafted in one form or statutory type (e.g., apparatus) may be considered to be intended to support a corresponding claim in another form or statutory type (e.g., method).

[0190] Because this disclosure is a legal document, various terms and phrases may be subject to administrative and judicial interpretation. It is hereby announced that the definitions provided in the following paragraphs and throughout this disclosure should be used to determine how any claims drafted based on this disclosure should be interpreted.

[0191] Reference to a singular item (i.e., a noun or noun phrase preceded by "a," "an," or "the") is intended to mean "one or more" unless the context clearly indicates otherwise. Thus, a reference to "an item" in a claim does not exclude additional instances of that item without attendant context. A "plurality" of an item refers to a collection of two or more items.

[0192] The term "may" is used herein in a permissive sense (i.e., having the potential or ability) and not in a mandatory sense (i.e., absolute).

[0193] The terms "comprise" and "include" and their variants are open ended and mean "including, but not limited to."

[0194] In this disclosure, when the term "or" is used in connection with a list of alternatives, it is generally understood to be used in an inclusive sense unless otherwise specified. Thus, a description of "x or y" is equivalent to "x or y or both," and thus, 1) instead of y 1) covers y but not x, 2) covers y but not x, and 3) covers both x and y. On the other hand, expressions such as "either x or y" make it clear that "or" is used in an exclusive sense.

[0195] The phrases "w, x, y, or z, or combinations thereof" or "at least one of w, x, y, and z" are intended to cover all possibilities, including a single element to the total number of elements in the set. For example, if we consider the set "w, x, y, z," these phrases cover any single element of the set (e.g., w but not x, y, z), any two elements (e.g., w and x but not y or z), any three elements (e.g., w, x, and y but not z), and all four elements. Thus, the phrase "at least one of w, x, y, and z" refers to at least one element of the set "w, x, y, z," and therefore covers all possible combinations in the list of elements. This phrase does not require that there is at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0196] In this disclosure, various "labels" may precede nouns or noun phrases. Unless otherwise specified, different labels used for a feature (e.g., "first circuit," "second circuit," "particular circuit," "given circuit," etc.) refer to different instances of the feature. In addition, labels such as "first," "second," and "third," when applied to features, do not imply any kind of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise specified.

[0197] In this specification, the term "based on" is used to describe one or more factors that influence the decision. This term does not exclude the possibility that additional factors may influence the decision. That is, the decision may be based only on the identified factors or may be based on the identified factors plus other unidentified factors. When considering the phrase "determining A based on B," this phrase specifies that B may be a factor used to determine A or influence the decision of A. This phrase does not exclude that the decision of A may also be based on other factors, such as C. This phrase is intended to cover embodiments in which A is determined only based on B. As used herein, the term "based on" is synonymous with the term "based at least in part on."

[0198] The phrases "in response to" and "in reaction to" describe one or more factors that cause an effect. This does not exclude the possibility that additional factors may affect or otherwise cause the effect, either jointly with or independently of the specified factors. That is, the effect may be responsive only to those factors or to other unspecified factors in addition to the specified factors. Consider the phrase "doing A in response to B." This phrase specifies that B is the factor that causes the performance of A or that causes a particular outcome of A. This phrase does not exclude that doing A may also be in response to other factors, such as C. This phrase does not preclude that doing A may also be in reaction to B and C jointly. This phrase is also intended to cover embodiments in which A is done only in response to B. As used herein, the phrase "in response to" is synonymous with the phrase "at least partially in reaction to." Similarly, the phrase "in response to" is synonymous with the phrase "at least partially in response to."

[0199] Within this disclosure, different entities (which may be variously referred to as "units," "circuits," other components, etc.) may be described or claimed as being "configured" to perform one or more tasks or operations. The formula "an entity configured to perform one or more tasks" is used herein to refer to a structure (i.e., something physical). More specifically, this formula is used to indicate that such a structure is arranged to perform one or more tasks during operation. A structure may be said to be "configured" to perform a task even if it is not currently operating. Thus, an entity described or claimed as being "configured" to perform a task represents something physical, such as a device, circuit, system, etc., having a processor unit and a memory that stores executable instructions to perform such a task. This expression is not used herein to represent something intangible.

[0200] In some cases, various units / circuits / components may be described herein as performing a series of tasks or operations, and it will be understood that these entities are "configured" to perform these tasks / operations, even if not specifically stated otherwise.

[0201] The term "configured" is not intended to mean "configurable." For example, an unprogrammed FPGA is not considered to be "configured" to perform a particular function. However, this unprogrammed FPGA may be "configurable" to perform that function. After appropriate programming, the FPGA may be configured to perform that particular function.

[0202] For purposes of filing a U.S. patent application based on this disclosure, any recitation in the appended claims that a structure is "configured to" perform one or more tasks is expressly not intended to invoke 35 U.S.C. § 112(f) with respect to the recited element. Thus, if an applicant wishes to invoke 35 U.S.C. § 112(f) during prosecution of a U.S. patent application based on this disclosure, the applicant will recite the claim element using "means for" [performing a function].

[0203] Different "circuits" may be described in this disclosure. These circuits or "electrical circuits" constitute hardware that includes various types of circuit elements, such as combinational logic, clocked storage (e.g., flip-flops, registers, latches, etc.), finite state machines, memory (e.g., random access memory, embedded dynamic random access memory), programmable logic arrays, etc. Circuits may be custom designed or taken from standard libraries. In various implementations, circuits may include digital components, analog components, or a combination of both, as appropriate. Certain types of circuits may be generally referred to as "units" (e.g., decode units, arithmetic logic units (ALUs), functional units, memory management units (MMUs), etc.). Such units may also be referred to as circuits or electrical circuits.

[0204] Thus, the circuits / units / components and other elements of the disclosure illustrated in the drawings and described herein include hardware elements such as those described in the preceding paragraphs. Often, the internal arrangement of hardware elements within a particular circuit can be specified by describing the function of that circuit. For example, a particular "decode unit" is described as performing the function of "processing the opcode of an instruction and routing the instruction to one or more of a plurality of functional units," meaning that the decode unit is "configured" to perform this function. The specification of this function is sufficient to suggest to one of ordinary skill in the computer arts a set of possible structures for the circuit.

[0205] In various embodiments, as described in the preceding paragraphs, circuits, units, and other elements may be defined by the functions or operations they are configured to perform. The arrangement of such circuits / units / components relative to one another and the manner in which they interact form a microarchitectural definition of the hardware that is ultimately fabricated in an integrated circuit or programmed into an FPGA to form a physical implementation of the microarchitectural definition. As such, a microarchitectural definition is recognized by those skilled in the art as a structure from which many physical implementations may be derived, all of which fall into the broader structure described by the microarchitectural definition. That is, a person skilled in the art presented with a microarchitectural definition provided in accordance with this disclosure will, without undue experimentation and applying ordinary skill, implement the structure by coding the circuit / unit / component description in a hardware description language (HDL) such as Verilog or VHDL. HDL descriptions are often expressed in a way that makes them appear functional. However, to those skilled in the art, this HDL description is a method used to translate the structure of a circuit, unit, or component into the next level of implementation details. Such HDL descriptions may take the form of behavioral code (usually not synthesizable), register transfer language (RTL) code (usually synthesizable in contrast to behavioral code), or structural code (e.g., a netlist that specifies logic gates and their connectivity). The HDL description is then synthesized against a library of cells designed for a given integrated circuit fabrication technology, modified for timing, power, and other reasons, resulting in a final design database that is sent to a foundry to generate masks and, ultimately, an integrated circuit. Some hardware circuits, or portions thereof, may be custom designed in a schematic editor and incorporated into the integrated circuit design along with the synthesized circuit. An integrated circuit may include transistors and other circuit elements (e.g., passive elements such as capacitors, resistors, inductors, etc.), and interconnects between the transistors and circuit elements. In some embodiments, multiple integrated circuits may be implemented coupled together to implement the hardware circuit, and / or in some embodiments, discrete elements may be used.Alternatively, the HDL design may be synthesized into a programmable logic array, such as a field programmable gate array (FPGA), and implemented in the FPGA. This separation between the design of groups of circuits and the subsequent low-level implementation of those circuits typically results in a scenario where the circuit or logic designer does not specify a particular set of structures for the low-level implementation beyond describing what the circuit is configured to do, because this process occurs at different stages of the circuit implementation process.

[0206] The fact that many different low-level combinations of circuit elements may be used to implement the same specification of a circuit results in numerous equivalent structures of that circuit. As mentioned above, these low-level circuit implementations may differ due to variations in manufacturing technology, the foundry selected to manufacture the integrated circuit, the libraries of cells provided for a particular project, etc. The choices made by different design tools or methodologies to create these different implementations may often be arbitrary.

[0207] Furthermore, for a given embodiment, it is common for a single implementation of a particular functional specification of a circuit to include a large number of devices (e.g., millions of transistors). Thus, due to the volume of this information, it is impractical to provide a complete description of the low-level structure used to implement a single embodiment, let alone the many possible sets of equivalent implementations. Therefore, this disclosure describes the structure of the circuit using functional abbreviations commonly used in the industry.

Claims

1. receiving, at a computer system, user input related to setting by a user of one or more music controls on a user interface, the music controls controlling one or more audio properties of the musical content; generating, at the computer system, a set of initial music generation parameters based on the received user input; generating, by a music generator on said computer system, digital output musical content from a plurality of digital musical fragments, said digital output musical content comprising a series of different digital output musical content generated from said plurality of digital musical fragments, said music generator generating said series of different digital output musical content in audio form by sequentially interpreting one or more of said music generation parameters starting from said set of initial music generation parameters according to a set of interpretation rules; determining at said computer system an enhancement input, said enhancement input including a determination of whether said digital output musical content has been generated by a computer implemented music generator or by a human composer; adjusting, with an algorithm, at least one of the one or more music generation parameters for the music generator based on the reinforcement input; generating new digital output musical content from the plurality of digital musical pieces based on the adjusted at least one of the one or more music generation parameters; The method includes:

2. The method of claim 1 , wherein receiving the user input comprises receiving a user selection of the music control settings while music content is being played via the user interface.

3. 2. The method of claim 1, wherein the music controls control one or more of the following audio properties: gain, high pass cutoff, low pass cutoff, reverb send level, reverb length, delay level, delay time, bass, beat, pad, top, melody, backing melody, chords, and effects.

4. 2. The method of claim 1 , wherein determining the enrichment input comprises classifying the digital output musical content with a classifier on the computer system using a set of one or more trained classifiers to determine whether the digital output musical content has been generated by the computer-implemented music generator or by the human composer.

5. 5. The method of claim 4, wherein the reinforcement input corresponds to a prediction made by the classifier about the probability that the digital output musical content has been generated by the computer-implemented music generator or by the human composer.

6. 5. The method of claim 4, further comprising classifying the digital output musical content with an additional set of one or more trained classifiers to determine a prediction of one or more artists associated with the digital output musical content.

7. The method of claim 1 , wherein the reinforcement input comprises a binary input to the algorithm, the binary input being a determination of whether the digital output musical content meets a listening threshold.

8. The method of claim 1 , wherein the one or more music generation parameters in the music generator are implemented by a trained machine learning algorithm.

9. The method of claim 1 , wherein the one or more music generation parameters are parameters that control the selection and processing of the music fragments.

10. receiving, at the computer system, user input associated with a setting of the one or more music controls by the user on the user interface during playback of the digital output musical content; modifying at least one of the one or more music generation parameters at the computer system based on the received user input; generating, by the music generator, additional digital output music content from the plurality of digital music pieces in accordance with the modified at least one of the one or more music generation parameters; The method of claim 1 further comprising:

11. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a computer, receiving user input related to setting by a user of one or more music controls on a user interface, the music controls controlling one or more audio properties of the musical content; generating a set of initial music generation parameters based on the received user input; generating digital output musical content from a plurality of digital musical fragments by a music generator, the digital output musical content comprising a series of different digital output musical content generated from the plurality of digital musical fragments, the music generator generating the series of different digital output musical content in audio form by sequentially interpreting one or more of the music generation parameters starting from the set of initial music generation parameters according to a set of interpretation rules; determining an enhancement input, the enhancement input including a determination of whether the digital output musical content is generated by a computer-implemented music generator or a human composer; adjusting, with an algorithm, at least one of the one or more music generation parameters for the music generator based on the reinforcement input; generating new digital output musical content from the plurality of digital musical pieces based on the adjusted at least one of the one or more music generation parameters; A non-transitory computer readable medium on which operations including

12. 12. The non-transitory computer-readable medium of claim 11, further comprising classifying the output musical content with a classifier using a set of one or more trained classifiers to determine a prediction of whether the digital output musical content has been generated by the computer-implemented music generator or by the human composer.

13. 12. The non-transitory computer-readable medium of claim 11, wherein adjusting the at least one of the one or more music generation parameters for the music generator comprises adjusting the at least one of the one or more music generation parameters such that the music generator generates output music content that sounds like it was composed by a human.

14. 12. The non-transitory computer-readable medium of claim 11, wherein the one or more music generation parameters are parameters that control selection and processing of the musical fragments, and processing of the musical fragments includes applying the musical controls to the digital output musical content.

15. 12. The non-transitory computer-readable medium of claim 11, wherein the music controls control one or more of the following audio properties: gain, high pass cutoff, low pass cutoff, reverb send level, reverb length, delay level, delay time, bass, beat, pad, top, melody, backing melody, chords, and effects.

16. 12. The non-transitory computer-readable medium of claim 11 , wherein adjusting the at least one of the one or more music generation parameters for the music generator includes providing a binary input to the algorithm, the binary input being a prediction of whether the output musical content will meet a listening threshold.

17. one or more processors, at least one processor associated with a music generator implementing at least one algorithm; one or more memories having program instructions stored therein; wherein the program instructions, when executed by the one or more processors, cause the one or more processors to: receiving user input related to setting by a user of one or more music controls on a user interface, the music controls controlling one or more audio properties of the musical content; generating a set of initial music generation parameters based on the received user input; generating digital output musical content from a plurality of digital musical fragments by a music generator, the digital output musical content comprising a series of different digital output musical content generated from the plurality of digital musical fragments, the music generator generating the series of different digital output musical content in audio form by sequentially interpreting one or more of the music generation parameters starting from the set of initial music generation parameters according to a set of interpretation rules; determining an enhancement input, the enhancement input including a determination of whether the digital output musical content is generated by a computer-implemented music generator or a human composer; adjusting, with an algorithm, at least one of the one or more music generation parameters for the music generator based on the reinforcement input; generating new digital output musical content from the plurality of digital musical pieces based on the adjusted at least one of the one or more music generation parameters; A device that performs the above function.

18. 20. The apparatus of claim 17, wherein at least one processor includes a classifier, and further program instructions are executable by the classifier to determine whether the digital output musical content has been generated by the computer-implemented music generator or by the human composer.

19. 20. The apparatus of claim 17, wherein the reinforcement input comprises a binary input to the algorithm, the binary input being a determination of whether the digital output musical content meets a listening threshold.

20. 20. The apparatus of claim 17, wherein the music controls control one or more of the following audio properties: gain, high pass cutoff, low pass cutoff, reverb send level, reverb length, delay level, delay time, bass, beat, pad, top, melody, backing melody, chords, and effects.

Images