Scalable Similarity-Based Adaptive Music Mix Generation

The system addresses the challenge of creating high-quality music mixes by using pitch interval space vectors to analyze and suggest harmonically compatible music clips, enhancing the efficiency and quality of mix generation.

JP2024542254AActive Publication Date: 2024-11-13DISTRIBUTED CREATION INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024530463
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-01-26
Publication Date
2024-11-13
Estimated Expiration
2043-01-26

AI Technical Summary

Technical Problem

Creating music mixes from large collections of music clips is challenging due to the difficulty in finding musically compatible sounds, as existing systems lack efficient methods for determining harmonic and other musical attributes of clips, leading to frustration and low-quality mixes.

Method used

A computer-implemented system uses pitch interval space vectors to calculate harmonic compatibility between music clips, allowing for efficient identification of harmonically compatible clips, incorporating additional attributes like rhythmic, spectral, and timbral compatibility, and employing a cloud-based service for scalable music mix generation.

Benefits of technology

Enables the creation of high-quality music mixes by efficiently suggesting musically compatible clips, reducing user frustration and improving the perceived quality of mixes through advanced harmonic and attribute analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542254000001_ABST
    Figure 2024542254000001_ABST
Patent Text Reader

Abstract

A scalable similarity-based matched music mix generation is disclosed. Music clips are projected into a pitch interval space, and the musical compatibility between clips is calculated as a distance or similarity in the pitch interval space. The distance or similarity between clips reflects the degree to which the clips are harmonically matched. The distance or similarity between a candidate music clip and a partial mix in the pitch interval space can be used to determine whether the candidate music clip is harmonically matched with the partial mix. The indexable feature space can be both beats per minute (BPM) and musical key independent, allowing for rapid determination of harmonic compatibility between potentially millions of music clips. A graphical user interface-based user application allows a user to easily find combinations of clips from a library that produce perceptually high quality mixes that are harmonically and pleasingly sounding and reflect the harmonic principles of music.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates generally to the field of computer generated music, and more particularly to a new and useful computer implemented system and method for scalable similarity-based adaptive music mix generation. [Background technology]

[0002] Creating a music mix involves the creation and assembly of music tracks. This creative endeavor is often associated with DJing and electronic dance music (EDM). Recently, creating a music mix has been made easier by online collections of royalty-free sounds in digital format. An example of such a collection is the sound sample library available from SPLICE.COM, Santa Monica, California and New York, New York. Such libraries may contain thousands or even millions of sound samples. The size of such libraries poses a technical challenge to obtain sounds that match a criteria in a computationally efficient manner. To create a music mix, computer-based tools are needed that streamline the search, discovery, and acquisition of musically compatible sounds. The present invention provides such a new and useful system and method. [Brief description of the drawings]

[0003] [Figure 1] 1 illustrates a system for generating a matched music mix based on similarity, according to several variations. [Diagram 2] We present methods for generating and indexing beats per minute (BPM) independent per-clip pitch interval space vectors for music clips, in several variations. [Diagram 3] A plot shows the constant-Q transformation matrix of an example music clip with several variations. [Figure 4]4 shows plots of chroma saliency maps generated based on the constant-Q transformation matrix shown in FIG. 3 according to several variations. [Diagram 5] Plots show beats-per-minute independent chroma representations generated based on the chroma saliency map shown in FIG. 4 according to several variations. [Figure 6] 6 shows plots of two matrices containing real and imaginary components of beat-by-beat pitch interval space vectors generated from the BPM-independent chroma representation matrix of FIG. 5 according to several variations. [Figure 7] The plots show the results of concatenating the real and imaginary components of the two matrices of FIG. 6, according to several variations. [Figure 8] 8 shows the flattening of the matrix of FIG. 7 into a per-clip pitch interval space vector, according to several variations. [Figure 9] The values ​​of the per-clip pitch interval space vector shown in FIG. 8 are shown in waveform plots according to several variations. [Figure 10] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 11] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 12] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 13] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 14] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 15]1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 16] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 17] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 18] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 19] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 20] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 21] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 22] 1 illustrates various states of a graphical user interface of a stack-based music mixing application, according to several variations. [Figure 23] 1 illustrates a computer system upon which some variations may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] The following description of the preferred embodiments is not intended to limit the present disclosure to those preferred embodiments, but rather to enable any person skilled in the art to make and use the present disclosure.

[0005] The compatibility of the music clips (e.g., mixes, stems, or individual tracks) that make up a music mix can be very important to the perceived quality of the mix. A perceptually high quality mix is ​​one that is harmonically and pleasant sounding, reflecting, implementing, or satisfying the harmonic principles of music. Unfortunately, it may not be known in advance which combinations of music clips will produce a perceptually high quality mix. Thus, the ability to try out different combinations of music clips is useful. Along with the desire to experiment, there is also the desire to produce a perceptually high quality mix.

[0006] In some variations, the computer-implemented techniques disclosed herein assist users in easily discovering combinations of music clips that provide a perceptually high-quality music mix in the context of music mix creation. The techniques use a harmonic compatibility approach to balance the need to try out various music clips with the need to efficiently discover perceptually high-quality clips. The techniques include using a pitch interval space to calculate the harmonic compatibility between music clips as a distance or similarity between the music clips in the pitch interval space. The distance or similarity between the music clips in the pitch interval space reflects the degree to which the music clips are harmonically compatible. Given a candidate music clip to add to a partial mix of one or more music clips, the distance or similarity between the candidate music clip and the partial mix in the pitch interval space may be used to determine whether the candidate music clip is harmonically compatible with the partial mix. In some variations, an indexable feature space is provided that is independent of both beats per minute (BPM) and musical key. That is, the harmonic compatibility between clips can be determined even if the clips are in different BPMs or keys. Furthermore, the music clip index can be scaled to millions of music clips and can be used to identify music clips that are harmonically compatible with a given music clip with low latency (eg, less than 10 milliseconds).

[0007] As an example of the problem addressed by the techniques herein in some variations, consider a partial mix that combines music clips from a library of music clips provided by a music mixing computing system (e.g., a cloud-based music mixing computing system). A user of the system may then wish to add additional music clips (e.g., bass line stems) from the library to the partial mix. The music mixing system allows the user to browse, search, and access the music clips in the library. Such libraries may be large (e.g., thousands or millions of music clips). Without the help and guidance of the music mixing system, it is very difficult for the user to find music clips that match the partial mix. Thus, when creating a complete mix, the user may easily become frustrated or overwhelmed trying to find matching music clips. Thus, it is very important to streamline the music mix creation process by assisting the user in the process of finding matching music clips among a vast collection of music clips. Suitable assistance is important not only to operators of the music mixing system, who may have more users who use the system, create accounts, or are willing to upgrade their accounts, as exemplary benefits, but also to users themselves, who may use the music mixing system to streamline the music mix creation process. If the system suggests a music clip that is only rhythmically compatible (e.g., by onset density) with the partial mix, the resulting mix may be perceived as low quality. There may be another clip in the library that is more compatible with the partial mix and produces a perceptually higher quality mix. The technique extends the range of music attributes when determining the compatibility of a music clip to include harmonic attributes. Moreover, the technique can be used with more than just harmonic attributes. It can be used with any type of music attribute, such as rhythmic attributes, spectral attributes, and timbral attributes.

[0008] In some variations, the technique uses a harmonic compatibility approach in which the harmonic content of a music clip is represented as a multi-dimensional vector in pitch interval space (i.e., a "pitch interval space vector"). Each pitch interval space vector may have a unique location in pitch interval space, which represents a corresponding unique harmonic configuration. The distance or similarity between these pitch interval space vectors in pitch interval space may be calculated to determine the harmonic compatibility between the music clips. Furthermore, an element-wise linear combination of the pitch interval space vectors (e.g., by averaging or weighted averaging using the energy of the vectors) may be used to determine whether the candidate music clip is harmonically compatible with the partial mix. Specifically, the distance or similarity in pitch interval space between (a) an element-wise linear combination of the pitch interval space vectors of the music clips that make up the partial mix and (b) the pitch interval space vector of the candidate music clip reflects the degree to which the candidate music clip is harmonically compatible with the partial mix. Because these can be represented by a computer as vectors, computing element-wise linear combinations of the vectors and computing the distance or similarity between the vectors are relatively efficient computational operations. Thus, pitch interval space vectors allow a music mixing system to efficiently evaluate the harmonic compatibility of a large collection of candidate music clips.

[0009] The technique, in some variations, begins by receiving a request to suggest a music clip that is musically compatible with a partial mix of previously selected music clips. For example, the previously selected music clips may include a vocal clip and a piano clip. In response to receiving the request, the technique, in some variations, linearly combines the pitch interval space vectors of each of the previously selected music clips of the partial mix to generate a pitch interval space vector representing the harmonic attributes of the partial mix. The technique calculates a distance or similarity in pitch interval space between the pitch interval space vector of the partial mix and the pitch interval space vector representing the harmonic attributes of the candidate music clip. The technique, in some variations, responds to the request by suggesting a particular candidate music clip that is musically compatible with the partial mix based on the distance in pitch interval space between the pitch interval space vector representing the partial mix and the pitch interval space vector representing the harmonic attributes of the music clip. Returning to the previous example of this paragraph, the particular proposed music clip may be a baseline music clip that is harmonically compatible with the mix of the vocal clip and the piano clip. If the suggestion is adopted, a new partial mix is ​​formed. This process may be repeated each time with the new partial music mix adding music clips to or replacing music clips of the previous partial music mix until a satisfactory music mix is ​​found.

[0010] In addition to harmonic attributes, in some variations, the techniques herein rely on further musical attributes of the partial mix and candidate music clip when determining musical compatibility between the partial mix and the candidate music clip, such as rhythmic, spectral, or timbral attributes, such that compatibility determinations are not made solely based on the harmonic quality of the partial mix and the candidate music clip.

[0011] FIG. 1 illustrates a system for generating matched music mixes based on similarity, according to several variations. As indicated by the directional arrows labeled with numbers within circles, a music mix creation process is performed within the system. The labeled directional arrows represent data flow steps in the direction of the corresponding arrow from a personal electronic device 120 to a front end 102 of a music mixing service 100, or from the front end 102 of a music mixing service 100 to a personal electronic device 120 via one or more intermediate networks 130. Data may be transmitted over the network 130 using any suitable data communication network protocol, such as, for example, Internet Protocol (IP), Transmission Control Protocol (TCP), Hypertext Transfer Protocol (HTTP) (or its cryptographically secured variant HTTPS).

[0012] The computing environment of FIG. 1 is presented to explain an exemplary embodiment of the present invention. For purposes of discussion, the detailed description shows a specific example with respect to FIG. 1, and in the example, it is assumed that one computer system may communicate with another computer system, for example, a user electronic device (e.g., device 120) communicates with a remote computer system that provides at least one service (e.g., service 100). However, the present invention is not limited to any particular environment or device configuration. In particular, the device 120 / service 100 distinction is not required for the present invention, but is used to provide a framework for discussion. Rather, the present invention may be implemented in any type of system architecture or processing environment capable of supporting the inventive methodology presented herein, including a single device configuration. In any such configuration, data and information (e.g., music clips and pitch space vectors) may be exchanged between computing components according to a set of one or more application programming interfaces (APIs), which may be used within a single process (e.g., a procedure or function), or may be used between processes running on the same computing device (e.g., an inter-process API), or between processes running on different computing devices interconnected by a network (e.g., a network API).

[0013] Unless the context clearly indicates otherwise, the term "request" as used herein refers to a collection of one or more calls, invocations, or messages made, sent, or received via an API, and the term "response" refers to a collection of one or more calls, invocations, or messages made, sent, or received via an API that are triggered by a corresponding request. Additionally, references herein to a request or response received from an entity (e.g., a device) do not require the request or response to be received directly from the entity, and the request or response may pass through one or more intermediate entities before reaching the target entity. Similarly, references herein to a request or response sent to an entity (e.g., a device) do not require the request or response to be sent directly to the entity, and the request or response may pass through one or more intermediate entities on the way from the source entity.

[0014] 1, the techniques for generating a matched music mix based on similarity are implemented in a distributed computing environment in which client electronic devices (e.g., personal electronic device 120) interface with server electronic devices of a cloud-based service (e.g., music mixing service 100) via one or more data communication networks (e.g., intermediate network 130), while in some variations the techniques for generating a matched music mix based on similarity are performed by a single electronic device or by only a few electronic devices. For example, the techniques for generating a matched music mix based on similarity may be implemented on a smaller scale compared to cloud-based implementations by personal electronic devices such as a digital audio workstation (DAW) or a home or work personal computer.

[0015] The music mix creation process begins at step 1, where the electronic device 120 offers a selection of a stack template. A "stack" refers to a music clip generated in accordance with the techniques disclosed herein and may be comprised of a collection of multiple layered, synchronized, and musically compatible music clips. Thus, a stack is a music clip that may be comprised of other stacks or music clips.

[0016] In some variations, one or more of the music clips that make up a stack are included in each layer of the stack. For example, a layer of a stack may include drum music clips, bass music clips, guitar music clips, keys music clips, string music clips, vocal music clips, chord music clips, lead music clips, pad music clips, brass and woodwind music clips, synth music clips, sound effects clips, etc.

[0017] In some variations, the selected stack template may be one of a set of predefined stack templates selectable by the user 110 using a music mixing computer program or software application on the personal electronic device 120. For example, the set of predefined stack templates may be presented in a graphical user interface of the personal electronic device 120 for the user 110 to select one. The music mixing application may be a so-called mobile application, designed to run on the personal electronic device 120 and can be downloaded and installed using an application marketplace ("app store"), such as, for example, the GOOGLE PLAY STORE, the APPLE APP STORE, or the MICROSOFT STORE.

[0018] In some variations, the personal electronic device 120 is a portable electronic device, such as a smartphone or tablet electronic device. However, in some variations, the personal electronic device 120 is another type of electronic device. For example, the personal electronic device 120 may be a personal computer or a digital audio workstation (DAW). In some variations, the music mixing application is a mobile application, while in other variations, the music mixing application is a web browser-based application, or a thick client application, or a thin client application. The type of electronic device of the personal electronic device 120 is not required, nor is the type of application of the music mixing application. The user 110 and the personal electronic device 120 generally represent many possible different users and many possible different personal electronic devices having different possible types of music mixing applications installed that may simultaneously interface with the service 100 at any time.

[0019] In some variations, the selection of stack template received in step 1 indicates a genre, style, category, class, group, lineage, or type of music, etc. For example, the selected stack template may relate to one of dance, acoustic, random, ambient / drumless, lo-fi and hip-hop, trap / rap, etc. In response to the front end 102 receiving the selection of the stack template, the selection is provided to the back end 104 for further processing. In some variations, the back end 104 identifies a set of one or more predefined layers that make up the selected stack template. A "layer" refers to an individual musical part of a stack that can be configured by a user using the techniques disclosed herein. The set of predefined layers may vary for different stack templates that can be selected. For example, a dance stack template may include a drum layer, a key layer, a pad layer, a bass layer, and a synth layer, an acoustic stack template may include a drum layer, a pad layer, a bass layer, a lead layer, and a vocal layer, a random stack template may include a key layer, a bass layer, a string layer, and a drum layer, an ambient / drumless stack template may include a pad layer, a lead layer, a bass layer, a vocal layer, and a sound effect layer, a lo-fi and hip-hop layer may include a drum layer, a bass layer, a pad layer, and a vocal layer, and a trap / rap layer may include a drum layer, a key layer, a pad layer, a bass layer, and a synth layer. In the above examples, each stack template is composed of multiple predefined layers, but a stack may be composed of only one predefined layer. Furthermore, using the techniques disclosed herein, a user may add additional layers to a selected stack template and remove layers from a selected stack template.Thus, the selected stack template can be considered a starting point for the user to begin the music mix creation process, so that the user does not have to start from scratch, but instead can start from a pre-defined stack / mix, and the user can adjust the pre-defined stack / mix as needed using the techniques disclosed herein.

[0020] In some variations, the front end 102 provides the music mixing application of the personal electronic device 120 with access to the application programming interface (API) of the service 100 via an API endpoint of the front end 102. The API endpoint may be used by the personal electronic device 120 and other electronic devices to make requests for services and resources of the music mixing service 100 via the intermediate network 130. Such services and resources may include the ability to receive and respond to requests of steps 1, 3, and 5 shown in FIG. 1. When making requests for services or resources to the service 100 via the API endpoint, such as requests by the device 120 as in steps 1, 3, and 5, the API endpoint may be used along with a network protocol designation (e.g., HTTPS) in the Uniform Resource Indicator (URI). An example of an API endpoint includes a Domain Name Service (DNS) name of the front end 102.

[0021] In some variations, the API of the service 100 accessible through the API endpoint of the front end 102 conforms to a particular communication style. Potential styles that may be used include the Representational State Transfer (REST) ​​style or the Web Sockets style. The REST style is a stateless communication protocol that uses a request-response communication model. Thus, a new network connection (e.g., a Transmission Control Protocol (TCP) connection) may be established for each HTTP or HTTPS request. The Web Sockets style is a stateful communication protocol that allows full-duplex communication over a single network connection (e.g., a single TCP connection). The REST communication style is typically slower than the Web Sockets style in terms of sending network messages due to the overhead involved in establishing a network connection. However, the stateless nature of REST reduces memory and buffering requirements for transmitted data. Regardless of whether the front end 102 uses a REST style or a Web Socket style, data received and transmitted by the front end 102, such as data transmitted between the device 120 and the front end 102, may be encapsulated or formatted according to a data exchange format such as JavaScript Object Notation (JSON) or eXtensible Markup Language (XML).

[0022] In some variations, the music mixing service 100 itself, including the front end 102, back end 104, per-clip pitch interval space vector index 106, and sound library 108, typically conforms to or leverages a "cloud" computing model. The cloud computing model allows ubiquitous, convenient, on-demand network access to a shared pool of configurable resources such as networks, servers, storage applications, and services. The provider of the music mixing service 100 may offer its music mixing functionality to users according to a variety of different cloud computing models, including, for example, the Software-as-a-Service ("SaaS") model. In SaaS, when the music mixing service provider is a customer of a cloud infrastructure provider, the music mixing functionality is provided to users using the music mixing service provider's software application running on infrastructure provided by the cloud infrastructure provider. The application may be accessible from a variety of client devices, either through a thin-client interface, such as a web browser, or through an application programming interface. The infrastructure includes hardware resources, such as servers, storage, and network components, and the software deployed on the hardware infrastructure, that are necessary to support the music mixing functionality provided. Typically, in a SaaS model, the music mixing service provider does not manage or control the underlying infrastructure, including networks, servers, operating systems, storage, or individual application functionality, except for limited customer-specific application configuration settings.

[0023] The front end 102 and back end 104 typically represent a separation of concerns between a presentation layer of the music mixing service 100 and a data access / processing layer of the music mixing service 100. In some variations, the back end 104 implements an application programming interface (API) that is accessible by the electronic device 120 via the front end 102.

[0024] The sound library 108 includes a database of music clips. In some variations, the music clips are stored in the sound library 108 as digital audio signal sources, such as computer file system files or other data containers (e.g., computer database records) that include digital audio signal data. For example, the digital audio signal data included in the digital audio signal sources may represent a recording of a human performance or other aural performance, or may represent machine-generated music or sound. The digital audio signal data of the digital audio signal sources may be stored uncompressed, compressed in a lossless encoding format, or compressed in a lossy encoding format. With respect to the digital audio signal data of the digital audio signal sources, non-limiting examples of possible digital audio data formats, by their known file extensions, include .AAC, .AIFF, .AU, .DVF, .M4A, .M4P, .MP3, .OGG, .RAW, .WAV, and .WMA.

[0025] In some variations, the digital audio signal data of the music clips in the sound library 108 represents a loop. A loop is a repeatable section of audio material and may be created using a variety of music creation techniques, including but not limited to a microphone, a turntable, a digital sampler, a looper pedal, a synthesizer, a sequencer, a drum machine, a tape machine, a delay unit, programming using computer music software, and the like. A loop often includes a rhythmic pattern or notes or a chord sequence or progression that corresponds to a musical measure (e.g., 1, 2, 4, or 8 measures). Typically, a loop may be repeated indefinitely and still retain audible musical continuity. In some variations, the digital audio signal data of the music clips in the sound library 108 represents a track, stem, or mix in the form of a loop. The track, stem, or mix may be mono or stereo.

[0026] In some variations, library 108 includes hundreds, thousands, millions, or more music clips. For example, library 108 may be a collection of user, computer, or machine generated or recorded sounds, such as a music sample library provided by a cloud-based music creation and collaboration platform, such as the sound library available from SPLICE.COM of Santa Monica, Calif. and New York, NY.

[0027] While it is possible to apply the technique to a heterogeneous library 108 of music clips without distinguishing between different sound content categories of the music clips in the library 108, it may be advantageous to group the music clips into sound content categories. This may help to increase the efficiency of finding matching music clips within a particular sound content category, since a smaller number of candidate music clips in the library need to be considered (e.g., only candidate music clips that belong to a sound content category). This may also help to increase the accuracy of suggesting matching music clips, since music clips in the library that do not belong to a desired sound content category will not be suggested as suitable. For example, consider a library 108 that is divided into sound content categories based on instrument family. Such sound content categories may include vocals, strings, keyboards, woodwinds, brass, and percussion. In this case, a suggestion of a matching music clip may be made from within one of these sound content categories. Such suggestions require only music clips in the library 108 that belong to sound content categories to be considered, and not music clips that do not belong to the particular sound content category, thereby reducing the computational load of making the suggestions as fewer music clips need to be considered in the library 108. Furthermore, if a user desires suggestions of matching music clips in a particular sound content category, limiting the suggestions to only music clips in the sound content category can ensure that music clips in the desired sound content category are suggested.

[0028] In some variations, the different sound content categories grouped into audio tracks of the library 108 may reflect categorical differences in the statistical distribution of the underlying digital audio signals within the different sound content categories. In this way, the sound content categories may correspond to classes or types of statistical distributions. The top-level sound content categories may be further subdivided based on instruments, types of instruments, genres, moods, or other sound attributes suitable for the requirements of the current implementation to form a hierarchy of sound content categories. As an example, the hierarchy of sound content categories may include top-level sound content categories for loops and one-shots. Each of these top-level sound content categories may then include, at a second hierarchical level, a drums category and an instrument category. Each instrument category may include vocals and instruments other than drums. Each instrument category may be further subdivided at a third hierarchical level into instrument families (e.g., vocals, strings, keyboards, woodwinds, and brass sound content categories).

[0029] The above is only one non-limiting example of a possible sound content category hierarchy into which the library 108 of music clips may be categorized. Other categories are possible, and the technique is not limited to any category or category set or category hierarchy. Furthermore, the sound content categories may be heuristically or empirically selected according to the requirements of the current implementation, including based on different sound categories expected or found within the library 108, although the sound content categories may be learned or calculated according to a computer-implemented unsupervised clustering algorithm (e.g., an exclusive, overlapping, hierarchical, or probabilistic clustering algorithm).

[0030] For example, music clips in the library 108 may be grouped (clustered) into different clusters corresponding to sound content categories based on the similarity between one or more attributes extracted or detected from the digital audio signal data of the music clips. Such sound attributes on which the music clips may be clustered may include, for example, one or more of the following: statistical distribution of signal amplitude over time, zero crossing rate, spectral centroid, spectral density of the signal data, spectral bandwidth of the signal data, spectral flatness of the signal data, or harmonic attributes of the signal data. When clustering is performed, music clips that are more similar with respect to one or more of these sound attributes will naturally be more likely to be clustered together in the same cluster, and music clips that are less similar with respect to one or more of these sound attributes will naturally be less likely to be clustered together in the same cluster. It should be noted that while a music clip in the library can belong to only one sound content category, a music clip may belong to multiple sound content categories, for example, when an overlapping clustering algorithm is used to identify the sound content categories.

[0031] In some variations, music clips in library 108 are indexed in index 106 by the sound content category to which they belong or are assigned. In this manner, music clips in library 108 that belong to a particular sound content category can be efficiently identified using index 106. In some variations, a search for matching music clips is limited using index 106 to only music clips that belong to a specified or predetermined set of one or more sound content categories. For example, index 106 may be used to perform a matching music clip search in which the search space (the set of candidate music clips considered) is limited to only guitar music clips in library 108.

[0032] In some variations, the pitch interval space vector index per clip 106 indexes the music clips in the library 108 by the pitch interval space vectors per clip generated from the music clips. In some variations, the pitch interval space vectors per clip for a music clip are generated from a collection of pitch interval space vectors per beat generated for the music clip. The pitch interval space vectors per clip may represent the bars (e.g., 2, 4, 6, 8, 10, 12, 16, etc.) of the music clip consisting of a number of beats per bar (e.g., 1, 2, 4, 8, 16, etc.). For example, a pitch interval space vector per clip representing an 8 bar music clip with 4 beats per bar is generated from a pitch interval space vector of 32 beats. The dimensionality of the pitch interval space vector per beat is the number of pitch classes (e.g., 12) in some variations. A pitch class is a group of pitches related by octave and enharmonic equivalence. A pitch is a distinct sound having a distinct frequency. For example, the number of pitch classes may be 12, and each element of the beat-by-beat pitch interval space vector corresponds to one of the 12 pitch interval spaces, e.g., {element 0: pitch class C, 1: C#, 2: D, 3: D#, 4: E, 5: F, 6: F#, 7: G, 8: G#, 9: A, 10: A#, 11: B}.

[0033] In some variations, the pitch interval space represents human perception of pitch, chord, and key as well as principles of music theory as distances. The multi-level pitch configuration is represented by a 12-dimensional vector in the pitch interval space. In some variations, the multi-level pitch configuration is represented in the pitch interval space by a pitch interval space vector T(k), which is calculated as the discrete Fourier transform (DFT) of the pitch class distribution or chroma vector input c(n) as follows:

number

[0034] In the above formula, the following is included:

number

[0035] In some variations, the variable N is 12 and represents the dimension of the input chroma vector. The variable w(k) represents a weight derived from an empirical evaluation of dyadic consonance and is used to adjust the contribution of each dimension k of the pitch interval space. In some variations, for audio input, w(k) is in the set {3, 8, 11.5, 15, 14.5, 7.5}. In some variations, for symbolic input, w(k) is in the set {2, 11, 17, 16, 19, 7}. Because the remaining coefficients are symmetric, the variable k can range from 1 to 6 (or 0 to 5) (but need not be 1 to 12 (or 0 to 11)).

[0036] In some variations, the formula for T(k) is expressed as follows:

number

[0037] In some variations, the pitch interval space has musical properties including perceptual proximity: that is, an algebraic objective measure captures the perceptual characteristics of a set of pitches represented by pitch interval space vectors in the pitch interval space. In particular, the Euclidean and cosine distances between multi-level pitch configurations are consistent with human perception of pitches, chords, and keys, as well as principles of tonal Western music theory.

[0038] In some variants, pitch interval space also has the property of transposition invariance; that is, transposing a pitch configuration in pitch interval space by a semitone corresponds to a rotation of T(k). Thus, any transposition of a pitch interval space vector results in a vector of the same magnitude or distance from the center. This property is an important feature of Western tonal music resulting from 12-tone equal-tempered tuning, in the sense that it corresponds to Western listeners' perception of interval relationships in different regions as being similar. For example, the interval from C to G in the key of C major and the interval from C# to G# in the key of C# major are perceived as equivalent.

[0039] In some variations, the harmonic compatibility between two music clips is measured according to a computationally efficient algebraic distance or similarity metric. The distance or similarity metric is calculated using per-clip pitch interval space vectors representing the two music clips. In some variations, the distance or similarity metric is calculated as a sum of beat-wise pairwise cosine or Euclidean distances. Here, cosine distance refers to the complement of cosine similarity (e.g., 1-cosine similarity) and not angular distance (e.g., inverse cosine (cosine similarity)).

[0040] For example, consider two per-clip pitch interval space vectors generated for two music clips, each composed of elements of k per-beat pitch interval space vectors generated for the two music clips. For example, k may be 32, which corresponds to 8 musical bars with 4 beats per bar. In this case, each per-clip pitch interval space vector has 384 elements, since there are 32 12-element per-beat pitch interval space vectors. The harmonic compatibility of the two music clips MC1 and MC2 may then be calculated as follows:

number

[0041] In the above formula, bwV 1,k represents the beat-unit pitch interval space vector of the kth beat in one of the clip-unit pitch interval space vectors, and bwV 2,k represents the k-th beat-by-beat pitch interval space vector in the other of the two clip-by-clip pitch interval space vectors. In the above equation, the function d() represents an algebraic distance metric, such as cosine distance or Euclidean distance, applied to the two beat-by-beat pitch interval space vectors. In some variations, each beat-by-beat pitch interval space vector is normalized (e.g., L2 normalized) when used to calculate the algebraic distance metric. In some variations, the larger the value of the harmonic compatibility -1(MC1, MC2), the lower the harmonic compatibility between music clips MC1 and MC2 (the longer the distance between the music clips in pitch interval space). Also, the smaller the value of the harmonic compatibility -1(MC1, MC2), the higher the harmonic compatibility between music clips MC1 and MC2 (the shorter the distance between the music clips in pitch interval space).

[0042] In some variations, by exploiting the equivalence of Euclidean and cosine distance metrics, only a single algebraic distance calculation is required to calculate harmonic compatibility between music clips, without the sum of beat-level partial distance calculations. To do this, each per-beat pitch interval space vector is individually normalized with its L2 norm. A single algebraic distance calculation is then applied to the per-clip pitch interval space vector composed of the L2-normalized per-beat pitch interval space vectors as follows:

number

[0043] where cwV1 is the per-clip pitch interval space vector of music clip MC1 and cwV2 is the per-clip pitch interval space vector of music clip MC2. Each per-beat pitch interval space vector of cwV1 and each per-beat pitch interval space vector of cwV2 are normalized with their respective L2 (Euclidean) norms, which makes the sum of the per-beat cosine distances equivalent to a single Euclidean distance calculation at the clip level. In this way, harmonic compatibility between music clips can be determined in a scalable manner (e.g., extending to millions of music clips) using an approximate nearest neighbor algorithm.

[0044] In some variations, generating a per-clip pitch interval space vector for the music clip includes periodic short-term interval detection and spectral analysis performed on the digital audio signal data of the music clip. Interval detection identifies musical beats within the music clip. In some variations, up to a predetermined number of beats within the music clip are identified. For example, the predetermined number may be 32, representing 8 musical bars with 4 beats per bar. However, the predetermined number of beats is not required.

[0045] Various digital audio signal data processing techniques may be used to identify musical beats within a music clip audio signal. For example, techniques may identify note onsets in the energy or spectrum of the signal data, and then analyze the pattern of onsets to detect repeating patterns or quasi-periodic pulse trains. For example, beat tracking and bar detection methods may be used.

[0046] In some variations, a chroma representation is extracted from the digital audio signal data of the music clip at the beats identified by the interval detection by the spectral analysis of the per-clip pitch interval space vector generation. In some variations, the chroma representation of the beat is a 12-element vector ("chroma vector"), with each element corresponding to one of the 12 pitch classes of the equal-tempered chromatic scale. The value of an element in the chroma vector of the beat numerically indicates the prominence of the corresponding pitch class at the beat in the signal data. The chroma vector may be calculated by applying a filter bank to the time-frequency representation of the digital audio signal data. For example, the time-frequency representation may be obtained from either a short-time Fourier transform (STFT) or a constant-Q transform (CQT), the latter providing finer frequency resolution at low frequencies.

[0047] In some variations, the pitch interval space vectors per beat that constitute the pitch interval space vector per clip generated for the music clip are generated from the chroma vectors per beat. Specifically, the pitch interval vector per beat for a given beat of the music clip can be calculated as an L1-normalized discrete Fourier transform (DFT) of the chroma vector per beat generated for the beat, as in the formula for T(k) presented above. This can be performed for each chroma vector per beat to generate a set of pitch interval space vectors per beat that constitute the pitch interval space vector per clip for the music clip.

[0048] In some variations, a beats per minute (BPM) independent indexable feature space is provided for determining harmonic compatibility between clips. The indexable feature space uses a flat vector representation of the music clip of shape (1,N), which normalizes the duration of the clip with respect to a BPM independent measure. In some variations, the BPM independent measure is a predetermined number of bars and a predetermined number of beats per bar. In some variations, the flat vector representation is a BPM independent per-clip pitch interval space vector representation of the music clip.

[0049] 2 illustrates a method for generating a BPM-independent per-clip pitch interval space vector for a music clip according to some variations. Some or all of the operations 200 (or other processes described herein, or variations, or combinations thereof) are implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) performed under the control of one or more computer systems configured with executable instructions and collectively executed on one or more processors. The code is stored on a computer-readable storage medium, for example, in the form of a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) of the operations 200 are performed by the backend 104 of the music mixing service 100 of other figures.

[0050] At operation 202, a loopable music clip is obtained. For example, the loopable music clip may be obtained from the sound library 108. The loopable music clip has a predetermined number of musical bars and a predetermined number of beats per bar. For example, the predetermined number of bars may range from 2 bars to 16 bars, and the predetermined number of beats per bar may range from 2 bars to 8 bars. The loopable music clip may be any type of pitch-based music clip. For example, the loopable music clip may correspond to any of the above stack layers or sound content categories, such as bass, guitar, keys, strings, vocals, chords, leads, pads, brass and woodwinds, synths, sound effects, etc.

[0051] At operation 204, a constant-Q transform (CQT) of the music clip is calculated using 12 bins per octave. The output of this calculation may be a short-time Fourier transform (STFT)-like representation, with a resolution on the frequency axis corresponding to the resolution of a musical scale (e.g. the resulting frequency bins may be considered as piano notes). The number of frames may be specified by the time length of the clip and a window parameter of the STFT-like CQT calculation.

[0052] 3 shows a plot of the CQT matrix for an example music clip. One dimension of the matrix (x-axis / columns) represents frames and the other dimension (y-axis / rows) represents frequency. In this non-limiting example, there are 1200 frames.

[0053] At operation 206, a chroma saliency map is computed from the CQT. The chroma saliency map may represent the music clip in a manner that reveals the distribution of pitch classes in a chromatic scale. In other words, the chroma saliency map may represent the music clip in a manner that reveals the contribution or presence of specific notes or intervals in the chromatic scale. The CQT may span multiple octaves. In the computed chroma saliency map, each octave may be collapsed into a single bin, generating a matrix with 12 rows and N columns, the number of notes in the chromatic scale. The number of frames, N, may be kept the same as the CQT.

[0054] Figure 4 plots the chroma saliency map of the example music clip shown in Figure 3. One dimension of the map (x-axis / columns) represents the 1200 frames, and the other dimension of the map (y-axis / rows) represents the 12 pitch classes of the chromatic scale. Chroma values ​​are normalized to the range 0.0 to 1.0 within the map.

[0055] In some variants, as represented by operations 204 and 206, the chroma saliency map is calculated from the CQT according to a deterministic transformation. However, other deterministic or non-deterministic methods may be used to generate the chroma saliency map. For example, the chroma saliency map may be generated based on a machine learning model (e.g., an artificial neural network model) trained to generate chroma saliency maps from time-domain music clips or from their intermediate representations (e.g., their CQT representations). Thus, operations 204 and 206 should be seen as just one possible way to generate a chroma saliency map of a music clip. However, other methods may be used. For example, the chroma saliency map may be calculated based on a perceptual heuristic. For example, the heuristic may reflect that some pitch classes may be auditorily inperceptible due to masking effects, and therefore should not be represented in the chroma saliency map even if the pitch classes show quantitatively high energy. As an alternative to generating a chroma saliency map from a CQT, the chroma saliency map may be generated from a short-time Fourier transform (STFT) or other frequency-domain representation of the music clip. The chroma saliency map may also be generated from a time-domain representation of the music clip.

[0056] In some variations, the chroma saliency map includes a chromagram representation that includes a sequence of 12-dimensional vectors over time of the music clip, each vector corresponding to a frame of the chromagram representation and encoding the short-term energy distribution of the music clip for the frame over the 12 chroma subbands.

[0057] At operation 208, a BPM-independent chroma representation of the chroma saliency map is formed. To make the chroma saliency map BPM-independent, N chroma frames are aggregated (e.g., summed or averaged) to a beat-level resolution. For example, consider a music clip that is 8 bars long and in 4 / 4 time, the number of beats in the music clip is 32. Further, in this example, assume that the number of chroma frames, N, is 1200. Thus, in this example, 32 chunks of approximately 37.5 chroma frames are aggregated beat-by-beat to generate a 12-by-32 BPM-independent chroma representation matrix composed of one 12-dimensional chroma vector for each of the 32 beats.

[0058] Figure 5 shows a BPM-independent chroma representation matrix generated by aggregating beat-wise chroma vectors of the chroma saliency map matrix plotted in Figure 4. As shown, the 1200 chroma frames of the chroma saliency map of an exemplary music clip are aggregated beat-wise to 32 beats. One dimension of the matrix (x-axis / columns) represents the 32 beats, and the other dimension of the matrix (y-axis / rows) represents the 12 pitch classes of the chromatic scale. The chroma values ​​in the matrix are normalized to the range 0.0 to 1.0.

[0059] In operation 210, the real and imaginary components of a set of beat-wise pitch interval space vectors are calculated from the BPM-independent chroma representation (e.g., a 12-row, 32-column chroma representation matrix). This involves a Fourier transform of the real signal. For example, each 12-element column (e.g., each chroma vector) of the 12-row, 32-column chroma representation matrix can be considered as a time-domain signal. As a result, a Fourier transform of the signal produces a complex vector of 12 real and 12 imaginary values. Because each chroma vector is a real signal, the Fourier transform is symmetric and thus only the first half of the coefficients need to be retained, resulting in 6 real and 6 imaginary values ​​that make up the real and imaginary components of the 12-element beat-wise pitch interval space vector. The result of operation 210 can be two 6-row, M-column matrices composed of the real and imaginary components of the M beat-wise pitch interval space vectors, with the M columns of each matrix containing the real or imaginary components of the M beat-wise pitch interval space vectors. M represents the number of beats. For example, M can be 2, 4, 8, 16, 32, or 64 beats, or any other number of beats suitable to the requirements of the particular implementation at hand.

[0060] Figure 6 shows plots of two matrices containing the real and imaginary components of the 32 beat-per-beat pitch interval space vectors generated from the BPM-independent chroma representation matrix of Figure 5. One dimension of the matrix (x-axis / columns) represents the 32 beats, and the other dimension (y-axis / rows) represents the 6 real and 6 imaginary values ​​that make up the 32 beat-per-beat pitch interval space vectors.

[0061] Also, at operation 210, the real and imaginary components of each per-beat pitch interval space vector are concatenated to form a single 12-row, M-column matrix containing M per-beat pitch interval space vectors.

[0062] FIG. 7 shows the result of concatenating the real and imaginary components of the two matrices of FIG. 6 to generate a single matrix containing 32 pitch interval space vectors per beat, with each column of the matrix containing the pitch interval space vector per beat for each beat of the exemplary music clip.

[0063] At operation 212, the matrix of M pitch interval space vectors per beat is flattened into a pitch interval space vector per clip of shape (1, (12*M)). For example, if M is 32 beats, then the pitch interval space vector per clip has dimensions (1, 384). In some variations, the matrix is ​​flattened column-wise by concatenating the real and imaginary parts of each pitch interval space vector per beat to generate the pitch interval space vector per clip.

[0064] In some variations, each of the beat-by-beat pitch interval space vectors that make up the clip-by-clip pitch interval space vector is normalized by its L2 norm (also known as the 2-norm or Euclidean norm) before concatenating them to form the clip-by-clip pitch interval space vector. This normalization may be performed to solve the problem of identifying harmonically compatible music clips in a scalable manner with low latency, where a two-dimensional feature space representation provided by a matrix of M beat-by-beat pitch interval space vectors cannot be easily indexed with an index (e.g., an index that supports approximate nearest neighbor search), by exploiting the equivalence (proportionality) of unit vector Euclidean distance and unit vector cosine distance.

[0065] Figure 8 shows the flattening of the matrix of Figure 7 into a per-clip pitch interval space vector for an exemplary music clip in a color-coded plot in a computer graphical user interface, where the matrix is ​​flattened into a per-clip pitch interval space vector having 384 elements, including 12 elements for each of the 32 per-beat pitch interval space vectors. Figure 9 shows the values ​​of the 384-element per-clip pitch interval space vector in a waveform plot.

[0066] The feature space of the music clip is represented by a 12-row, M-column matrix containing M per-beat pitch interval space vectors. In operation 212, the matrix is ​​flattened, making the feature space indexable and the music clips searchable in a scalable manner. Flattening the two-dimensional matrix of M per-beat pitch interval space vectors into a one-dimensional per-clip pitch interval space vector as in operation 212 allows for the use of an approximate nearest neighbor search algorithm to rapidly identify harmonically compatible music clips, allowing the approximate nearest neighbor search, which typically only supports one-dimensional vectors, to be scaled to millions of indexed music clips. The formula for harmonic compatibility-2(MC1, MC2) discussed above represents how the harmonic compatibility between two music clips can be efficiently calculated using their respective per-clip pitch interval space vectors.

[0067] In operation 214, key-independent support of music clips is provided. Being key-independent allows for determining harmonic compatibility between clips of different music keys. Returning to the chroma representation, circularly shifting one element of a column as a time-domain signal is equivalent to transposing the original signal by one semitone. This property that a time shift in the time domain is equivalent to a phase rotation in the frequency domain allows for generating transpositions of music clips using rotations directly in pitch interval space. For example, music clips can be indexed so that they can be matched for harmonic compatibility across the 12 keys of the chromatic scale. To do this, the original per-clip pitch interval space vector generated in operation 212 can be rotated in 11 ways, which generates a total of 12 per-clip pitch interval space vectors including the original per-clip pitch interval space vector. Music clips can then be indexed with each of these vectors at index 106 to allow them to be matched across different keys. If a music clip in one key matches another music clip in a different key in terms of harmonic compatibility, then one of the music clips may be pitch-shifted using digital audio signal data processing techniques so that both clips are in the same key. In some variations, support is provided for only a few semitones (e.g., 3 semitones) above and below the musical key of the original, non-pitch-shifted music clip. This reduces the number of per-clip pitch interval space vectors into which the clips are indexed in the index 106, and thus the size of the index 106. Furthermore, this may prevent noticeable degradation in perceptual quality that may result from too large a pitch shift of the original music clip (e.g., more than 3 semitones above or below the chromatic scale).

[0068] At operation 216, the music clips are indexed with the generated per-clip pitch interval space vectors. In some variations, the music clips are indexed with the generated per-clip pitch interval space vectors at index 106 using an approximate nearest neighbor based index (e.g., a quantization-based index, a graph-based index, or a tree-based index) that supports an approximate nearest neighbor search. For example, a graph-based approach or a space-partitioned approximate nearest neighbor approach may be used. The approximate nearest neighbor approach may provide an acceptable tradeoff between performance (e.g., quickly identifying a set of one or more music clips that are close to a given music clip in pitch interval space), scalability (e.g., indexing a large number of music clips), and precision (e.g., query recall and accuracy).

[0069] In some variations, the backend 104 queries the index 106 using the "source" per-clip pitch interval space vector to identify an "answer" set of one or more music clips in the library 108 indexed by the per-clip pitch interval space vector, each of which has a close distance or similarity in pitch interval space to the source per-clip pitch interval space vector by an algebraic distance or similarity measure such as cosine distance or Euclidean distance. If an approximate nearest neighbor search is used, the answer set may not be the closest indexed music clips (but may be), depending on the closeness of the search. The number of music clips included in the answer set may be a predetermined number (e.g., a predetermined number of closest music clips in pitch interval space). Alternatively, the answer set may include all music clips that are within a predetermined threshold distance or similarity of the source music clip.

[0070] In some variations, the query also specifies a set of one or more query constraints that limit the set of indexed music clips included in the answer set. These constraints may be applied when collecting the answer set (e.g., using an approximate nearest neighbor technique) or may be applied as a post-search step to be applied to an initial answer set obtained from the search (e.g., after the initial answer set has been identified using an approximate nearest neighbor search using the source per-clip pitch interval space vectors as search keys). Multiple constraints may be applied concatenated; that is, if multiple constraints are specified, a music clip to be included in the answer set must satisfy all constraints. However, constraints may also be applied in isolation or using Boolean logic (e.g., expression of constraints using AND, OR, NOT, or precedence operators).

[0071] One constraint already mentioned is sound content category. For example, the answer set may be limited to music clips that belong to at least one of a set of one or more specified sound content categories. For example, the specified sound content categories may include drums, bass, guitar, keys, strings, vocals, chords, leads, pads, brass and woodwinds, synths, sound effects, etc., all of these sound content categories, a subset of these categories, or a superset of these.

[0072] Another constraint may be beats per minute (BPM). This constraint does not affect the BPM independence of the generated per-clip pitch interval space vector. However, to avoid stretching the music clips in a mix in time using a time-scaling algorithm that does not change the pitch of the music clips (e.g., the Waveform Similarity Overlap-Add (WSOLA) time-scaling algorithm), which would result in a noticeable degradation of the perceptual quality of the mix, a user may wish to limit the answer set to music clips that have a particular BPM or are within a particular BPM range as part of the mixing process. In some variations, the music clips in the library 108 are logically categorized by the index 106 into a set of non-overlapping BPM buckets, and the query specifies one of the buckets to which to restrict the search for matching music clips. For example, there may be three BPM buckets corresponding to low BPM, medium BPM, and high BPM. For example, the low BPM bucket may include music clips in library 108 that have a BPM below 100 BPM, the medium BPM bucket may include music clips in library 108 that have a BPM between 100 BPM and 150 BPM, and the high BPM bucket may include music clips in library 108 that have a BPM above 150 BPM.

[0073] Another possible constraint is the key of the music. This constraint does not affect the key independence of the generated per-clip pitch interval space vectors. However, as with BPM, a user may wish to restrict the answer set to music clips in a particular key or set of keys as part of the mixing process to avoid pitch-shifting music clips in a mix that would result in a noticeable degradation of the perceptual quality of the mix. In some variations, the query specifies a set of one or more pitch classes of the 12 pitch classes in the chromatic scale that restricts the answer set of matching music clips.

[0074] Another possible constraint is a chord progression or scale degree progression over multiple measures. For example, as part of the mixing process, a user may wish to limit the answer set to music clips that follow a specified chord progression (e.g., specified as a sequence of note names and corresponding measures) or a specified scale degree progression (e.g., specified as a sequence of scale degrees and corresponding measures). For example, a chord progression over four musical measures may be specified as Bm in the first measure, D in the second measure, Em in the third measure, and G followed by A in the fourth measure. Instead of a specified chord progression, a scale degree progression may be specified. For example, a scale degree progression over four musical measures may be the first degree (tonic) in the first musical measure, the third degree (middle) in the second musical measure, the fourth degree (subdominant) in the third musical measure, and the sixth degree (lower middle) followed by the seventh degree (leading) in the fourth musical measure. The chord progressions and scale degree progressions of the music clips in the library 108 may be identified using digital audio signal data processing techniques. In some variations, if the music clip includes a designated chord progression or designated scale degree progression according to the digital audio signal data processing techniques, the music clip fills the chord progression or scale degree progression.

[0075] Returning now to step 2 of FIG. 1, the service 100 returns a selected stack template pre-populated with a collection of one or more music clips selected from the library 108. The collection of one or more music clips that the service 100 selects to include in the stack template may be constrained by the genre / style of the selected stack template. For example, if the selected stack template is for a genre / style of "dance," then all of the music clips in the collection that the service 100 selects to include in the stack template may belong to the "dance" sound content category or may be indexed, tagged, or categorized by the service 100 as "dance" music clips.

[0076] In some variations, drum music clips or other non-pitched music clips in library 108 are not considered as candidates for determining harmonic compatibility between music clips. This is because drums and other percussion instruments that are played by hitting, shaking, or scraping (e.g., snare drums, bass drums, cymbals, tambourines, triangles, etc.) are typically considered non-pitched percussion instruments that produce weak fundamental frequencies. However, some percussion instruments, such as timpani and pitched toms, may have pitched characteristics. Thus, there may not be a clear line between pitched and non-pitched music clips in library 108. Digital audio signal data processing techniques may be applied to the music clips in library 108 to determine which music clips are sufficiently pitched (e.g., have a detectable fundamental frequency) and which are not pitched (e.g., have a weak fundamental frequency). The determination of pitched and non-pitched music clips in library 108 may be made by a user as an alternative to or in combination with the automatic determination (e.g., by confirming an initial automatic determination).

[0077] In some variations, instead of selecting a template to initiate the stack creation process, the user may select a single "seed" music clip to initiate the stack creation process. For example, the user may select a seed music clip from the library 108, for example, by browsing or searching the library 108. Alternatively, the user may record a music clip. For example, the user may use the electronic device 120 to record two, four, eight, or more musical bars. For example, the user may sing an eight-bar melody or play an instrument for eight bars, which is captured as a music clip at the electronic device 120 via a microphone of the electronic device 120 or via a microphone operably connected to the electronic device 120. In some variations, the per-clip pitch interval space vector of the recorded music clip is calculated at the electronic device 120 using techniques disclosed herein. Alternatively, the recorded music clip may be uploaded to the service 100, and the per-clip pitch interval space vector may be calculated by the service 100. The calculated per-clip pitch interval space vector may then be returned by the service 100 to the device 120 for use by the device 120.

[0078] Music clips recorded by device 120 may also be added to an existing stack that is in the process of being created. For example, a user may begin the stack creation process by selecting a stack template of one or more music clips. The user may add the recorded music clip to the current stack. For example, the stack template may begin a stack with a keys music clip, a drum music clip, and a guitar music clip. The user may then use device 120 to record a vocal melody that the user blends with the current stack. The user may then add the recorded music clip to the current stack to form a new stack that includes the keys music clip, the drum music clip, the guitar music clip, and the recorded vocal music clip. It should be noted that the recorded vocal music clip may be included in a stack by the user regardless of the similarity or distance in pitch interval space between the recorded vocal track and other pitch-based music clips in the stack. However, subsequent music clips selected from the library 108 to add to a stack, or to replace music clips in a stack, may be selected based on harmonic compatibility of the music clip with the recorded vocal music clip, by similarity or distance in pitch interval space. For example, after adding a recorded vocal music clip to a stack, a user may choose to replace the key music clip provided by the stack template with another key music clip that is harmonically compatible. The selection of the new key music clip may be based on harmonic compatibility of the new key music clip with the remaining pitch-based music clips in the stack, including the guitar music clip and the recorded vocal music clip.

[0079] Similar to starting a stack with a user recorded music clip or adding a user recorded music clip to a stack, a stack may be started with a music clip licensed by a recording artist or a music clip licensed by a recording artist may be added to an existing stack. For example, consider a situation in a music mixing contest where contestants use the stack application disclosed herein and a winner is selected based on the mix determined to be the best sounding, and the mix must include at least one music clip provided / licensed by a recording artist who is sponsoring or supporting the contest. In this example, each contestant may start the contest using a stack that includes a licensed music clip (e.g., a vocal melody sung by the recording artist) as a seed music clip.

[0080] Thus, the stack creation process can be initiated in ways other than by selecting a template, such as by selecting or recording a seed music clip.

[0081] At step 3, a matching music clip request is received from the electronic device 120. For example, assume that the stack template returned at step 2 includes one "vocal" music clip, or the seed music clip recorded, selected, or uploaded is a "vocal" music clip. Next, at step 3, a matching "key" music clip request is received. Upon receiving this matching "key" music clip request, the service 100 may identify a matching (e.g., best matching) "key" music clip using the per-clip pitch interval space vector of the "vocal" music clip in a query to the index 106. For example, the identification may be made based on an approximate nearest neighbor search using the per-clip pitch interval space vector of the "vocal" music clip in a query. At step 4, the matching music clip is returned to the electronic device 120 in response to the request at step 3. For example, the best matching "key" music clip is returned to the electronic device 120 and included in the current stack with the "vocal" music clip at the electronic device 120.

[0082] Steps 3 and 4 may be repeated iteratively until the user 110 determines a final stack. For example, after the matching "key" music clip request, at step 3, another matching music clip request may be received from the electronic device 120. This request may be for a "lead" music clip that matches a current harmonically matched partial mix ("partial mix") composed of a matching "vocal" music clip and a matching "key" music clip. Because the current partial mix includes multiple music clips, the individual per-clip pitch interval space vectors of the constituent music clips may be linearly combined (e.g., by simple linear addition) to form a "per-partial-mix" pitch interval space vector that represents the current partial mix. In some variations, the initial partial-mix-per-pitch interval space vector formed based on the linear combination of the constituent per-clip pitch interval space vectors is normalized at the beat level by L2 normalization to form a final partial-mix-per-pitch interval space vector. The service 100 may then use the partial-mix-per-pitch interval space vector to query the index 106 to identify a matching "lead" music clip. For example, the identification may be performed using partial-mix-wise pitch interval space vectors of matching "vocal" and "key" music clips based on an approximate nearest neighbor search for the query. In step 4, the matching music clips are returned to electronic device 120 in response to the request of step 3. For example, the "lead" music clip that best matches the current partial mix composed of the "vocal" and "key" music clips may be returned to electronic device 120 to form a new partial mix composed of the matching "vocal" music clips, the matching "key" music clips, and the matching "lead" music clips.

[0083] Next, the user 110 may wish to add a drum music clip to the current stack. Thus, at step 3, another matching music clip request may be received from the electronic device 120. This request may be for a "drums" music clip that matches the current stack, which is comprised of "vocal" music clips, "key" music clips, and "lead" music clips. However, because the "drums" music clip may be considered an unpitched music clip, the matching "drums" music clip may be selected by the service 100 using techniques other than the harmonic compatibility techniques disclosed herein. For example, the service 100 may randomly select a matching "drums" music clip from the library 108 according to a genre / style constraint (e.g., that of the stack template selected in step 1) or other user-specified or user-set constraints (e.g., BPM). While the unpitched music clip may be selected randomly according to the constraints, the unpitched music clip may also be selected differently according to the constraints. For example, the matching unpitched music clip may be selected according to the constraints based on the compatibility of the detected onset patterns in the unpitched music clip with the music clips that make up the current stack.

[0084] Next, the user 110 may wish to add a matching "bass" music clip after adding the "drums" music clip to the current stack. Thus, at step 3, yet another matching music clip request may be received by the service 100. This request may be for a matching "bass" music clip that matches a current partial mix composed of a matching "vocal", "key" and "lead" music clip (note that for purposes of determining harmonic compatibility, the "drums" music clip and other non-pitched music clips may be excluded from the partial mix). The service 100 may form a partial mix-per-pitch interval space vector by linearly combining the per-clip pitch interval space vectors of the matching "vocal", "key" and "lead" music clips that make up the current partial mix, and subsequently L2-normalize the partial mix-per-pitch interval space vector at the beat level. The service 100 may then use the partial mix-per-pitch interval space vector to query the index 106 to identify a matching "bass" music clip. For example, the identification may be performed using a partial mix unit pitch interval space vector for the query based on an approximate nearest neighbor search. In step 4, the matching music clips are returned to the electronic device 120 in response to the request of step 3. For example, the "bass" music clip that best matches the current partial mix composed of the matching "vocal" music clip, the matching "key" music clip, and the matching "lead" music clip may be returned to the electronic device 120 to form a new partial mix composed of the matching "vocal" music clip, the matching "key" music clip, the matching "lead" music clip, and the matching "bass" music clip, and a new current stack composed of the new partial music mix and the "drums" music clip.

[0085] At step 5, a request may be received by the service 100 from the electronic device 120 to share the current stack as a finished mix. For example, the current stack may be rendered as a music clip on the electronic device 120 or the service 100 and included in the library 108, stored as a digital audio signal data source on the electronic device 120, or uploaded or shared to an online social media platform (e.g., TIKTOK social networking service owned by BYTEDANCE of Beijing, China), sent as an attachment to an electronic mail (email) message or text (SMS) message, uploaded to a cloud-based data storage service or a centrally hosted network file system, or exported in a data format that can be imported into digital audio workstation (DAW) software for further processing.

[0086] 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22 show various states of a graphical user interface of a stack-based music mixing application according to several variations. The techniques for determining harmonic compatibility between pitch-based music clips described herein may be used to support a stack-based music mixing application. Note that although the following describes a user electronic device performing certain operations and the music mixing service 100 performing other operations, the distribution of operations performed need not be strictly as described. For example, some or all of the operations described as being performed by the service 100 may instead be performed by the user electronic device. Furthermore, although the stack-based music mixing application is described as a mobile application on a mobile computing device, the stack-based music mixing application may take other forms and run on other types of computing devices. For example, the stack-based music mixing application may be included in digital audio workstation software running on a workstation computer or laptop computer.

[0087] Additionally, variations of the stack-based music mixing application are possible. For example, in one variation, a user may select a collection of one or more 8-bar music clips in a digital audio workstation application running on the user's electronic device. A plug-in or extension of the digital audio workstation application may interface with the service 100 over the network 130 to obtain music clips or collections of music clips in the library 108 that are harmonically compatible with the selected collection of music clips. In this instance, the selected collection of music clips may or may not be present in the library 108 or indexed by the index 106. The digital audio workstation software or a plug-in or extension thereto may generate a per-clip pitch interval space vector for the selected collection of music clips using techniques disclosed herein and transmit the generated per-clip pitch interval space vector over the network 130 to the service 100, which uses the per-clip pitch interval space vector to search for harmonically compatible music clips using techniques disclosed herein.

[0088] FIG. 10 illustrates a personal electronic device 1000 (e.g., device 120 of FIG. 1) having a graphical user interface (GUI) 1002. The GUI 1002 presents options 1006 for selecting a stack template, indicated in a text banner 1004. The collection of options 1006 corresponds to different musical genres / styles. A user may select one of these to begin the mix creation process. As previously mentioned, a stack-based mixing application may support other ways of beginning the mix creation process besides selecting a stack template. For example, the GUI 1002 may provide graphical user interface controls for selecting a seed music clip from the library 108 (e.g., by searching or browsing the library 108), uploading a seed music clip, or recording a seed music clip via the microphone capabilities of the device 1000.

[0089] 11 shows a personal electronic device 1000 having a graphical user interface (GUI) 1002 in which a user has selected the "acoustic" stack template option 1108 (e.g., by a touch gesture on the touch-sensitive surface of the device 1000).

[0090] FIG. 12 illustrates the personal electronic device 1000 having a graphical user interface (GUI) 1202 that is displayed in response to a user selecting the "Acoustic" stack template option 1108 as shown in FIG. 11. The GUI 1202 includes a text banner 1204 that presents an initial name for the stack being created. In this example, the initial name is "My Stacks," but this may be changed by the user. For example, upon selecting the text banner 1204 (e.g., by a touch gesture or other user input), the user is provided with a graphical user interface control (e.g., a text entry box control) in the GUI 1202 to change the initial name to one desired by the user. Also in the GUI 1202, GUI elements 1206, 1208, 1210, 1212, and 1214 represent music clips in the current stack. Each GUI element 1206, 1208, 1210, 1212, and 1214 representing a music clip indicates the type / genre / style of the music clip (e.g., "Drums", "Pads", "Bass", "Lead", "Vocals", etc.) and the name of the music clip (e.g., "SC VIOLA 60 COMBOFGD"). In this example, in accordance with the techniques disclosed herein, the indicated music clips are automatically selected by the service 100 for inclusion in the selected stack template. As a result, the "Pads", "Bass", "Lead", and "Vocals" music clips corresponding to GUI elements 1208, 1210, 1212, and 1214 form a harmonically compatible submix that matches the "Drums" music clip represented by GUI element 1206. GUI 1202 also includes a GUI control 1216 for requesting the addition of a new compatible music clip to the current stack. GUI control 1218 is for selecting a new collection of music clips to populate the currently selected stack template. Selecting control 1218 discards the current collection of music clips corresponding to GUI elements 1206, 1208, 1210, 1212, and 1214, and automatically selects a new matching collection of music clips to populate the selected stack template.A GUI control 1220 controls whether to play the audio of the current stack as a mix through the speaker 1224 of the device 1000. A musical note 1226 represents the sound of the current stack output from the speaker 1224 of the device 1000. A GUI control 1222 is for sharing the current stack. In some variations, when the GUI control 1220 is set to play the current stack, the current stack including each of the constituent music clips is played in a loop so that the user can hear what the current stack will sound like as a mix. The constituent music clips may be time-shifted or pitch-shifted by the device 1000 or the service 100 as needed to match or synchronize with the other constituent music clips. Each of the GUI elements 1206, 1208, 1210, 1212, and 1214 may include a playback progress indicator (e.g., 1228) that indicates where the respective music clip playback is currently at. For example, the playback indicator 1228 may move from left to right as the current stack is played in a loop, and when one play of the music clip represented by the GUI element 1206 is completed, playback of the music clip may restart from the beginning of the music clip, in which case the indicator 1228 restarts at the left edge of the GUI element 1206 and moves (animates) toward the right edge of the GUI element 1206 as playback progresses. In FIG. 13 and subsequent figures, the playback indicator is not shown to provide a clear example and to avoid unnecessarily obscuring other aspects of the disclosed techniques. Thus, the omission of the playback indicator from other figures does not imply that the playback indicator is not compatible with the techniques shown in those other figures.

[0091] 13 shows a personal electronic device 1000 having a GUI 1202, where a user is making a selection 1330 of a music clip to replace in the current stack. The selection 1330 may be made by a suitable user input, such as, for example, a right swipe touch gesture on the touch-sensitive surface of the device 1000. In this example, the user makes the selection 1330 to replace the music clip represented by the GUI element 1210 with a matching "base" music clip.

[0092] 14 illustrates the personal electronic device 1000 with a GUI 1402 in response to a user making a selection 1330 to replace the current "bass" music clip in the current stack. As a result of the selection 1330, the "FH2 FILTER LOOP PONG BASS" music clip has been replaced with the "FE2 DRM120 BACKBEAT" music clip, which has been determined to be harmonically compatible with the partial mix composed of the music clips represented by GUI elements 1208, 1212, and 1214 (recall that unpitched music clips are not included in the harmonic compatibility determination). Thus, a new partial mix is ​​formed composed of the music clips represented by GUI elements 1208, 1410, 1212, and 1214. The selection 1330 also causes the new current stack to be played in a looped mix, as indicated by the sound 1426 output by speaker 1224. In this way, the user can have an aural perception of what the new current stack will sound like as a mix with the new "bass" music clip.

[0093] Figure 15 illustrates personal electronic device 1000 having the GUI 1402 shown in Figure 14, where a user has made a selection 1532 of a music clip in the current stack to delete. Selection 1532 may be made by a suitable user input, such as, for example, a swipe touch gesture on the touch-sensitive surface of device 1000. In this example, the user makes selection 1532 to delete the "Pad" music clip represented by GUI element 1208.

[0094] 16 illustrates the personal electronic device 1000 with a GUI 1602 in response to a user making a selection 1532 to remove the "Pad" music clip from the current stack. As a result of the selection 1532, the "Pad" music clip is no longer part of the current stack. The sound 1626 output by the speaker 1224 reflects the playback of the current stack without the deleted "Pad" music clip, allowing the user to audibly perceive what the new current stack would sound like as a mix without the deleted "Pad" music clip.

[0095] Figure 17 illustrates a personal electronic device 1000 having the GUI 1602 shown in Figure 16. A user has now made a selection 1734 to add a new music clip to the current stack. The selection 1734 is made by sending appropriate user input to the GUI control 1216. For example, the selection 1734 may be made by a pressing touch gesture on the touch-sensitive surface of the device 1000, or the like.

[0096] 18 illustrates the personal electronic device 1000 with a GUI 1802 in response to a user making a selection 1734 to add a new layer to the current stack. The current stack continues to play in a loop as indicated by the sound 1626. The GUI 1802 includes a text banner 1804 that prompts the user to select a layer type for the new clip to be added. The GUI 1802 presents a collection of layer types 1836 as selectable options. The GUI 1802 also provides a cancel option 1838 that allows the user to cancel the current operation and return to the GUI state corresponding to the GUI 1602. As previously mentioned, the stack-based mixing application may support other methods of adding music clips to the current stack besides selecting a layer type. For example, the GUI 1802 may provide graphical user interface controls for selecting a music clip from the library 108 to add to the current stack (e.g., by searching or browsing the library 108), uploading a music clip to the service 100 to add to the current stack, or recording a music clip via the microphone functionality of the device 1000 to add to the current stack. In these instances, the selected, uploaded, or recorded music clip may be added to the current stack without regard to the harmonic compatibility of the additional music clip with the music clips in the current stack. However, the harmonic compatibility of the added music clip may be considered when selecting the next track to include in the current stack.

[0097] Figure 19 shows the personal electronic device 1000 with the GUI 1802 shown in Figure 18. Here, the user has selected 1940 a "Key" layer type for a new music clip to add to the current stack. The current stack continues playing in a mixed loop, as indicated by sound 1626.

[0098] 20 illustrates the personal electronic device 1000 with GUI 2002 in response to a selection 1940 of a "key" layer type. As a result, a new "key" music clip has been added to the current stack, as represented by GUI element 2042. The new "key" music clip is determined to be harmonically compatible with the current partial mix composed of the "bass" music clip represented by GUI element 1410, the "lead" music clip represented by GUI element 1212, and the "vocal" music clip represented by GUI element 1214, and a new partial mix composed of the "bass" music clip, the "lead" music clip, the "vocal" music clip, and the "key" music clip, and a new current stack composed of the new partial mix and the "drums" music clip are formed. A sound 2026 reflecting the new current stack is now output from speaker 1224 so that the user can hear what the new current stack will sound like as a mix with the new "key" music clip.

[0099] Fig. 21 shows personal electronic device 1000 with the GUI 2002 shown in Fig. 20. The user has now made a selection 2144 to share the current stack as a finished mix. For example, selection 2144 may be made with an appropriate touch gesture (e.g., a press touch gesture) on the touch-sensitive surface of device 1000.

[0100] FIG. 22 illustrates the personal electronic device 1000 with a GUI 2202 in response to the selection 2144 shown in FIG. 21. The GUI 2202 includes a text banner 2204 prompting the user to select a desired stack sharing method. As a result of the selection 2144, the mix playback of the current stack is stopped. Using GUI control 2254, playback of the current stack may be resumed from the speaker 1224. The GUI 2202 provides a GUI control 2246 for exporting the current stack / mix as a music clip to a social media platform (e.g., the aforementioned TIKTOK platform). A GUI control 2248 provides an option to save or export the current stack / mix as a music clip to the device 1000 (e.g., stored in a file system file, database, or shared memory segment). GUI control 2250 provides further sharing options, such as sharing the current stack / mix as a music clip via an email message attachment or a text (SMS) message attachment, or uploading the current stack / mix as a music clip to a cloud-based data storage service or a centrally hosted network file system. GUI 2202 also provides a cancel GUI control 2252 that allows the user to cancel the sharing operation and return to the GUI state corresponding to GUI 2002.

[0101] In some variations, the GUI 2202 provides a user option to export the stack to a digital audio workstation (DAW) so that the user can continue the music creation process. For example, the GUI 2202 may provide an option to export the generated stack and import it into music production software such as, for example, ABLETON LIVE, PRO TOOLS, CUBASE, etc. From there, the user may use the generated stack as a section in a new song composed by the user using the music production software.

[0102] In at least some embodiments, a system for implementing some or all of the techniques described herein may include a general-purpose computer system, such as computer system 2300 shown in FIG. 23, that includes or is configured to access one or more computer-accessible media. In the embodiment shown, computer system 2300 includes one or more processors 2310 connected to a system memory 2320 via an input / output (I / O) interface 2330. Computer system 2300 further includes a network interface 2340 connected to I / O interface 2330. Although FIG. 23 illustrates computer system 2300 as a single computing device, in various embodiments computer system 2300 may include one computing device or any number of computing devices configured to work together as a single computer system 2300.

[0103] In various embodiments, computer system 2300 may be a uniprocessor system including one processor 2310, or a multiprocessor system including several processors 2310 (e.g., two, four, eight, or another suitable number). Processor 2310 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 2310 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as the x86, ARM, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In a multiprocessor system, each of processors 2310 may generally, but not necessarily, implement the same ISA.

[0104] System memory 2320 may store instructions and data accessible by processor 2310. In various embodiments, system memory 2320 may be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated embodiment, program instructions and data implementing one or more desired functions, such as the methods, techniques, and data described above, are shown stored in system memory 2320 as service code 2325 (e.g., executable to implement service 100, in whole or in part) and data 2326.

[0105] In some embodiments, the I / O interface 2330 may be configured to coordinate I / O traffic between the processor 2310, the system memory 2320, and any peripheral devices in the device, including the network interface 2340 and / or other peripheral interfaces (not shown). In some embodiments, the I / O interface 2330 may perform any necessary protocol conversions, timing conversions, or other data conversions to convert data signals from one component (e.g., the system memory 2320) into a format suitable for use by another component (e.g., the processor 2310). In some embodiments, the I / O interface 2330 may include support for devices attached via various types of peripheral buses, such as, for example, variations on the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. In some embodiments, the functionality of the I / O interface 2330 may be split into two or more separate components, such as, for example, a northbridge and a southbridge. Also, in some embodiments, some or all of the functionality of the I / O interface 2330, such as the interface to the system memory 2320, may be incorporated directly into the processor 2310.

[0106] Network interface 2340 may be configured to enable data exchange between computer system 2300 and other devices 2360 connected to network 2350, such as other computer systems or devices shown in FIG. 1. In various embodiments, network interface 2340 may support communications over any suitable wired or wireless general-purpose data network, such as, for example, a type of Ethernet network. Additionally, network interface 2340 may support communications over a telecommunications / telephone network, such as an analog voice network or a digital fiber communications network, communications over a storage area network (SAN), such as a Fibre Channel SAN, and / or communications over any other suitable type of network and / or protocol.

[0107] In some embodiments, computer system 2300 includes one or more offload cards 2370A or 2370B (including one or more processors 2375 and possibly one or more network interfaces 2340) connected using an I / O interface 2330 (e.g., a bus implementing a version of the Peripheral Component Interconnect Express (PCI-E) standard or another interconnect such as the QuickPath Interconnect (QPI) or UltraPath Interconnect (UPI)). For example, in some embodiments, computer system 2300 can function as a host electronic device (e.g., operating as part of a hardware virtualization service) that hosts computing resources such as computing instances, and one or more offload cards 2370A or 2370B execute a virtualization manager that can manage the computing instances executing on the host electronic device. By way of example, in some embodiments, offload cards 2370A or 2370B can perform computing instance management operations such as pausing and / or unpausing computing instances, starting and / or terminating computing instances, performing memory transfer / copy operations, etc. These management operations, in some embodiments, may be performed by offload card 2370A or 2370B in coordination with (e.g., in response to requests from) a hypervisor executed by other processors 2310A-2310N of computer system 2300. However, in some embodiments, the virtualization manager implemented by offload card 2370A or 2370B may service requests from other entities (e.g., from the computing instance itself) and may not coordinate with (or provide services to) any separate hypervisor.

[0108] In some embodiments, the system memory 2320 may be one embodiment of a computer-accessible medium configured to store the aforementioned program instructions and data. However, in other embodiments, the program instructions and / or data may be received, transmitted, or stored on different types of computer-accessible media. The computer-accessible medium may include any non-transitory storage or memory medium, such as, for example, magnetic or optical media, such as disks or DVDs / CDs, connected to the computer system 2300 via the I / O interface 2330. The non-transitory computer-accessible storage medium may also include any volatile or non-volatile media that may be included as the system memory 2320 or another type of memory in some embodiments of the computer system 2300, such as RAM (e.g., SDRAM, double data rate (DDR) SDRAM, SRAM, etc.), read-only memory (ROM), etc. Additionally, the computer-accessible medium may include a transmission medium or a signal, such as an electrical, electromagnetic, or digital signal, conveyed over a communication medium, such as a network and / or a wireless link, such as may be implemented via the network interface 2340.

[0109] The various embodiments discussed or suggested herein may be implemented in a variety of operating environments, which in some cases may include one or more user computers, computing devices, or processing devices that may be used to run any of a number of applications. User or client devices may include any of a number of general purpose personal computers, such as desktop or laptop computers running standard operating systems, as well as cellular, wireless, and handheld devices that may run mobile software and support a number of networking and messaging protocols. Such systems may also include a number of workstations running any of a variety of commercially available operating systems and other well-known applications for purposes such as development and database management. These devices may also include other electronic devices, such as dummy terminals, thin clients, gaming systems, and / or other devices capable of communicating over a network.

[0110] Most embodiments use at least one network well known to those skilled in the art to support communications using any of a variety of widely available protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Common Internet File System (CIFS), Extensible Messaging and Presence Protocol (XMPP), AppleTalk, etc. Networks may include, for example, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), the Internet, an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0111] In embodiments using a web server, the web server may run any of a variety of server or mid-tier applications, such as an HTTP / S server, a File Transfer Protocol (FTP) server, a Common Gateway Interface (CGI) server, a data server, a Java server, a business application server, etc. The server may also be capable of executing programs or scripts in response to requests from user devices, such as by executing one or more web applications, which may be implemented as one or more scripts or programs written in any programming language, such as Java, C, C#, or C++, or any scripting language, such as Perl, Python, PHP, or TCL, as well as combinations thereof. The server may also include database servers, including, but not limited to, commercially available database servers from Oracle, Microsoft, Sybase, IBM, etc. Database servers may be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0112] The environments disclosed herein may include various data stores and other memory and storage media mentioned above. These may reside in various locations, such as on storage media local to (and / or resident in) one or more of the computers, or on storage media remote from any or all of the computers across a network. In a particular set of embodiments, information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any files necessary to perform functions belonging to a computer, server, or other network device may be stored locally and / or remotely, as appropriate. Where the system includes computerized devices, each such device may include hardware elements that may be electrically connected via a bus, including, for example, at least one central processing unit (CPU), at least one input device (e.g., a mouse, keyboard, controller, touch screen, or keypad), and / or at least one output device (e.g., a display device, printer, or speaker). Such a system may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices, such as random access memory (RAM) or read-only memory (ROM), as well as removable media devices, memory cards, flash cards, and the like.

[0113] Such devices may also include computer-readable storage media readers, communication devices (e.g., modems, network cards (wireless or wired), infrared communication devices, etc.), and working memory, as previously described. The computer-readable storage media readers may be connected to or configured to accept computer-readable storage media, which represent remote storage devices, local storage devices, fixed storage devices, and / or removable storage devices, as well as storage media for temporarily and / or more permanently containing, storing, transmitting, and retrieving computer-readable information. The systems and various devices also typically include a number of software applications, modules, services, or other elements located within at least one working memory device, including operating systems and application programs, such as client applications or web browsers. It should be understood that alternative embodiments may have numerous variations from those described above. For example, customized hardware may also be used, and / or particular elements may be implemented in hardware, software (including portable software, such as applets), or both. Additionally, connections to other computing devices, such as network input / output devices, may be employed.

[0114] The storage media and computer readable media containing the code or portions of code may include any suitable media known or previously used in the art, including, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any manner or technology for storing and / or transmitting information, such as computer readable instructions, data structures, program modules, or other data. Storage media and communication media include RAM, ROM, Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disk (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a system device. Based on the disclosure and teachings provided herein, one skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0115] In the above description, various embodiments have been described. For purposes of explanation, specific configurations and details have been set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that the embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0116] In the foregoing description and the appended claims, reference may be made to columns (e.g., columns of a matrix) or x-axis (e.g., x-axis of a plot), and reference may be made to rows (e.g., rows of a matrix) or y-axis (e.g., y-axis of a plot). Unless the context clearly indicates otherwise, any reference in the foregoing description or the appended claims to a column may be replaced with a row, and vice versa, and any reference to the x-axis may be replaced with the y-axis, and vice versa, without loss of generality.

[0117] Bracketed text and dashed blocks (e.g., large dashes, small dashes, dots, and dots) are used herein to describe optional operations that add additional features to some embodiments. However, such notations should not be understood to mean that these are the only options or optional operations, or that solid blocks are not optional in a particular embodiment.

[0118] Unless the context clearly dictates otherwise, the term "or" is used in the foregoing specification and appended claims in an inclusive (and not exclusive) sense, and thus, for example, when used to join a list of elements, the term "or" may mean one, some, or all of the listed elements.

[0119] Unless the context clearly indicates otherwise, the terms "comprising," "including," "having," "based on," and "encompassing," etc., are used in the foregoing specification and appended claims in an open-ended manner and do not exclude additional elements, features, acts, or operations.

[0120] Unless the context clearly dictates otherwise, conjunctive language such as the phrase "at least one of X, Y, and Z" is understood to convey that an item, term, etc. can be X, Y, or Z, or combinations thereof. Thus, such conjunctive language is not intended to require by default the meaning that at least one of X, at least one of Y, and at least one of Z are each present.

[0121] As used in the foregoing detailed description and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise.

[0122] In the foregoing detailed description and the appended claims, terms such as first, second, etc. are used herein to describe various elements in some instances, but these elements should not be limited to these terms, unless the context clearly dictates otherwise. These terms are used only to distinguish one element from another element. For example, a first computing device may be referred to as a second computing device, and similarly, a second computing device may be referred to as a first computing device. Although both the first computing device and the second computing device are computing devices, they are not the same computing device.

[0123] In the foregoing specification, the techniques have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for scalable similarity-based adaptive music mix generation, comprising: receiving a request for a music clip that is harmonically compatible with the indicated collection of one or more music clips; identifying a particular music clip that is harmonically compatible with the represented set of music clips over a predetermined number of musical beats based on a first pitch interval spatial representation of the represented set of music clips and a second pitch interval spatial representation of the particular music clip; providing a response to the request, the response indicating the particular music clip that has been identified as harmonically compatible with the indicated set of music clips over the predetermined number of musical beats based on the first pitch interval spatial representation of the indicated set of music clips and the second pitch interval spatial representation of the particular music clip; Including, A method performed by one or more computer systems.

2. the represented collection of one or more music clips includes a plurality of music clips; each music clip of said plurality of music clips is represented by a respective pitch interval space representation; 2. The method of claim 1, further comprising generating the first pitch interval spatial representation of the represented collection of music clips spanning the predetermined number of musical beats based on the respective pitch interval spatial representation of each music clip of the plurality of music clips.

3. including the particular music clip in a current music clip stack that includes the indicated collection of music clips; The method of claim 1 further comprising:

4. identifying the particular music clip as being harmonically compatible with the indicated collection of music clips over the predetermined number of musical beats based on a distance or similarity between the first pitch interval spatial representation and the second pitch interval spatial representation in pitch interval space; The method of claim 1 further comprising:

5. The method of claim 1 , wherein each of the first pitch interval spatial representation and the second pitch interval spatial representation is independent of beats per minute.

6. presenting a graphical user interface indicating that the particular music clip is harmonically compatible with the indicated collection of music clips. The method of claim 1 further comprising:

7. The method of claim 1 , wherein the request includes the first pitch interval spatial representation of the indicated collection of music clips.

8. The method of claim 1 , wherein the response includes an identifier for the particular music clip.

9. calculating a set of per-beat pitch interval spatial representations for the predetermined number of musical beats based on a chroma saliency map of the particular music clip, and forming the second pitch interval spatial representation based on the set of per-beat pitch interval spatial representations; The method of claim 1 further comprising:

10. 10. The method of claim 1, wherein the predetermined number of beats is 2, 4, 8, 16, 32, or 64.

11. 1. A system comprising one or more computer systems having one or more processors, the one or more computer systems implementing a music mixing service, the music mixing service including instructions that, when executed by the one or more processors, Computing a set of beat-by-beat pitch interval space vectors based on a chroma saliency map of the first music clip; forming a first per-clip pitch interval space vector for the first music clip based on the set of per-beat pitch interval space vectors; identifying a second music clip that is harmonically compatible with the first music clip based on a distance or similarity between a second per-clip pitch interval space vector formed for the second music clip in pitch interval space and the first per-clip pitch interval space vector formed for the first music clip; The system causes the one or more computer systems to execute the above.

12. Calculating the set of beat-by-beat pitch interval space vectors based on the chroma saliency map of the first music clip includes: generating a beats-per-minute independent chroma representation for said chroma saliency map; applying a Fourier transform to said beats per minute independent chroma representation signal; The system of claim 11 , comprising:

13. 12. The system of claim 11, wherein each of the set of beat-by-beat pitch interval space vectors is a 12-dimensional vector containing six real and six imaginary components resulting from a Fourier transform of a beats-per-minute independent chroma representation signal generated based on the chroma saliency map.

14. 12. The system of claim 11, wherein forming the first per-clip pitch interval space vector based on the set of per-beat pitch interval space vectors comprises concatenating the set of per-beat pitch interval space vectors.

15. The music mixing service includes instructions that, when executed by the one or more processors, indexing the first music clip with the first per-clip pitch interval space vector in an index that supports approximate nearest neighbor searches using the per-clip pitch interval space vector as a query key; The system of claim 11 , further comprising:

16. The music mixing service includes instructions that, when executed by the one or more processors, indexing said second music clip with a second per-clip pitch interval space vector in the index; further executing on said one or more computer systems; 12. The system of claim 11, wherein identifying the second music clip that is harmonically compatible with the first music clip includes performing an approximate nearest neighbor search of the index using the first per-clip pitch interval space vector as a query key.

17. The system of claim 16 , wherein the index is a quantization-based index, a tree-based index, or a graph-based index.

18. The music mixing service includes instructions that, when executed by the one or more processors, receiving a request for a music clip that is harmonically compatible with the first music clip; providing a response to said request indicating said second music clip; The system of claim 11 , further comprising:

19. The music mixing service includes instructions that, when executed by the one or more processors, performing all of the calculating of the set of per-beat pitch interval space vectors, the forming of the first per-clip pitch interval space vector, and the identifying of the second music clip in response to receiving the request; 20. The system of claim 18, further comprising:

20. The music mixing service includes instructions that, when executed by the one or more processors, presenting a graphical user interface indicating that the first music clip is harmonically compatible with the second music clip; The system of claim 11 , further comprising:

Citation Information

Patent Citations

  • Music information retrieval using 3D search algorithm

    JP2007183921A

  • Music editing device and method, and program

    JP2008164932A

  • Sound material retrieval device

    JP2010271398A

  • Method, system, and computer-readable medium for creating song mashups

    US20210201863A1

  • Information processing device and method, and program

    WO2020218075A1