Supervised metric learning for music structure features

Abstract

Disclosed herein are devices, systems, and methods related to implementing supervised metric learning during training of a deep neural network model. In an example, an audio input can be received, where the audio input includes a plurality of song segments from a plurality of songs. For each song segment, an alignment function can be performed to center the song segment based on determined beat information, creating a plurality of aligned song segments. For each song segment of the plurality of song segments, an embedding vector can be obtained from a deep neural network. Thus, a batch of aligned song segments can be selected from the plurality of aligned song segments, such that a training tuple can be selected. A loss metric can be generated based on the selected training tuple, and one or more weights of the deep neural network model can be updated based on the loss metric.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Application No. 17 / 502,890, filed October 15, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to machine learning, and in particular to methods and systems for training deep neural network models using supervised metric learning. Background Technology

[0004] In the field of Music Structure Analysis (MSA), many algorithms use conventional features such as Mel-frequency cepstral coefficients (MFCC) and pitch category profiles (PCP). Designing suitable features for MSA is challenging because many aspects of music—including pitch, timbre, rhythm, and dynamics—are important for perceptual structure. Past MSA methods have combined inputs from multiple features; however, caution must be exercised in doing so, at least because different features work better than others at different time scales. While supervised learning methods have been used to model musical boundaries and choruses, these methods are often incompatible with existing MSA processing pipelines; instead, new post-processing methods are needed to implement such approaches. Furthermore, previous supervised learning methods were limited to solving specific problems or versions of MSA; namely, segmentation and chorus detection. Existing methods do not consider segmentation and labeling. A supervised learning method that minimizes the loss incurred due to performing segmentation and labeling tasks simultaneously remains to be developed.

[0005] The aspects disclosed herein have been described in consideration of these and other general considerations. Furthermore, although relatively specific problems may be discussed, it should be understood that these examples are not intended to address specific problems identified in the background art or elsewhere in this disclosure. Summary of the Invention

[0006] According to at least one example of this disclosure, a method for implementing supervised metric learning during the training of a deep neural network model is described. The method may include implementing a deep neural network configured to receive songs and output embeddings representing the songs, and implementing a music structure analysis framework configured to receive embeddings, segment the embeddings, and detect repetitive portions of the songs. In the example, training of the deep neural network is implemented via supervised metric learning, comprising: receiving audio input including multiple song segments from multiple songs; determining beat information for each of the multiple song segments; performing an alignment function based on the beat information to center the song segments for each of the multiple song segments, thereby creating multiple aligned song segments; obtaining embeddings from the deep neural network model for each of the multiple song segments; selecting a batch of aligned song segments from the multiple aligned song segments, the batch of aligned song segments being associated with the same song among the multiple songs; sampling the selected batch of aligned song segments and selecting training tuples; generating a loss metric based on the selected training tuples; and updating one or more weights of the deep neural network model based on the loss metric.

[0007] According to at least one example of this disclosure, an apparatus for implementing supervised metric learning during the training of a deep neural network model is described. The apparatus may include at least one processor and a non-transitory computer-readable medium therein storing computer program code including instructions for one or more programs that, when executed by the processor, cause the processor to: implement a music structure analysis framework configured to receive embeddings representing songs from a deep neural network model, segment the embeddings, and detect repeated portions of the songs; and perform supervised metric learning during training of the deep neural network model by: receiving audio input comprising multiple song segments from multiple songs; determining beat information for each of the multiple song segments; performing an alignment function based on the beat information to center the song segments for each of the multiple song segments, thereby creating multiple aligned song segments; obtaining embeddings from the deep neural network for each of the multiple song segments; selecting a batch of aligned song segments from the multiple aligned song segments, the batch of aligned song segments being associated with the same song among the multiple songs; sampling the selected batch of aligned song segments and selecting training tuples comprising: generating a loss metric based on the selected training tuples; and updating one or more weights of the deep neural network model based on the loss metric.

[0008] Any one of the above-mentioned aspects, in combination with any other aspect of the above-mentioned aspects. Any one of the one or more aspects described herein.

[0009] The present invention is provided in a simplified form to introduce the chosen concepts, which are further described in the detailed description below. This present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the description which follows, and will be apparent in part from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0010] The embodiments will be more readily understood in conjunction with the following figures and in view of the following description, wherein the same reference numerals denote the same elements, in the figures:

[0011] Figure 1 A block diagram depicts an example training platform according to the examples of this disclosure.

[0012] Figure 2 A self-similarity matrix (SSM) according to an example of this disclosure is shown.

[0013] Figure 3 Additional details of a deep neural network according to an example of this disclosure are described.

[0014] Figure 4 Additional details are depicted in the example of the alignment audio input signal according to this disclosure.

[0015] Figure 5 Details of example data structures based on examples of this disclosure are described.

[0016] Figure 6 A block diagram depicts the physical components (e.g., hardware) of a computing device that can implement aspects of the present disclosure.

[0017] Figure 7 A flowchart illustrating an example of training a deep neural network model using supervised metric learning, according to this disclosure, is shown. Detailed Implementation

[0018] In the following detailed description, reference is made to the accompanying drawings, which form part of this specification, illustrating specific aspects or examples. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from this disclosure. The aspects may be practiced as methods, systems, or apparatuses. Therefore, the aspects may take the form of hardware implementations, entirely software implementations, or implementations combining software and hardware aspects. Consequently, the following detailed description should not be considered limiting, and the scope of this disclosure is defined by the appended claims and their equivalents.

[0019] Music structure analysis (MSA) methods traditionally search for musically meaningful patterns in audio, such as, but not limited to, homogeneity, repetition, novelty, and regularity in segment length. Handcrafted audio features, such as Mel-frequency cepstral coefficients (MFCCs) or chromatograms, are frequently used to elicit these patterns. However, as more section labels (e.g., verse, chorus, bridge) become available, supervised feature learning can be used to make these patterns more explicit and improve the performance of MSA methods. According to examples in this disclosure, a deep neural network is trained to utilize metric learning methods so that if two spectrogram inputs have the same section type (based on the annotation), embeddings that are close to each other are provided for the two spectrogram inputs; otherwise, the deep neural network provides embeddings that are far apart from each other. Thus, a batch sampling scheme can be used to ensure that the labels in the training pairs are interpreted in a meaningful way. The trained deep neural network model extracts features that can be used by existing MSA algorithms and generally improves existing MSA algorithms in both intra-dataset and cross-dataset scenarios.

[0020] According to examples in this disclosure, supervised metric learning is used during the training of a deep neural network model. For example, for a given song, audio segments located in different, widely spaced segments are embedded, as well as audio segments from the same, closer segments. Therefore, the model can better capture the homogeneity and repetitive characteristics of song structure relative to segment labels (e.g., solos, choruses, and bridges). To this end, a batch sampling scheme can be utilized to ensure that the labels in the training pairs are interpreted in a meaningful way. By using embeddings as a feature of existing MSA algorithms, the supervised method discussed in this paper supports segmentation and labeling tasks. Because embeddings carry structural information about the music, they can serve applications such as loop detection, music composition, and automated DJing.

[0021] Although metric learning has been applied to improve MSA using unsupervised methods—that is, if audio segments in a segment are temporally close to each other, they are presumed to belong to the same category, and if they are presumed to belong to different categories—many false positive and false negative pairs are used in training by design. According to the examples in this disclosure, the supervised strategy discussed here differs from previous methods at least by developing supervised feature learning aimed at improving existing MSA algorithms. An overview of MSA algorithms, corpora, and evaluation methods is provided by: Zhengshan Shi and Gautham J. Mysore. 2018. LoopMaker: Automatically Creating Music Loops from Pre-recorded Music. In the memoir of the 2018 CHI Conference on Human Factors in Computational Systems (CHI'18). Association for Computational Mechanics, New York, NY, Paper No. 454, pp. 1–6. DOI: https: / / doi.org / 10.1145 / 3173574.3174028, the contents of which are incorporated herein by reference in their entirety.

[0022] Figure 1 A block diagram depicts an example training platform according to the examples of this disclosure. Figure 1 The block diagram 100 shown may include three stages. The first stage includes a feature extraction stage 102; the second stage involves a mining and training stage 104; and the third stage includes a validation stage 106. In the example, the feature extraction stage 102 may include at least two modules. Audio features may first be synchronized with the beat or downbeat of the audio input clip at the audio input and alignment module 108. Therefore, the beat and downbeat may be determined or estimated at the beat / downbeat module 110, aligned at the audio input and alignment module 108, and provided as audio input to train the deep neural network model 112. In the example, the beat and downbeat may be estimated at the beat / downbeat module 110 using the madmom method. Therefore, the trained deep neural network model 112 can provide the song's embedding vector as the output of the subsequent MSA algorithm 114 to complete the task.

[0023] The mining and training phase 104 comprises four modules: a batching module 116, a mining module 118, a loss module 120, and a distance module 122. Batching involves dividing the training data into batches of fixed size at the batching module 110. To allow for reasonable comparisons within a batch of training samples, a batch may contain only samples from the same song. The mining module 118 provides an algorithm to select informative training tuples (e.g., pairs with different labels but high similarity) to compute the loss given the embeddings and labels of the examples in a given batch. While conventional metric learning methods generally use all tuples in a batch (or uniformly sample them) to train the model, using informative subsets accelerates convergence and provides better models and results as batch size increases.

[0024] According to examples in this disclosure, loss module 120 may utilize a multi-similarity loss function to unify aspects of multiple weighted schemes in music information retrieval (MIR) applications, rather than using traditional contrastive loss and triple loss functions typically used in deep metric learning applications. Therefore, the distance metric determined by distance module 122 can specify the geometric relationship between the output embeddings provided by the trained deep neural network. Common metrics include, but are not limited to, Euclidean distance and cosine similarity.

[0025] For the validation phase, the MSA algorithm 114 is used to generate boundary and label outputs, and the model's learning state is validated based on music structure analysis. While one or more open-source packages implementing one or more validation algorithms can be implemented, including but not limited to the Music Structure Analysis Framework (MSAF), algorithms designed for different tasks can be used to link training with different objectives.

[0026] As previously described, the audio input and alignment module 108 can receive audio input 101 and embed the output in sync with the strong beat by aligning the center of the input window with the center of each measure interval. A similar process can be applied when aligning the center of the input window with the center of the detected beat. In the example, the input window may be longer than the duration of a measure, where a measure may be a fixed length of time; therefore, additional contextual audio may exist before and after the strong beat interval. A windowing function can be applied to the audio input 101 to modify or eliminate the audio context. As an example, a separate mode can be applied to the audio input such that the contextual audio is zeroed out. As another example, a Hann-mode can be applied to the audio input, where the Hann-mode includes a Hann-shaped ramp from zero to one applied to the contextual audio. As another example, using a center mode, the input window may remain unchanged. According to the examples of this disclosure, the Hann-mode is utilized, which indicates that the contextual audio is useful, but ensures that the deep neural network model should focus on the signal around the center of the bar.

[0027] In the example, for each segment of audio input 101, the structural annotation contains two timestamps defining the interval and a label. This label can be an explicit function (e.g., introduction, solo, chorus) or an abstract symbol, such as, but not limited to, A, A', B, C, etc., indicating repetition. The training examples can then be labeled based on the center label in audio input 101. As used herein, a training example aligned with the i-th beat / strong beat of the j-th song can be referred to as... Where x and y are the audio and the tag, respectively.

[0028] As previously mentioned, the proposed batch sampling algorithm implemented by batch module 116 ensures that no overlapping song samples are batch sampled. Therefore, the labels provide supervisory meaning when comparing a batch of samples. For example, it is undesirable to compare a duet segment of song A with a duet segment of song B, because there is no prior knowledge about whether such duet segments should be embedded close to or far from each other in space.

[0029] Table 1

[0030]

[0031] The algorithm in Table 1 provides an example process for one period, i.e., a complete pass through the training dataset. The original input sequence, including various audio inputs 101, is shuffled to ensure that the batches are diverse and contain song segments throughout the song. Lines 4-6 of the algorithm ensure that when a song requires multiple batches, the last batch is full by copying examples from the song. Once a batch is sampled (line 8), the miner module 118 can be executed to select information pairs from that batch to compute the loss in order to provide an update to the deep neural network model 112.

[0032] In the example, the multi-similarity framework uses three types of similarity to estimate the importance of potential pairs: self-similarity (Sim-S), positive relative similarity (Sim-P), and negative relative similarity (Sim-N). While many existing deep metric learning methods consider only one of these types when designing their loss functions, the multi-similarity framework provides weights for important pairs by considering all three types of similarity, thereby improving performance and accuracy.

[0033] If implemented using a multi-similarity framework, for anchors If the anchor and the example have the same label (i.e., ), then example This will result in a positive pair, otherwise it will result in a negative pair (i.e., Therefore, the miner module 118 can determine the Sim-P's for each positive / negative pair for the anchor and select challenging pairs when certain conditions are met. During the loss phase, the loss module 120 can utilize Sim-S's and Sim-N's to compute the weights of the positive and negative pairs, respectively, where the weights correspond to the gradients used to update the deep neural network model. In summary, by using the MultiSimilarity framework, the miner module 118 minimizes intra-class dissimilarity. The loss module 120 simultaneously maximizes intra-class similarity and minimizes inter-class similarity. In other words, in the case of music, the desired result is to embed song fragments of the same type into tightly clustered groups, and clusters of different fragment types into groups that are far apart from each other.

[0034] Regarding the MSA algorithm 114, typical inputs include a sequence of feature vectors. The MSA algorithm 114 can then output predicted timestamps and abstract labels for each audio segment identified by those timestamps. For example, structured segmentation tasks can be implemented using the MSA algorithm 114 to identify large-scale non-overlapping segments of a given audio signal (e.g., solo, chorus). Potential applications include, but are not limited to, improving in-track navigation, generating enhanced segment-level music recommendation systems, and creating educational visualization tools for better understanding musical works. This task is typically divided into two sub-problems: boundary detection and structure grouping. The former identifies the start and end times of each musical segment in a piece of music, while the latter labels these segments based on their acoustic similarity. As another example, most music structure algorithms accept different types of features to discover structural relationships in harmony, timbre, loudness, or combinations thereof. Each of these features can depend on additional analysis parameters such as sampling rate, FFT size, and jump count. Furthermore, MSA can utilize a beat tracker to aggregate all features at the beat level, thus obtaining a so-called beat-synchronized representation. This process reduces the number of feature vectors while introducing rhythm invariance. As discussed in this paper, proximity features of similar song segments can be used as additional features by MSA.

[0035] Figure 2 Four self-similar matrices (SSMs)200 are proposed according to examples of this disclosure. Figure 2 The SSM 200 proposed in this paper is for the same song using different features. More specifically, the pairwise Euclidean distance matrix can be calculated first, and a Gaussian kernel can be applied to it to derive pairwise similarity. The two matrices on the left utilize a deep neural network trained by a multi-similarity mining module 118 and a loss module 120. The two matrices on the right are based on two traditional features, MFCC and PCP. Figure 2 The described and learned features can significantly enhance the blocks indicated by the image, thereby reducing the complexity faced by the implemented MSA algorithm. Figure 2Additionally, results 204 related to three MSA algorithms 114 are presented. The first proposed MSA algorithm is a spectral clustering algorithm; the second is a convex NMF (cnmf) algorithm; and the third is foote+fmc2d (using Foote's algorithm for segmentation and FMC2d for annotation). In the examples, temporal resolution can affect one or more MSA algorithms when compared to ground reality; therefore, some MSA algorithms may be more accurate when using beat synchronization features instead of strong beat synchronization features.

[0036] Figure 3 Additional details of a deep neural network model 112 according to an example of this disclosure are depicted. As previously described, the audio input 304 provided to the deep neural network model 112 may be specified as a window of the duration of the waveform audio (e.g., eight seconds), and the output from the deep neural network model 112 may be a multidimensional embedding vector 308, which may be provided to the MSA algorithm. Of course, the duration may be greater than or less than eight seconds. In the examples, a two-stage architecture may be used, where the audio is transformed into a time-frequency representation before entering the deep neural network model 112; however, it should be understood that a fully end-to-end deep neural network is also conceivable. Examples of two-stage architectures or models include, but are not limited to, Harmonic-CNN and ResNet-50. In some examples, the model is open source; in other examples, a non-open source model may be utilized. According to the examples of this disclosure, the final layer (e.g., the final two layers) of each model may be replaced by an embedding module 312. In the case where the last two layers typically consist of a dense layer plus an S-shaped layer, in the example of this disclosure, the embedding module 312, which replaces the last two layers, may include a linear layer 314, a leaky ReLU layer 316, a batch normalization layer 318, another linear layer 320, and an L2 normalization layer 322 at the output. Although a specific model and model type have been described, it should be understood that any model with a similar purpose can be used for the deep neural network model 112.

[0037] Figure 4Additional details associated with the aligned audio input signal 404 according to an example of this disclosure are described. As previously described, the audio input and alignment module 108 may receive audio input 101, such as audio signal 404, and embed the output in sync with strong beats 406 and 408 by aligning the center of the input window with the center 416 of each measure interval. A similar process may be applied if the center of the input window is aligned with the center of a determined beat. As previously described, the beat / strong beat module 110 may provide timestamps associated with strong beats 406 and 408, thereby defining the strong beat window. As previously described, the input window “w” 409 may be longer than the duration of a measure (e.g., where the measure is specified as a fixed amount or time), thus additional contextual audio may exist before and after the strong beat interval. Therefore, a windowing function may be applied to the audio input 101 to modify or eliminate the audio context. As an example, a separate mode 412 may be applied to the audio input such that the contextual audio is set to zero. As another example, Hann-mode 414 can be applied to audio input, where the Hann-mode includes a zero-to-one Hann-shaped ramp applied to the context audio. As another example, using center mode 410, the input window can remain unchanged.

[0038] Figure 5 Details of example data structures 502 and 512 according to the present disclosure are depicted. More specifically, data structure 502 may be stored in memory and may include one or more previously labeled training examples. For example, one or more previously labeled training examples may include a song identifier (e.g., Song_ID 504) and an associated label 508, wherein label 508 indicates an explicit function (e.g., introduction, solo, chorus) or an abstract symbol, such as, but not limited to, A, B, C, etc. Thus, data structure 502 may include training examples, wherein the training examples comprise audio prior to alignment by input and alignment module 108. Thus, since input and alignment module 108 may receive audio input 101, such as audio signal 404, and synchronize the output embedding with strong beats 406 and 408 by aligning the center of the input window with the center 416 of each bar interval, once aligned, the training examples, along with alignment information, may be stored in data structure 512, wherein, similar to data structure 502, data structure 512 may be stored in memory. Data structure 512 may include a song identifier (e.g., Song_ID 516) and associated labels 522 indicating explicit functions (e.g., introduction, solo, and chorus) or abstract symbols (such as, but not limited to, A, B, C, etc.). Additionally, data structure 512 may include timestamps 518 and 520 specifying the start and end times of the beat / strong beat.

[0039] Figure 6This is a block diagram illustrating the physical components (e.g., hardware) of a computing device 600 by which aspects of the present disclosure capable of performing one or more operations as described above and herein can be practiced. The computing device components described below are applicable to the aforementioned computing device. In a basic configuration, computing device 600 may include at least one processing unit 602 and system memory 604. Depending on the configuration and type of the computing device, system memory 604 may include, but is not limited to, volatile storage devices (e.g., random access memory), non-volatile storage devices (e.g., read-only memory), flash memory, or any combination of these memories.

[0040] System memory 604 may include operating system 605 and one or more program modules 606 adapted to perform the various aspects disclosed herein. For example, operating system 605 may be adapted to control the operation of computing device 600. Furthermore, the aspects of this disclosure may be practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. This basic configuration is as follows: Figure 6 The components within the dashed line 618 are shown. The computing device 600 may have additional features or functionality. For example, the computing device 600 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage devices... Figure 6 The diagram shows a removable storage device 612 and a non-removable storage device 614.

[0041] As described above, several program modules and data files may be stored in system memory 604. When implemented on at least one processing unit 602, program module 606 may perform processes including, but not limited to, one or more aspects, as described herein. Application 607 may include one or more of the following: audio input and alignment module 108, beat / strong beat module 110, batch processing module 116, excavator module 118, loss module 120, and distance module 122, as... Figure 1-5 As described in more detail below. In some examples, application 607 may include one or more of MSA algorithm 114 and / or deep neural network model 112, such as Figure 1-5 More detailed descriptions are available in the text.

[0042] Furthermore, aspects of this disclosure can be implemented in circuits containing discrete electronic components, in packages or integrated electronic chips containing logic gates, in circuits utilizing microprocessors, or on a single chip containing electronic components or a microprocessor. For example, aspects of this disclosure can be practiced via a System-on-a-Chip (SOC), where each or more of the components shown in FIG9 can be integrated onto a single integrated circuit. Such SOC devices may include one or more processing units, graphics units, communication units, system virtualization units, and various application functionalities, all integrated (or “programmed”) onto a chip substrate as a single integrated circuit. When operating via an SOC, the functionality described herein regarding the ability to switch client protocols can be operated via dedicated logic integrated on a single integrated circuit (chip) with other components of computing device 600. Aspects of this disclosure can also be practiced using other techniques capable of performing logical operations, such as AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. Furthermore, aspects of this disclosure can be practiced in general-purpose computers or any other circuit or system.

[0043] The computing device 600 may also have one or more input devices 615, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. Output devices 616, such as a display, speaker, printer, etc., may also be included. The above devices are examples, and other devices may be used. The computing device 600 may include one or more communication connections 617 that allow communication with other computing devices 650. Examples of suitable communication connections 617 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0044] As used herein, the term "computer readable media" may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, or program modules. System memory 604, removable storage device 612, and non-removable storage device 614 are examples of computer storage media (e.g., memory storage devices). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other article of manufacture that can be used to store information and is accessible by computing device 600. Any such computer storage media may be part of computing device 600. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0045] Communication media may be embodied in computer-readable instructions, data structures, program modules, or other data (such as carrier waves or other transmission mechanisms) in modulated data signals, and include any information transmission medium. The term "modulated data signal" describes a signal having one or more characteristics that are set or altered in a manner that encodes information in the signal. As an example, and not a limitation, communication media may include wired media, such as wired networks or direct-line connections, and wireless media, such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0046] Figure 7 Additional details are provided for a method 700 for training a deep neural network model using supervised metric learning, in which, for a given song, the deep neural network model will embed audio segments of some different parts of the song that are farther apart than audio segments from the same part. Figure 7 The general sequence of steps in method 700 is shown. Generally, method 700 begins at 702 and ends at 734. Method 700 may include more or fewer steps, or may proceed in a different order than... Figure 7 The steps are arranged in the manner shown. Method 700 can be implemented as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer-readable medium. In illustrative terms, method 700 is implemented by a computing device (e.g., 600) associated with a user. However, it should be understood that aspects of method 700 can be executed by one or more processing devices, which may include, but are not limited to, computing device 600 or a server. Furthermore, method 700 can be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SOC), neural processing unit, or other hardware device. References will be incorporated herein by reference. Figure 1-6 Method 700 is explained by describing the system, components, modules, software, data structures, user interfaces, etc.

[0047] Method 700 begins at 702, and the process can proceed to 704. At 704, the computing device receives audio input comprising multiple song segments from multiple songs. The method can proceed to 706, where the computing device determines beat information. In the example, the beat information may correspond to the timestamp of the beat and / or the timestamp of the strong beat. As previously mentioned, the beat information can be estimated at the beat / strong beat module 110 using the madmom method. Of course, other methods and approaches for obtaining beat and / or strong beat information are foreseeable. Method 700 can then proceed to 708, where, for each of the multiple song segments, an alignment function is performed to center the song segments based on the beat information, thereby creating multiple aligned song segments. In the example, the beat / strong beat module 110 ( Figure 1 It can be provided with the strong shot 406 and strong shot 408 ( Figure 4 The timestamp associated with the input window ("w") defines the forced entry window. As mentioned earlier, the input window "w" 409 ( Figure 4 The duration of a measure can be longer than that of a bar (e.g., when a measure is specified as a fixed amount or time), so additional contextual audio can exist before and after the strong beat interval. Therefore, a windowing function can be applied to audio input 101 ( Figure 1 This allows you to modify or eliminate the audio context. As an example, a standalone mode can be applied to the audio input, causing the context audio to be zeroed. As another example, a Hann-mode can be applied to the audio input, where the Hann-mode includes a Hann-shaped ramp from zero to one applied to the context audio. As yet another example, using a center mode, the input window can remain unchanged.

[0048] Method 700 can then proceed to 710, where for each of the multiple song segments, an embedding is obtained from the deep neural network. Furthermore, a batch of aligned song segments can be selected from the multiple aligned song segments, wherein this batch of aligned song segments is associated with the same song among the multiple songs. Method 700 can then proceed to 712, where a training tuple is selected from the selected batch of aligned song segments. Based on the training tuple, a loss metric can be generated at 714, and one or more weights of the deep neural network model can be updated based on the loss metric at 716. Method 700 can then end at 718. Thus, once trained, the deep neural network outputs embeddings of song segments with the same label, which are closer to each other than embeddings of song segments with different labels.

[0049] The phrases “at least one,” “one or more,” “or,” and “and / or” are open-ended expressions that are both conjunction and disjunctive in operation. For example, each of the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” “A, B, and / or C,” and “A, B, or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0050] The term “a” or “an” refers to one or more of the entities in question. Therefore, the terms “a” (or “an”), “one or more”, and “at least one” are used interchangeably herein. It should also be noted that the terms “comprising,” “including,” and “having” are used interchangeably.

[0051] As used herein, the term "automatic" and its variations refer to any process or operation that is typically continuous or semi-continuous and requires no material human input for its execution. However, a process or operation can be automatic if input is received prior to its execution, even if the execution of the process or operation uses material or non-material human input. If a human input influences how a process or operation is executed, that input is considered material. Human input that consents to the execution of a process or operation is not considered "material."

[0052] Any of the steps, functions, and operations discussed in this article can be executed continuously and automatically.

[0053] Exemplary systems and methods of this disclosure have been described for computing devices. However, to avoid unnecessarily obscuring this disclosure, several known structures and devices have been omitted from the foregoing description. Such omissions should not be construed as limiting. Specific details have been set forth to provide an understanding of this disclosure. However, it should be understood that this disclosure may be practiced in many ways other than those set forth herein.

[0054] Furthermore, while the exemplary aspects described herein illustrate various components of the co-located system, some components of the system may be located remotely, in a remote portion of a distributed network (such as a local area network and / or the Internet), or within a dedicated system. Therefore, it should be understood that system components may be combined into one or more devices, such as servers, communication equipment, or co-located on specific nodes of a distributed network, such as analog and / or digital telecommunications networks, packet-switched networks, or circuit-switched networks. As can be understood from the foregoing description, for computational efficiency reasons, system components may be arranged anywhere within the distributed network of components without affecting the operation of the system.

[0055] Furthermore, it should be understood that the various links connecting the elements can be wired or wireless links, or any combination thereof, or any other known or later developed elements (one or more) capable of supplying data to and / or transmitting data from the connected elements. These wired or wireless links can also be secure links and capable of transmitting encrypted information. For example, the transmission medium used as the link can be any suitable carrier for electrical signals, including coaxial cables, copper wires, and optical fibers, and can be in the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0056] Although the flowchart has been discussed and illustrated for a specific sequence of events, it should be understood that changes, additions, and omissions to that sequence may be made without materially affecting the operation of the disclosed configuration and aspects.

[0057] Several variations and modifications to this disclosure may be used. It is possible to provide some features of this disclosure without providing others.

[0058] In another configuration, the systems and methods of this disclosure may be implemented in conjunction with a dedicated computer, a programmable microprocessor or microcontroller and peripheral integrated circuit elements (one or more), an ASIC or other integrated circuit, a digital signal processor, hardwired electronic or logic circuitry (such as discrete component circuitry), a programmable logic device or gate array (such as a PLD, PLA, FPGA, PAL), a dedicated computer, any similar equipment, etc. Generally, any device (one or more) or means capable of implementing the methods illustrated herein can be used to implement various aspects of this disclosure. Exemplary hardware that can be used with this disclosure includes computers, handheld devices, telephones (e.g., cellular, internet-enabled, digital, analog, hybrid, etc.), and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile storage devices, input devices, and output devices. Furthermore, alternative software implementations, including but not limited to distributed processing or component / object distributed processing, parallel processing, or virtual machine processing, may also be constructed to implement the methods described herein.

[0059] In another configuration, the disclosed method can be readily implemented using software from an object-oriented or object-based software development environment that provides portable source code usable on various computer or workstation platforms. Alternatively, the disclosed system can be implemented, partially or entirely, in hardware using standard logic circuitry or VLSI designs. Whether a system according to this disclosure is implemented using software or hardware depends on the system's speed and / or efficiency requirements, specific functionalities, and the particular software or hardware system or microprocessor or microcomputer system used.

[0060] In another configuration, the disclosed methods can be implemented in part in software, which may be stored on a storage medium and executed on a programmed general-purpose computer in cooperation with a controller and memory, a dedicated computer, a microprocessor, etc. In these cases, the systems and methods of this disclosure can be implemented as programs embedded in a personal computer, such as applets. This can be implemented as a CGI script, a resource residing on a server or computer workstation, or as a routine embedded in a dedicated measurement system, system component, etc. The system can also be implemented by physically integrating the system and / or method into a software and / or hardware system.

[0061] As described herein, this disclosure is not limited to standards and protocols. Other similar standards and protocols not mentioned herein exist and are included in this disclosure. Furthermore, the standards and protocols mentioned herein, as well as other similar standards and protocols not mentioned herein, are periodically replaced by faster or more efficient equivalents with substantially the same functionality. Such alternative standards and protocols with the same functionality are considered equivalents included in this disclosure.

[0062] In various configurations and aspects, this disclosure includes components, methods, processes, systems, and / or apparatuses substantially as depicted and described herein, including various combinations, sub-combinations, and subsets thereof. Those skilled in the art, upon understanding this disclosure, will understand how to make and use the systems and methods disclosed herein. In various configurations and aspects, this disclosure includes providing apparatus and processes in the absence of items not depicted and / or described herein, or in various configurations or aspects thereof, including in the absence of such items that may have already been used in prior apparatus or processes, for example, to improve performance, achieve simplicity, and / or reduce the cost of implementation.

[0063] This disclosure relates to systems and methods for using supervised metric learning during the training of deep neural network models, based at least on the examples provided in the following sections:

[0064] (A1) In one aspect, some examples include methods for implementing supervised metric learning during the training of a deep neural network model. The method may include: implementing a deep neural network configured to receive a song and output an embedding representing the song; and implementing a music structure analysis framework configured to receive the embedding, segment the embedding, and detect repetitive parts of the song, wherein training of the deep neural network is implemented via supervised metric learning, comprising: receiving an audio input including multiple song segments from multiple songs; determining beat information for each of the multiple song segments; performing an alignment function based on the beat information to center the song segments for each of the multiple song segments, thereby creating multiple aligned song segments; obtaining an embedding from the deep neural network model for each of the multiple song segments; selecting a batch of aligned song segments from the multiple aligned song segments, the batch of aligned song segments being associated with the same song among the multiple songs; sampling the selected batch of aligned song segments and selecting training tuples; generating a loss metric based on the selected training tuples; and updating one or more weights of the deep neural network model based on the loss metric.

[0065] (A2) In some examples of A1, each song fragment includes a label corresponding to the repeated part of the corresponding song.

[0066] (A3) In some examples of A1-A2, the method further includes: for each of the multiple song segments, applying a windowing function to modify the audio context associated with the corresponding song segment.

[0067] (A4) In some examples of A1-A3, the selected training tuples consist of multiple aligned song segments with different labels.

[0068] (A5) In some examples of A1-A4, the method further includes shuffling multiple aligned song segments before selecting a batch of aligned song segments from multiple aligned song segments.

[0069] (A6) In some examples of A1-A5, once the deep neural network model is trained, it outputs embeddings for song segments with the same label that are closer to each other than embeddings of song segments with different labels.

[0070] (A7) In some examples of A1-A6, the method further includes providing an output embedding to a music structure analysis framework that segments and detects repeated song parts.

[0071] (A8) In some examples of A1-A7, the deep neural network includes an embedding module that includes a linear layer, a leaky ReLU layer, a batch normalization layer, and a normalization layer.

[0072] In another aspect, some examples include a system comprising one or more processors and a memory coupled to the one or more processors, the memory storing one or more instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods described herein (e.g., A1-A8 above).

[0073] In another aspect, some examples include a computer-readable storage medium storing one or more programs executed by one or more processors of a device, the one or more programs including instructions for performing any of the methods described herein (e.g., A1-A8 above).

[0074] (B1) On one hand, some examples include devices for implementing supervised metric learning during the training of deep neural network models. The apparatus may include at least one processor and a non-transitory computer-readable medium storing computer program code therein, the computer program code including instructions for one or more programs, which, when executed by the processor, cause the processor to: implement a music structure analysis framework configured to receive embeddings representing songs from a deep neural network model, segment the embeddings, and detect repeated portions of the songs; and perform supervised metric learning during training of the deep neural network model by: receiving audio input comprising multiple song segments from multiple songs; determining beat information for each of the multiple song segments; performing an alignment function based on the beat information to center the song segments for each of the multiple song segments, thereby creating multiple aligned song segments; obtaining embeddings from the deep neural network for each of the multiple song segments; selecting a batch of aligned song segments from the multiple aligned song segments, the batch of aligned song segments being associated with the same song among the multiple songs; sampling the selected batch of aligned song segments and selecting training tuples, the training tuples including: generating a loss metric based on the selected training tuples; and updating one or more weights of the deep neural network model based on the loss metric.

[0075] For example, the foregoing description of aspects of the present disclosure refers to block diagrams and / or operational illustrations of methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions marked in the blocks may not occur in the order shown in any flowchart. For example, two blocks shown consecutively may actually be performed substantially simultaneously, or these blocks may sometimes be performed in reverse order, depending on the functionality / actions involved.

[0076] The descriptions and illustrations of one or more aspects provided in this application are not intended to limit or define the scope of this disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey ownership and enable others to make and use the claimed disclosure as the best mode. The claimed disclosure should not be construed as limited to any aspect, example, or detail provided in this application. Various features (structural and methodological) are intended to be selectively included or omitted, whether shown and described in combination or separately, to produce embodiments having a particular set of features. Given the descriptions and illustrations provided in this application, those skilled in the art will envision variations, modifications, and alternatives falling within the spirit of the broader aspects of the overall inventive concept embodied in this application, without departing from the broader scope of the claimed disclosure.

Claims

1. A method for implementing supervised metric learning during the training of a deep neural network model, the method comprising: Implement a deep neural network model, which is configured to receive a song and output an embedding representing the song; as well as A music structure analysis framework is implemented, which is configured to receive the embedding, segment the embedding, and detect repeated portions of the song. The training of the deep neural network model is achieved through supervised metric learning, which includes: It can receive audio input including multiple song segments from multiple songs; For each of the multiple song segments, determine the beat information; For each of the plurality of song segments, an alignment function is performed based on the beat information to center the song segment, and a windowing function is applied to the song segment based on the center of the song segment, the windowing function removing at least a portion of the audio context of the song segment, thereby creating a plurality of aligned song segments; For each of the plurality of song segments, an embedding is obtained from the deep neural network model; Select a batch of aligned song segments from the plurality of aligned song segments, wherein the batch of aligned song segments is associated with the same song among the plurality of songs; Sample the selected batch of aligned song segments and select training tuples; Generate a loss metric based on the selected training tuples; and The deep neural network model is updated with one or more weights based on the loss metric.

2. The method of claim 1, wherein each song fragment includes a tag corresponding to a repeated portion of the corresponding song.

3. The method of claim 2, wherein the selected training tuple comprises a plurality of aligned song segments with different labels.

4. The method according to claim 2, further comprising: Before selecting the batch of aligned song segments from the plurality of aligned song segments, the plurality of aligned song segments are shuffled.

5. The method of claim 2, wherein the deep neural network model, once trained, outputs embeddings for song segments with the same label, the embeddings of song segments with the same label being closer to each other than the embeddings of song segments with different labels.

6. The method according to claim 5, further comprising: The output embedding is provided to the music structure analysis framework, which segments and detects repeated song parts.

7. The method according to claim 1, wherein the deep neural network model includes an embedding module, the embedding module including a linear layer, a leaky ReLU layer, a batch normalization layer, and a normalization layer.

8. An apparatus comprising: At least one processor and a non-transitory computer-readable medium therein storing computer program code, the computer program code including instructions for one or more programs, the instructions causing the processor, when executed by the processor, to: Implement a deep neural network model, which is configured to receive a song and output an embedding representing the song; as well as A music structure analysis framework is implemented, which is configured to receive an embedding representing a song, segment the embedding, and detect repeated parts of the song; The training of the deep neural network model is achieved through supervised metric learning, which includes: It can receive audio input including multiple song segments from multiple songs; For each of the multiple song segments, determine the beat information; For each of the plurality of song segments, an alignment function is performed based on the beat information to center the song segment, and a windowing function is applied to the song segment based on the center of the song segment, the windowing function removing at least a portion of the audio context of the song segment, thereby creating a plurality of aligned song segments; For each of the plurality of song segments, an embedding is obtained from the deep neural network model; Select a batch of aligned song segments from the plurality of aligned song segments, wherein the batch of aligned song segments is associated with the same song among the plurality of songs; Sample the selected batch of aligned song segments and select training tuples; Generate a loss metric based on the selected training tuples; and The deep neural network model is updated with one or more weights based on the loss metric.

9. The apparatus of claim 8, wherein each song fragment includes a tag corresponding to a repeated portion of the corresponding song.

10. The apparatus of claim 9, wherein the selected training tuple comprises a plurality of aligned song segments with different labels.

11. The apparatus of claim 9, wherein implementing the supervised metric learning during the training of the deep neural network model further comprises: Before selecting the batch of aligned song segments from the plurality of aligned song segments, the plurality of aligned song segments are shuffled.

12. The apparatus of claim 9, wherein once trained, the deep neural network model outputs embeddings for song segments with the same label, the embeddings of song segments with the same label being closer to each other than the embeddings of song segments with different labels.

13. The apparatus of claim 9, wherein the deep neural network model includes an embedding module, the embedding module including a linear layer, a leaky ReLU layer, a batch normalization layer, and a normalization layer.

14. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to: Implement a deep neural network model, which is configured to receive a song and output an embedding representing the song; as well as A music structure analysis framework is implemented, which is configured to receive the embedding, segment the embedding, and detect repeated portions of the song. The training of the deep neural network model is achieved through supervised metric learning, which includes: It can receive audio input including multiple song segments from multiple songs; For each of the multiple song segments, determine the beat information; For each of the plurality of song segments, an alignment function is performed based on the beat information to center the song segment, and a windowing function is applied to the song segment based on the center of the song segment, the windowing function removing at least a portion of the audio context of the song segment, thereby creating a plurality of aligned song segments; For each of the plurality of song segments, an embedding is obtained from the deep neural network model; Select a batch of aligned song segments from the plurality of aligned song segments, wherein the batch of aligned song segments is associated with the same song among the plurality of songs; Sample the selected batch of aligned song segments and select training tuples; A loss metric is generated based on the selected training tuples; and The deep neural network model is updated with one or more weights based on the loss metric.

15. The computer-readable storage medium of claim 14, wherein each song fragment includes a tag corresponding to a repeated portion of the corresponding song.

16. The computer-readable storage medium of claim 15, wherein the selected training tuple comprises a plurality of aligned song segments with different labels among the plurality of aligned song segments.

17. The computer-readable storage medium of claim 15, wherein the instructions, when executed by the processor, cause the processor to: Before selecting the batch of aligned song segments from the plurality of aligned song segments, the plurality of aligned song segments are shuffled.

Images