Computing systems and methods for detecting a mashup of music

Abstract

Embodiments of the present disclosure relate to computing systems and methods for detecting a mashup of music. A computing system for detecting a mashup of music is provided, including processing circuitry configured to receive audio data and encode the audio data to extract audio features that are indexed as a time series of the audio data. The processing circuitry is further configured to identify candidate breakpoints in the audio data based on the audio features. The processing circuitry is further configured to input the candidate breakpoints to a trained neural network classifier that classifies each of the candidate breakpoints as a music boundary or not a music boundary, wherein the trained neural network classifier has been trained on a training dataset of audio samples. The processing circuitry is further configured to determine a plurality of audio segments in the audio data based on the candidate breakpoints that are labeled as music boundaries, and output an identification of each of the audio segments.

Description

Background Technology

[0001] The rise of user-generated content (UGC) platforms, coupled with the increasing popularity of medleys—music tracks that combine fragments from multiple songs or tracks—has presented new challenges for music identification. Medley tracks are often characterized by significant alterations to the original songs, such as changes in pitch, rhythm, and arrangement, making them more difficult to identify using traditional music identification technologies. Traditional song identification systems, which heavily rely on database matching and audio fingerprinting, struggle with medley tracks because they require pre-storing each fragment of the medley and accurately identifying each part. This approach results in overly complex systems that are difficult to maintain and scale effectively. The challenges of detecting medley music have real-world implications for digital platforms, music streaming services, content creators, and rights holders. These stakeholders must navigate the complexities of accurate music identification to ensure proper attribution and policy compliance while maintaining a seamless user experience. Summary of the Invention

[0002] This paper provides a computational system for detecting music medleys. In one example, the computational system includes processing circuitry and a memory storing instructions that, during execution, cause the processing circuitry to receive audio data. The processing circuitry is also configured to encode the audio data to extract audio features indexed as a time series of the audio data. The processing circuitry is further configured to identify candidate breakpoints in the audio data based on the audio features via a breakpoint detection algorithm. The processing circuitry is also configured to input the candidate breakpoints into a trained neural network classifier, thereby classifying each candidate breakpoint as either a music boundary or not a music boundary. The trained neural network classifier has been trained on a training dataset of audio samples, each audio sample being labeled as either a music boundary or not. The processing circuitry is further configured to determine multiple audio segments in the audio data based on the candidate breakpoints labeled as music boundaries. The processing circuitry is also configured to output an identifier for each audio segment among the audio segments.

[0003] This synopsis is provided to present a simplified selection of concepts, which will be further described in the detailed implementations below. This synopsis is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that address any or all of the shortcomings pointed out in any part of this disclosure. Attached Figure Description

[0004] Figure 1 A schematic diagram of an example computational system for detecting a medley of music, implemented according to an example of this disclosure, is shown.

[0005] Figure 2The spectrum is shown with frequency on the y-axis and time on the x-axis.

[0006] Figure 3 It shows Figure 1 An example implementation of a trained neural network classifier for a computational system.

[0007] Figure 4 It shows according to Figure 1 A flowchart of a computerized method for detecting music medleys, implemented as an example of a computing system.

[0008] Figure 5 An example computing environment of this disclosure is shown, in which implementation can be performed. Figure 1 The computing system. Detailed Implementation

[0009] While existing music identification techniques exist to address some of the identified problems, those based on audio fingerprinting models are particularly unsuitable for detecting medleys. These systems work by matching audio segments against a vast database of known songs, relying on the unique characteristics of the audio to find similarities. However, these traditional methods encounter significant limitations when it comes to medleys where different songs or pieces are spliced ​​together, altered, or remixed.

[0010] For example, medley music often involves changes to the pitch, tempo, or rhythmic structure of the original songs. Audio fingerprinting methods sensitive to such changes cannot identify the individual song components within a medley, leading to incorrect or incomplete identification. Furthermore, medley tracks often string together multiple musical fragments into a single cohesive piece. Some fragments may overlap partially (at boundaries) or completely (such as when adding extra background beats to a DJ mix). Traditional song identification systems struggle to identify these segmented audio fragments because they were not originally designed to detect such complex, multi-source content. Finally, existing song databases used to match audio content must cover a wide range of genres, eras, and styles to be effective. Medley identification requires the database to not only contain a large number of songs but also ensure the uniqueness of track identification. For example, versions of songs with different rhythms or keys must still be mapped to a single, unified song entity. Creating and maintaining such a unique and comprehensive database is challenging.

[0011] These challenges lead to a major shortcoming in current systems: they lack the ability to handle the variability introduced by medley music, which limits their ability to provide accurate and comprehensive song identification. As a result, accurately detecting medley tracks remains a difficult task, and many existing systems are prone to errors when faced with alternating or medley music.

[0012] like Figure 1The diagram schematically illustrates a computing system 10 provided to solve the aforementioned problem of detecting music medleys. The computing system 10 includes processing circuitry 12, memory 14, one or more input devices 16 for receiving user input, and one or more output devices 18 (such as a display). The computing system 10 can be used as a client device, which may be a device such as a smartphone, tablet computer, smartwatch, or personal computer. Furthermore, the computing system 10 including the processing circuitry 12 may be one of a plurality of server devices, and the processing circuitry 12 may be a single processor of a single server device or multiple processors of multiple server devices.

[0013] Memory 14 stores instructions that, during execution, cause processing circuitry 12 to receive audio data 20, which may include a spectrogram. Audio data 20 may consist of instrumental pieces or music of various genres used as background music on user-generated social media platforms. Audio data 20 may be collected from user content and stored on the platform. Additionally, audio data 20 may be stored on the computing device of a user on the social media platform.

[0014] Briefly turn Figure 2 , Figure 2 A spectrum is shown with frequency (ranging from 0 Hz to 8192 Hz) on the y-axis and time (ranging from 0 seconds to 350 seconds) on the x-axis. The intensity of each pixel represents the amplitude of the frequency at each time point, with brighter pixels indicating higher amplitude and darker pixels indicating lower amplitude. Two distinct vertical dashed lines are present near the 175-second and 200-second marks, indicating significant events or changes in the signal within these time intervals. Breakpoint analysis, as described below, is performed to identify and characterize these changes in the signal.

[0015] Before discussing breakpoint analysis below, the audio data 20 can be preprocessed for effective breakpoint analysis. While the original waveform audio signal is inherently time-series data, it is often too imprecise and verbose for effective analysis. To enhance the information content of the signal, the original audio is transformed into a Mel spectrogram, which provides a more detailed and informative representation of the signal. Alternatively, other audio representations, such as Mel frequency cepstral coefficients (MFCC) or constant Q transform (CQT), or even combinations thereof, can be used, provided these representations capture the time variations in the audio signal. This transformation results in an input of the form [T, F], where T is the number of time steps and F is the number of feature dimensions in the frequency domain.

[0016] Furthermore, to improve computational efficiency, a downsampling step is introduced. Specifically, one frame is selected from every N consecutive frames, or alternatively, a pooling operation (average or max pooling) is applied to each consecutive group of N frames along the time axis. On the frequency axis, low-pass or high-pass filters can be applied to focus on the frequency range exhibiting the most significant changes, depending on the needs of the analysis. These techniques reduce the complexity of subsequent calculations and improve the overall efficiency of the process.

[0017] Once the audio data 20 is preprocessed into a feature matrix [T, F], a breakpoint detection algorithm 28 is applied to identify candidate breakpoints 30 (potential segmentation points) in the audio data 20. The processing circuit 12 is configured to encode the audio data 20 via encoder 24 to extract audio features 26, where the audio features 24 are indexed as a time series of the audio data 20. The processing circuit 12 is also configured to identify candidate breakpoints 30 in the audio data 20 based on the audio features 26 used for breakpoint detection via the breakpoint detection algorithm 28. Specifically, the extracted audio features 26 are input to the breakpoint detection algorithm 28 to identify moments in the audio data 20 where significant changes occur, typically referred to as change points, thereby detecting shifts in the underlying signal or time series. The breakpoint detection algorithm 28 calculates the variance of the audio features 26 over time and identifies candidate breakpoints 30 as the time points at which the calculated variance changes.

[0018] More specifically, candidate breakpoints 30 are evaluated using a cost function that is minimized to determine the optimal number of breakpoints in the audio data 20. Formally, given the input sequence... The goal of breakpoint detection is to find the intervals that divide a sequence into (such as...) The optimal segmentation. The quality of the segmentation is measured by the cost function:

[0019] in Quantifying the homogeneity of signals between two consecutive segments. Low cost indicates relatively homogeneous signals within a segment, while high cost indicates significant variability. An intuitive way to calculate this cost is to use the mean shift method, where the segments... The cost is calculated as follows:

[0020] in From arrive The mean of the segments.

[0021] To determine the optimal number of breakpoints K, optimization techniques such as dynamic programming are used, which leads to... The time complexity is [insert time complexity here]. More efficient algorithms, such as PELT (Pruned Exact Linear Time), can be used to reduce the time complexity to [insert time complexity here]. , where K is the number of breakpoints and T is the number of time frames.

[0022] The key consideration is determining the number of breakpoints, K. This can be empirically set based on the expected typical number of changes in segments of a medley of music. Alternatively, the elbow method can be used for automatic searching of the optimal K. The steps for applying the elbow method are as follows:

[0023] (i) Search range from 1 to of value

[0024] (ii) For each K, perform breakpoint detection and record the corresponding cost.

[0025] (iii) Plot the cost as a function of K and mark the “elbow” point, where the cost reduction begins to plateau. The optimal K corresponds to this elbow point. The basic principle behind this approach is that as K increases, the cost decreases, but beyond a certain point (the elbow), the rate of cost reduction slows significantly, indicating that adding more breakpoints has a diminishing return on improving the segmentation.

[0026] It will be understood that two main approaches to change point detection can be utilized: online methods, which detect changes in real time (suitable for real-time processing and anomaly detection) and therefore analyze only a portion of the dataset of the track up to the current time; and offline methods, which analyze the complete dataset of the track and benefit from the additional context provided by both preceding and following segments of data around the candidate breakpoint. Offline methods may be more suitable when the complete audio (e.g., a complete music track) is available for analysis, as it allows for more accurate segmentation.

[0027] Once candidate breakpoint 30 or the optimal set of breakpoints is identified... These time points are then passed to the next stage of the process. These candidate breakpoints 30 represent potential boundaries between the individual songs in the medley, providing valuable information for further analysis or segmentation. Essentially, candidate breakpoints 30 indicate moments when significant changes occur in the audio data 20. For example, these changes can occur within the same song, such as transitioning from a verse to a chorus, or between different songs spliced ​​together. The problem is framed as a binary classification for each breakpoint, where the task is to determine whether each candidate breakpoint 30 truly marks a boundary between two different songs.

[0028] To perform the above task, a trained neural network classifier 32 is utilized. Processing circuitry 12 is configured to input candidate breakpoints 30 into the trained neural network classifier 32, thereby classifying each candidate breakpoint 30 as either a musical boundary 34 or not a musical boundary 36. A musical boundary 34 can be a song boundary, a transition between different parts within a single piece of music, such as from verse to chorus, or from one movement to another in classical music. The trained neural network classifier 32 is trained to classify audio data 20 as a musical boundary when the classifier's confidence score is greater than a threshold probability.

[0029] A trained neural network classifier 32 has been trained on a training dataset of audio samples, where each audio sample is labeled as either a music boundary 34 or not a music boundary 36. Furthermore, the audio samples in the training data are encoded by an audio encoder to extract a time series of audio features for each audio sample. The extracted time series of audio features is then provided as input to the classifier during training.

[0030] Figure 3 It shows Figure 1 An example structure of a trained neural network classifier 32 in the computing system 10 is shown. The trained neural network classifier 32 can be based on a neural network architecture. For example, a recurrent neural network architecture can be used. In a specific example, the trained neural network classifier can be based on a convolutional recurrent neural network (CRNN).

[0031] exist Figure 3 The illustrated example shows the sequence of processing stages within the CRNN architecture. The input to the neural network is a Mel spectrogram 90, representing audio data 20 transformed into a visual format capturing frequency content over time. The Mel spectrogram is processed through successive 2D CNN (Convolutional Neural Network) blocks 92. These blocks 92 apply two-dimensional convolutional layers to extract relevant features from the spectrogram 90, enhancing the representation of the audio signal. After the convolutional blocks 92, the extracted features are fed into LSTM (Long Short-Term Memory) layers 94. These layers 94 are designed to process sequential data and capture temporal dependencies, which is crucial for understanding the progression of the audio signal over time. After the LSTM layers 94, the data is further processed by an MLP (Multilayer Perceptron) layer 96. This MLP layer 96 performs additional computations to refine the features and prepare them for classification. The final stage is the output layer 98, which provides classification probabilities, indicating the likelihood of different classes. These probabilities help determine the boundaries between different musical segments. For example, if the probability is low (i.e., below a threshold confidence level), the segment is considered a part of a single song.

[0032] Calculate probability scores for all candidate breakpoints. After calculating the probability scores for all breakpoints, select the highest score as the final score for the entire song (considered a medley).

[0033]

[0034] Any breakpoint with a score higher than a predefined threshold is identified as a musical boundary between two different songs or musical segments. Processing circuitry 12 is configured to determine multiple audio segments 40 in the audio data 20 via audio segment determination module 38 based on candidate breakpoints marked as musical boundaries 34. Processing circuitry 12 is also configured to output an identifier for each audio segment 44 in the audio segments for downstream analysis of each audio segment.

[0035] Using the boundaries of each detected music segment, the original track can be divided into segments for further analysis. For example, when audio data 20 corresponds to media content published on a social media platform, the social media platform may include a policy enforcement engine configured to enforce one or more content policies on the media content hosted on the social media platform. When the media content includes audio data 20 corresponding to a medley of music, the policy compliance of each audio segment within the audio segments is checked individually at the policy enforcement engine of the social media platform. For example, audio segments determined by the policy enforcement engine to be non-compliant with platform policies will be flagged for manual review, programmatically rejected for publication, programmatically removed, or programmatically flagged with a compliance failure notification.

[0036] Furthermore, each segment can be processed using a music fingerprinting system or a cover song detection algorithm. The results of these systems can help refine previous steps by filtering out potential errors. For example, if the music fingerprints of two consecutive segments match a single song in a fingerprint database, this can indicate a misclassification in boundary detection, thus indicating the need for error correction. Additionally, detailed metadata can be generated for each segment, including start and end times, and how each segment maps back to the original music. This is particularly useful when processing segments or fragments, as it provides contextual information about how the segments fit into the broader structure of the original track. This detailed segment-level data can be used for a range of downstream applications, including copyright management (e.g., identifying and removing unlicensed music) or ensuring compliance with music use regulations (e.g., detecting and removing non-compliant content).

[0037] Figure 4 It shows according to Figure 1A flowchart of a computerized method 200 for detecting music medleys, implemented as an example of a computing system 10, is provided. At step 202, the method may include receiving audio data. At step 204, the method may include encoding the audio data to extract audio features, wherein the audio features are indexed as a time series of the audio data. At step 206, the method may include identifying candidate breakpoints in the audio data based on the audio features via a breakpoint detection algorithm.

[0038] At step 208, the method may include feeding candidate breakpoints into a trained neural network classifier, thereby classifying each candidate breakpoint as either a musical boundary or not a musical boundary. The trained neural network classifier has been trained on a training dataset of audio samples, each audio sample being labeled as either a musical boundary or not a musical boundary. At step 210, the method may include determining multiple audio segments in the audio data based on the candidate breakpoints labeled as musical boundaries. At step 212, the method may output an identifier for each audio segment within the audio segments.

[0039] The systems and methods described above enhance the accuracy and flexibility of mashup detection by focusing on identifying the boundaries between different musical sections within a mashup, regardless of any transformations applied to the original song or music. This improvement enables digital platforms, content creators, and copyright holders to more effectively manage and protect their music, ensuring accurate identification and credit for both original and derivative works.

[0040] In some embodiments, the methods and processes described herein can be attached to a computing system of one or more computing devices. In particular, such methods and processes can be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0041] Figure 5 A non-limiting embodiment of a computing system 300 capable of implementing one or more of the methods and processes described above is schematically illustrated. The computing system 300 is shown in a simplified form. The computing system 300 can implement the methods and processes described above and... Figure 1 The computing system 10 is illustrated in the figure. Components of the computing system 300 may be included in one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, video game devices, mobile computing devices, mobile communication devices (e.g., smartphones) and / or other computing devices, as well as wearable computing devices (such as smartwatches and head-mounted augmented reality devices).

[0042] The computing system 300 includes a logic processor 302, volatile memory 304, and non-volatile storage device 306. The computing system 300 may optionally include a display subsystem 308, an input subsystem 310, a communication subsystem 312, and / or... Figure 5 Other components not shown.

[0043] The logic processor 302 includes one or more physical devices configured to execute instructions. For example, the logic processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, change the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0044] A logic processor may include one or more physical processors configured to execute software instructions. Additionally or alternatively, a logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of logic processor 302 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Components of the logic processor may optionally be distributed across two or more separate devices, which may be remotely located and / or configured for coordinated processing. Aspects of the logic processor may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. In this context, it will be understood that these virtualized aspects run on different physical logic processors on various different machines.

[0045] The non-volatile storage device 306 includes one or more physical devices configured to store instructions executable by a logic processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 306 can be transformed—for example, to store different data.

[0046] Non-volatile storage device 306 may include removable and / or built-in physical devices. Non-volatile storage device 306 may include optical storage, semiconductor memory, and / or magnetic storage, or other mass storage technologies. Non-volatile storage device 306 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It will be understood that non-volatile storage device 306 is configured to retain instructions even when power to non-volatile storage device 306 is cut off.

[0047] Volatile memory 304 may include a physical device including random access memory. Volatile memory 304 is typically used by logic processor 302 to temporarily store information during the processing of software instructions. It will be understood that when power to volatile memory 304 is cut off, volatile memory 304 will typically not continue storing instructions.

[0048] Various aspects of the logic processor 302, volatile memory 304, and non-volatile storage device 306 can be integrated into one or more hardware logic components. For example, such hardware logic components may include field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SOCs), and complex programmable logic devices (CPLDs).

[0049] The terms "module," "program," and "engine" can be used to describe aspects of a computing system 300 typically implemented in software by a processor to perform specific functions using portions of volatile memory, involving transformation processing specifically configured for the processor to perform those functions. Thus, a module, program, or engine can be instantiated via logic processor 302 using portions of volatile memory 304 to execute instructions stored in non-volatile storage device 306. It will be understood that different modules, programs, and / or engines can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass individuals or groups of executable files, data files, libraries, drives, scripts, database records, etc.

[0050] When included, the display subsystem 308 can be used to present a visual representation of the data stored by the non-volatile storage device 306. The visual representation may take the form of a graphical user interface (GUI). When the methods and processes described herein change the data stored by the non-volatile storage device, and thus change the state of the non-volatile storage device, the state of the display subsystem 308 can also be changed to visually represent the changes in the underlying data. The display subsystem 308 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logic processor 302, the volatile memory 304, and / or the non-volatile storage device 306 in a shared housing, or such display devices may be peripheral display devices.

[0051] When included, the input subsystem 310 may include or interface with one or more user input devices, such as a keyboard, mouse, touchscreen, camera, or microphone.

[0052] When included, the communication subsystem 312 can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 312 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via wired or wireless local area networks or wide area networks, broadband cellular networks, etc. In some embodiments, the communication subsystem may allow the computing system 300 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0053] The following paragraphs provide additional description of the subject matter of this disclosure. One aspect provides a computational system for detecting music medleys, including processing circuitry and a memory storing instructions that, during execution, cause the processing circuitry to receive audio data. The processing circuitry is further configured to encode the audio data to extract audio features, wherein the audio features are indexed as a time series of the audio data. The processing circuitry is further configured to identify candidate breakpoints in the audio data based on the audio features via a breakpoint detection algorithm. The processing circuitry is further configured to input the candidate breakpoints into a trained neural network classifier, thereby classifying each candidate breakpoint as a music boundary or not a music boundary, wherein the trained neural network classifier has been trained on a training dataset of audio samples, each audio sample being labeled as a music boundary or not a music boundary. The processing circuitry is further configured to determine multiple audio segments in the audio data based on the candidate breakpoints labeled as music boundaries. The processing circuitry is further configured to output an identifier for each audio segment among the audio segments.

[0054] In this respect, the breakpoint detection algorithm calculates the variance of audio features over time and identifies candidate breakpoints as the time points at which the calculated variance changes.

[0055] In this regard, candidate breakpoints are evaluated using a cost function that is minimized to determine the optimal number of breakpoints in the audio data.

[0056] In this respect, the trained neural network classifier is trained to classify audio data as music boundaries when the classifier's confidence score is greater than a threshold probability.

[0057] In this respect, the audio samples in the training data are encoded by an audio encoder to extract the time series of audio features for each audio sample, and the extracted time series of audio features is provided as input to the classifier during training.

[0058] In this respect, audio data includes spectrograms.

[0059] In this respect, audio data corresponds to media content published on social media platforms, and the social media platforms, including policy enforcement engines, are configured to enforce one or more content policies on the media content hosted on the social media platforms.

[0060] In this regard, when media content contains audio data corresponding to a medley of music, the policy compliance of each audio segment in the audio clip is checked separately at the policy enforcement engine of the social media platform.

[0061] In this respect, trained neural network classifiers are based on convolutional recurrent neural networks (CRNNs).

[0062] On the other hand, a computerized method for detecting music medleys is provided. This method includes encoding audio data to extract audio features, wherein the audio features are indexed as a time series of the audio data. The method also includes identifying candidate breakpoints in the audio data based on the audio features via a breakpoint detection algorithm. The method further includes feeding candidate breakpoints as input to a trained neural network classifier, thereby classifying each candidate breakpoint as either a music boundary or not a music boundary, wherein the trained neural network classifier has been trained on a training dataset of audio samples, each audio sample being labeled as either a music boundary or not. The method also includes determining multiple audio segments in the audio data based on candidate breakpoints labeled as music boundaries. The method further includes outputting an identifier for each audio segment in the audio segments.

[0063] In this respect, the breakpoint detection algorithm calculates the variance of audio features over time and identifies candidate breakpoints as the time points at which the calculated variance changes.

[0064] In this regard, candidate breakpoints are evaluated using a cost function that is minimized to determine the optimal number of breakpoints in the audio data.

[0065] In this respect, the trained neural network classifier is trained to classify audio data as music boundaries when the classifier's confidence score is greater than a threshold probability.

[0066] In this respect, the audio samples in the training data are encoded by an audio encoder to extract the time series of audio features of each audio sample, and the extracted time series of audio features is provided as input to the classifier during training.

[0067] In this respect, audio data corresponds to media content published on social media platforms, and the social media platforms, including policy enforcement engines, are configured to enforce one or more content policies on the media content hosted on the social media platforms.

[0068] In this regard, when media content contains audio data corresponding to a medley of music, the policy compliance of each audio segment in the audio clip is checked separately at the policy enforcement engine of the social media platform.

[0069] In this respect, audio data includes spectrograms.

[0070] In this respect, trained neural network classifiers are based on convolutional recurrent neural networks (CRNNs).

[0071] On the other hand, a non-transient computer-readable medium including instructions stored thereon is provided. In this aspect, when executed by processing circuitry, the instructions cause the processor to perform the following steps. The processor is configured to perform the step of receiving audio data. The processor is also configured to perform the step of encoding the audio data to extract audio features, wherein the audio features are indexed as a time series of the audio data. The processor is also configured to perform the step of identifying candidate breakpoints in the audio data based on the audio features via a breakpoint detection algorithm. The processor is also configured to perform the step of inputting a trained neural network classifier into the candidate breakpoints, thereby classifying each candidate breakpoint as a musical boundary or not a musical boundary, wherein the trained neural network classifier has been trained on a training dataset of audio samples, each audio sample being labeled as a musical boundary or not a musical boundary. The processor is also configured to perform the step of determining multiple audio segments in the audio data based on the candidate breakpoints labeled as musical boundaries. The processor is also configured to perform the step of identifying each audio segment in the output audio segments.

[0072] In this respect, the breakpoint detection algorithm calculates the variance of audio features over time and identifies candidate breakpoints as the time points at which the calculated variance changes.

[0073] The “and / or” used in this article is defined as inclusive or ∨, as specified in the following truth table:

[0074] It will be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered limiting, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions illustrated and / or described may be performed in the illustrated and / or described order, in another order, in parallel, or omitted. Similarly, the order of the above processes may also be changed.

[0075] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computing system for detecting medleys of music, comprising: Processing circuitry and memory, storing instructions that, during execution, cause the processing circuitry to: Receive audio data; The audio data is encoded to extract audio features, which are then indexed as a time series of the audio data. Candidate breakpoints in the audio data are identified based on the audio features using a breakpoint detection algorithm. The candidate breakpoints are input into a trained neural network classifier, thereby classifying each candidate breakpoint as a musical boundary or not a musical boundary. The trained neural network classifier has been trained on a training dataset of audio samples, and each audio sample is labeled as a musical boundary or not a musical boundary. Based on the candidate breakpoints marked as music boundaries, multiple audio segments in the audio data are determined; and Output the identifier of each audio segment in the audio segment.

2. The computing system according to claim 1, wherein The breakpoint detection algorithm calculates the variance of the audio features over time and identifies the candidate breakpoints as the time points when the calculated variance changes.

3. The computing system according to claim 2, wherein The candidate breakpoints are evaluated using a cost function, which is minimized to determine the optimal number of breakpoints in the audio data.

4. The computing system according to claim 1, wherein The trained neural network classifier is trained to classify the audio data as the music boundary when the confidence score of the classifier is greater than a threshold probability.

5. The computing system according to claim 1, wherein The audio samples in the training data are encoded by an audio encoder to extract the time series of the audio features for each audio sample, and the extracted time series of the audio features is provided as input to the classifier during training.

6. The computing system according to claim 1, wherein The audio data includes a spectrogram.

7. The computing system according to claim 1, wherein The audio data corresponds to media content published on a social media platform, and the social media platform, including a policy enforcement engine, is configured to enforce one or more content policies on the media content hosted on the social media platform.

8. The computing system according to claim 7, wherein When the media content includes audio data corresponding to a medley of the music, each audio segment in the audio segment is individually checked for policy compliance at the policy enforcement engine of the social media platform.

9. The computing system according to claim 1, wherein The trained neural network classifier is based on a convolutional recurrent neural network (CRNN).

10. A computerized method for detecting medleys of music, comprising: Receive audio data; The audio data is encoded to extract audio features, which are then indexed as a time series of the audio data. Candidate breakpoints in the audio data are identified based on the audio features using a breakpoint detection algorithm. The candidate breakpoints are input into a trained neural network classifier, thereby classifying each candidate breakpoint as a musical boundary or not a musical boundary. The trained neural network classifier has been trained on a training dataset of audio samples, and each audio sample is labeled as a musical boundary or not a musical boundary. Based on the candidate breakpoints marked as music boundaries, multiple audio segments in the audio data are determined; and Output the identifier of each audio segment in the audio segment.

11. The computerized method according to claim 10, wherein The breakpoint detection algorithm calculates the variance of the audio features over time and identifies the candidate breakpoints as the time points when the calculated variance changes.

12. The computerized method according to claim 11, wherein The candidate breakpoints are evaluated using a cost function, which is minimized to determine the optimal number of breakpoints in the audio data.

13. The computerized method according to claim 12, wherein The trained neural network classifier is trained to classify the audio data as the music boundary when the confidence score of the classifier is greater than a threshold probability.

14. The computerized method according to claim 12, wherein The audio samples in the training data are encoded by an audio encoder to extract the time series of the audio features for each audio sample, and the extracted time series of the audio features is provided as input to the classifier during training.

15. The computerized method according to claim 10, wherein The audio data corresponds to media content published on a social media platform, and the social media platform, including a policy enforcement engine, is configured to enforce one or more content policies on the media content hosted on the social media platform.

16. The computerized method according to claim 15, wherein When the media content includes audio data corresponding to a medley of the music, each audio segment in the audio segment is individually checked for policy compliance at the policy enforcement engine of the social media platform.

17. The computerized method according to claim 10, wherein The audio data includes a spectrogram.

18. The computerized method according to claim 10, wherein The trained neural network classifier is based on a convolutional recurrent neural network (CRNN).

19. A non-transient computer-readable medium comprising instructions stored thereon, which, when executed by processing circuitry, cause the processor to perform the following steps: Receive audio data; The audio data is encoded to extract audio features, which are then indexed as a time series of the audio data. Candidate breakpoints in the audio data are identified based on the audio features using a breakpoint detection algorithm. The candidate breakpoints are input into a trained neural network classifier, thereby classifying each candidate breakpoint as a musical boundary or not a musical boundary. The trained neural network classifier has been trained on a training dataset of audio samples, in which each audio sample is labeled as a musical boundary or not a musical boundary. Based on the candidate breakpoints marked as music boundaries, multiple audio segments in the audio data are determined; and Output the identifier of each audio segment in the audio segment.

20. The non-transient computer-readable medium of claim 19, wherein... The breakpoint detection algorithm calculates the variance of the audio features over time and identifies the candidate breakpoints as the time points when the calculated variance changes.

Images