Neural network model for audio track tag generation

Through the training and application of deep neural network model, the problem of automatic identification of music structural elements is solved, and the accurate identification and extraction of music structural elements in the audio track is achieved, which simplifies the music processing process.

CN119072739APending Publication Date: 2024-12-03FACE CUTE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380034637.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-26
Filing Date
2023-05-08
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to automatically identify and identify music structural elements, especially when the user's subjective opinions exist, and for computing devices, there are challenges to identify music structural elements in new songs or sound tracks.

Method used

The deep neural network model is used to train the music theory labels and clip boundary identification to generate the music theory labels and clip boundary within the audio track, and then automatically identify the music structural elements in the audio track.

Benefits of technology

Automatic identification and identification of music structural elements is realized, simplifying the extraction and mixing of music structural elements, and improving the accuracy and efficiency of music processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072739A_ABST
    Figure CN119072739A_ABST
Patent Text Reader

Abstract

Systems and methods for identifying music theory tags of audio tracks are described. More specifically, a first training set of audio portions may be generated from a plurality of tracks, segments within the plurality of tracks being tagged according to a plurality of music theory tags. The deep neural network model may then be trained using the first training set as input, a first loss function for music theory tag identification of the audio portions of the first training set, and a second loss function for segment boundary identification within the audio portions of the first training set. In an example, a music theory tag identifier and a segment boundary identifier are generated by a deep neural network model. A first audio track is received, and fragment boundary identifiers and music theoretical tags of fragments in the first audio track are generated using a deep neural network model.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 804,198, filed on May 26, 2022, and entitled “NEURAL NETWORK MODEL FOR AUDIO TRACK LABEL GENERATION,” the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] Popular songs can be organized into common music structure elements according to music theory. Some music structure elements may only occur once, such as a specific verse or solo in a song, and other elements may be repeated twice or more, such as a chorus or bridge. In some scenarios, identifying these music structure elements in a song promotes extracting an audio "thumbnail" or preview of the song, because it is easier to extract more familiar elements (e.g., chorus) for the thumbnail, rather than simply randomly extracting from the song. In other scenarios, a user may wish to create a mix or mashup of several songs, which may make it easier to copy or place in a musically pleasing order specific sections of songs with a common theme by having a marking structure element. However, several users may have different subjective opinions on when a music structure element begins or which music structure element exists. The timing of the music structure element and the subjective nature of the selection bring challenges to the music structure elements of a new song or track that are automatically identified by a computing device.

[0004] It is with respect to these and other general considerations that the embodiments have been described.Furthermore, although relatively specific problems have been discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified in the background. Summary of the invention

[0005] Aspects of the present disclosure relate to a neural network model for identifying music theory tags for audio tracks.

[0006] In one aspect, a computer-implemented method of training a neural network model for identifying music theory labels for audio tracks is provided. A first training set of audio portions is generated from a plurality of audio tracks. Segments within the plurality of audio tracks may be labeled according to a plurality of music theory labels. A deep neural network model is trained using the first training set as input, a first loss function for music theory label identification of the audio portions of the first training set, and a second loss function for segment boundary identification within the audio portions of the first training set. The music theory label identification and segment boundary identification are generated by the deep neural network model. A first audio track is received. Segment boundary identifications for segments within the first audio track are generated using the deep neural network model. Music theory labels for the segments within the first audio track are generated using the deep neural network model.

[0007] In another aspect, a method for identifying music theory tags for an audio track is provided. An audio track is received. The audio track is divided into a first set of audio portions. Music theory tag identifiers and segment boundary identifiers for the first set of audio portions are generated using a deep neural network model. The generated music theory tag identifiers for the first set of audio portions are merged, and the segment boundary identifiers for the first set of audio portions are merged. Segments within the audio track are identified using the merged segment boundary identifiers. Corresponding music theory tags for the identified segments are identified.

[0008] On the other hand, a non-transitory computer-readable storage medium includes instructions executable by one or more processors, which when executed by the one or more processors cause the one or more processors to: generate a first training set of audio portions from multiple audio tracks, wherein segments within the multiple audio tracks are labeled according to multiple music theory labels; train a deep neural network model using the first training set as input, a first loss function for music theory label identification of the audio portions of the first training set, and a second loss function for segment boundary identification within the audio portions of the first training set, wherein the music theory label identification and the segment boundary identification are generated by the SpecTNT neural network model; receive a first audio track; generate segment boundary identifications for segments within the first audio track using the deep neural network model; and generate music theory labels for the segments within the first audio track using the deep neural network model.

[0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0011] Figure 1A block diagram illustrating an example of a system for training a neural network model for identifying music theory tags for audio tracks according to an example embodiment.

[0012] Figure 2 A block diagram illustrating an example of a neural network model for providing segment boundary identification and music theory label identification according to an example embodiment.

[0013] Figure 3 A diagram showing an example neural network model for providing segment boundary identification and music theory label identification according to an example embodiment.

[0014] Figure 4 A diagram showing an example logic flow for providing segment boundary identification and music theory tag identification according to an example embodiment.

[0015] Figure 5 A diagram illustrating example multiple label probability curves according to an example embodiment.

[0016] Figure 6 Graph showing example multiple label probability curves, boundary probability curves, and post-processing music theory label identification according to example embodiments.

[0017] Fig. 7A A flow chart illustrating an example method of training a neural network model for identifying music theory tags for audio tracks according to an example embodiment.

[0018] Figure 7B A flow chart illustrating an example method of identifying music theory tags for an audio track according to an example embodiment.

[0019] Figure 8 A block diagram is provided to illustrate example physical components of a computing device with which aspects of the present disclosure may be practiced.

[0020] Fig. 9 and Fig.10 is a simplified block diagram of a mobile computing device with which aspects of the present disclosure may be practiced. DETAILED DESCRIPTION

[0021] In the following detailed description, reference is made to the accompanying drawings forming part thereof, and specific embodiments or examples are shown by way of illustration in the accompanying drawings. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. The embodiments may be practiced as methods, systems, or devices. Therefore, the embodiments may take the form of hardware implementations, complete software implementations, or implementations combining software and hardware aspects. Therefore, the following detailed description should not be regarded as restrictive, and the scope of the present disclosure is limited by the appended claims and their equivalents.

[0022] The present disclosure describes various examples of a computing device having an audio processor configured to train a neural network model for identifying music theory labels for audio tracks. Non-overlapping segments within the audio tracks are pre-labeled with appropriate music theory labels. In some examples, the music theory labels correspond to music theory structures, such as intro ("intro"), verse, chorus, bridge, coda, or other suitable labels. In other examples, the music theory labels correspond to non-structural music theory elements, such as vibrato, harmonics, chords, etc. In still other examples, the music theory labels correspond to key signature changes, rhythm changes, etc. In some examples, the audio processor identifies music theory labels for overlapping segments, such as labels for key signatures, rhythm changes, and structures (i.e., intro, verse, chorus).

[0023] In some examples, the audio processor divides the audio track into portions of fixed duration, such as 15 seconds, 24 seconds, 60 seconds, or another suitable duration. The neural network model is then trained using these portions as input, a first loss function for music theory label identification of the audio portions, and a second loss function for segment boundary identification within the audio portions. The music theory label identification and segment boundary identification are generated by the SpecTNT neural network model, where the music theory label identification estimates the music theory label (e.g., verse, chorus) for the segment identification between the identified segment boundaries. Once trained, the neural network model can be used to identify music theory labels for audio tracks, even when these tracks have a shorter duration than typical tracks (e.g., 20 seconds versus 3 minutes or more). In some scenarios, the neural network model marks segments within the audio track or audio portion for automatic preview extraction of the song. For example, the identified chorus segment of the song can be used to generate a preview because the chorus is generally considered to be the 'most prominent' and 'most attractive' segment of the song.

[0024] This and many other embodiments of a computing device are described herein. For example, Figure 1 A block diagram of an example of a system 100 for training a neural network model for identifying music theory labels for an audio track is shown in accordance with an example embodiment. System 100 includes a computing device 110 configured to train a neural network model, such as neural network model 118 and / or neural network model 128. In some examples, computing device 110 is configured to perform music structure analysis on an audio track or audio portion. In some examples, system 100 may also include a data repository 120 communicatively coupled to computing device 110 via network 140.

[0025] Computing device 110 may be any type of computing device, including a smartphone, a mobile computer, or a mobile computing device (e.g., Devices, laptops, notebooks, tablets, such as the Apple iPad TM, netbook, etc.), or a stationary computing device such as a desktop computer or PC (personal computer). Computing device 110 may be configured to communicate with a social media platform, a cloud processing provider, a software-as-a-service provider, or other suitable entity, for example, using social media software and a suitable communication network. Computing device 110 may be configured to execute one or more software applications (or "apps") and / or services and / or manage hardware resources (e.g., processor, memory, etc.) that may be used by a user of computing device 110.

[0026] The computing device 110 includes an audio processor 111 and a neural network model 118. Figure 1 In the example shown, the audio processor 111 includes a boundary processor 112, a segment processor 114, and a post-processor 116. In other examples, one or more of the boundary processor 112, the segment processor 114, and the post-processor 116 may be formed as a combined processor. In some examples, at least some portions of the audio processor 111 may be combined with the neural network model 118, for example by including a neural network processor or other suitable processor configured to implement a neural network model. In other words, the neural network model 118 may be integrated with the audio processor 111 and implemented using or as a neural network processor. In some examples, the neural network model 118 is omitted from the computing device 110, and a neural network model 128 is used instead.

[0027] Boundary processor 112 is configured to generate segment boundary identifications within an audio portion. For example, boundary processor 112 may receive an audio portion and identify boundaries within the audio portion that correspond to changes in music theory tags. Typically, the boundaries identify non-overlapping segments within a song or excerpt that have a particular music theory tag. As an example, an audio portion having a duration of 24 seconds may begin with a four-second intro, followed by an eight-second verse, followed by a ten-second chorus and a two-second verse (e.g., the first part of the verse). In this example, boundary processor 112 may generate segment boundary identifications at 4 seconds, 12 seconds, and 22 seconds. In some examples, boundary processor 112 communicates with neural network model 118 and / or neural network model 128 to identify the boundaries.

[0028] The segment processor 114 is configured to generate a music theory tag identification for an audio portion. In various examples, the music theory tag identification can be selected from a plurality of music theory tags. In some examples, at least some of the plurality of music theory tags represent structural elements of the music. Examples of music theory tags may include an intro ("intro"), verse, chorus, bridge, instrumental (e.g., a guitar solo or a bass solo), coda, silence, or other suitable tags. In some examples, the segment processor 114 identifies a probability that a particular audio portion or a segment or timestamp within a particular audio portion corresponds to a particular music theory tag from a plurality of music theory tags. In other examples, the segment processor 114 identifies the most likely music theory tag for a particular audio portion (or a segment or timestamp within a particular audio portion). In still other examples, the segment processor 114 identifies the start and stop times when a music theory tag within the audio portion is active. See below for details. Figure 5 and Figure 6 Additional details of the music theory tag identification are provided. In some examples, segment processor 114 communicates with neural network model 118 and / or neural network model 128 to generate the music theory tag identification.

[0029] The post-processor 116 is configured to improve the training of the neural network model 118 by providing a comparison loss function between the ground truth of the input (e.g., an audio track from the source audio 130 with labeled segments and boundaries) and the outputs of the boundary processor 112 (e.g., segment boundary identifications) and the segment processor 114 (e.g., music theory labels).

[0030] Neural network model 118 is trained using audio processor 111 and is configured to process the audio portion to provide segment boundary identification and music theory labels within the audio portion. In some examples, neural network model 118 includes one or more blocks of a transformer neural network model in a spectro-temporal transformer. Neural network model 128 is generally similar to neural network model 118, but is stored remotely from computing device 110 (e.g., at data repository 120).

[0031] Data repository 120 may include one or more of any type of storage mechanism, including a disk (e.g., in a hard drive), an optical disk (e.g., in an optical drive), a tape (e.g., in a tape drive), a memory device such as a RAM device, a ROM device, etc., and / or any other suitable type of storage medium. For example, data repository 120 may store neural network model 128 and / or source audio 130 (e.g., an audio track used to train neural network model 118 and / or 128). In some examples, data repository 120 provides source audio 130 to audio processor 111 for training neural network model 118 and / or neural network model 128. In some examples, one or more data repositories 120 may be co-located (e.g., housed in one or more nearby buildings with associated components such as backup power, redundant data communications, environmental control, etc.) to form a data center, or may be arranged in other ways. Thus, in one embodiment, one or more of data repositories 120 may be a data center in a distributed collection of data centers.

[0032] The source audio 130 includes multiple tracks, such as songs, parts or excerpts from songs, etc. As used herein, a track can be a single song containing several individual tracks such as a guitar track, a drum track, a vocal track, etc., or can include only one track as a single instrumental or input, or a mixed track with multiple sub-tracks. Typically, multiple tracks within the source audio 130 are labeled using music theory tags for non-overlapping segments within the tracks. In some examples, different groups of tracks within the source audio 130 can be labeled using different music theory tags. For example, one group of tracks can use five tags (e.g., intro, verse, pre-chorus, chorus, coda), while another group uses seven tags (e.g., silence, intro, verse, refrain, bridge, instrumental, coda). Some groups can allow segment subtypes (e.g., verse A, verse B) or composite tags (e.g., instrumental chorus).

[0033] In some examples, the audio processor 111 is configured to convert labels between tracks from different groups to use the same multiple music theory labels. This label conversion improves training opportunities by allowing consistent training between tracks from different groups. Example groups of tracks within the source audio 130 may include SALAMI-pop, RWC-Pop, Harmonix, Isophonics, or other suitable groups that may use different numbers of music theory labels, or different music theory labels for equivalent segments of tracks. For example, one group may use "instrumental", while another group may use "solo", and yet another group may distinguish between "guitar solo", "bass solo", and "piano solo". The audio processor 111 may convert each of these labels into "instrumental" or another suitable label so that tracks from different groups may be easily used to train the neural network model 118.

[0034] The network 140 may include one or more networks, such as a local area network (LAN), a wide area network (WAN), an enterprise network, the Internet, etc., and may include one or more of wired and / or wireless portions. The computing device 110 and the data repository 120 may include at least one wired or wireless network interface that enables communication with each other (or with an intermediate device, such as a web server or a database server) via the network 140. Examples of such network interfaces include, but are not limited to, an IEEE 802.11 wireless LAN (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth® interface, a cellular network ... TM An interface or a near field communication (NFC) interface. Examples of network 140 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), the Internet, and / or any combination thereof.

[0035] Figure 2 A block diagram of an example of a neural network model 200 for providing segment boundary identification 220 and music theory label identification 230 according to an example embodiment is shown. In some examples, the neural network model 200 generally corresponds to the neural network model 118. The neural network model 200 can receive an input 210, such as an audio portion from the source audio 130, and provide the segment boundary identification 220 and the music theory label identification 230. Figure 2 In the example shown, segment boundary identifiers 220 are provided as probability rankings over the duration of the audio portion, and music theory label identifiers 230 are provided as probability rankings for each of five music theory labels: bridge label 231, intro label 232, verse label 233, chorus label 234, and coda label 235. The vertical spikes in segment boundary identifiers 220 correspond to possible starts and / or ends of segments. Figure 2 As shown, the neural network model 200 identifies seven segments, including the first segment as the prelude, the second segment as the verse, the third segment as the chorus, the fourth segment as the verse, the fifth segment as the chorus, the sixth segment as the verse, and the seventh segment as the ending.

[0036] Using the generated segment boundary identifications 220 and music theory label identifications 230, the neural network model 200 is trained using one or more loss functions such as a boundary loss function 240 and a segment loss function 250. In some examples, the segment loss function 250 is a first sum of a first weighted binary cross entropy between the generated music theory label identifications 230 and the corresponding music theory labels of the segments within the input 210 (e.g., the ground truth segment labels from the source audio 130). The segment loss function 250 may also include a connectionist temporal localization loss function configured to model the ordinal order of the music theory labels. In some scenarios, such as "pop" music, music tracks follow one of several common progressions of music theory labels, such as verse, chorus, verse, chorus, etc. A connectionist temporal localization loss function improves the responsiveness of the neural network model 200 to a track with a music theory label that follows a common progression of music theory labels, for example by weighting a second music theory label (e.g., chorus) more heavily following a first music theory label (e.g., verse) when the sequence of the first and second labels is common for tracks.

[0037] In some examples, boundary loss function 240 is a second sum of second weighted binary cross entropies between the segment boundary identifications and corresponding boundaries of the segments within input 210 .

[0038] Figure 3 A diagram of an example neural network model 300 for providing segment boundary identification and music theory label identification according to an example embodiment is shown. In some examples, neural network model 300 generally corresponds to neural network model 118 and / or 200, and receives input 310 (e.g., similar to input 210) and provides output 320 (e.g., similar to segment boundary identification 220 and music theory label identification 230). Figure 3In the example shown, the neural network model 300 is implemented as a Spectrum Time Transformer Transformer (SpecTNT) neural network model. The features of the SpecTNT neural network model are based on the interaction between the two-stage transformer encoders, in particular the spectral transformer and the time transformer. The spectral transformer is configured to extract spectral features via frequency class tokens (FCTs) for each time step, where the FCT is an aggregate embedding that characterizes harmonic and timbre information. The time transformer then exchanges local information (i.e., FCTs) along the time axis. This self-attention step can help discover structural patterns related to novelty, homogeneity, and repeatability. For example, the self-attention mechanism can allow frames around a boundary to pay attention to the boundary, and frames with the same function to pay attention to each other. Due to its hierarchical design, SpecTNT allows a smaller number of parameters compared to other neural network models.

[0039] The neural network model 300 is a SpecTNT model and includes three modules: a two-dimensional residual network (ResNet) 342 that extracts intermediate information from the input 310 at the front end; a stack of two or more SpecTNT blocks 344; and a linear layer 346 that provides an output 320 of target probabilities at various time steps. In some examples, the input 310 is a raw audio portion directly from the source audio 130. In other examples, the audio processor 111 performs a harmonic constant Q transform (HCQT) on the raw audio portion to generate the input 310. In one example, the ResNet 342 includes a convolutional layer using a kernel size of 3, five instances of the SpecTNT block 344 are applied simultaneously, 96 feature maps with 4 attention heads are used for the spectral encoder of each SpecTNT block 344, and 96 feature maps with 8 attention heads are used for the temporal encoder.

[0040] Figure 4 A diagram of an example logic flow 400 for providing segment boundary identification and music theory label identification according to an example embodiment is shown. The logic flow 400 includes a neural network model 418 that generally corresponds to the neural network model 118, 128, 200 and / or 300. The neural network model 418 is configured to receive the audio track 410 and generate the music theory label identification and the segment boundary identification. In some examples, the audio processor 111 divides the audio track into multiple audio portions. In other words, the audio portions can be sub-portions 420 of the same audio track. For example, an audio track with a duration of 2 minutes (120 seconds) can be divided into 20 second portions, 24 second portions, 40 second portions, or another suitable duration.

[0041] In some examples, portion 420 is non-overlapping, so that the 120-second track is divided into six 20-second portions. In other examples, portion 420 is overlapping, but offset by a predetermined duration, such as two seconds, three seconds, or another suitable duration. For example, the audio processor 111 can generate a first audio portion using a sliding window across the first track. In other words, for a predetermined duration of three seconds of overlap, the 120-second track is divided into approximately 40 portions: a first portion from 0 to 20 seconds; a second portion from 3 to 23 seconds; a third portion from 6 to 26 seconds, etc. Advantageously, dividing the track into portions of shorter duration increases the number of samples available for training the neural network model 418 and improves the accuracy of the neural network model 418. In addition, training the neural network model 418 using audio portions of shorter duration improves the analysis accuracy of short tracks (e.g., 20 or 25 second tracks). In some examples, the audio portion is "oversized" to accommodate the end of the track. For example, for an overlapping three second predetermined duration, a 28 second audio track is divided into 3 parts: a first part from 0 to 20 seconds; a second part from 3 to 23 seconds; and a third part from 6 to 28 seconds. In other examples, the audio portion is "downsized" to fit the end of the audio track. For example, for an overlapping three second predetermined duration, a 28 second audio track is divided into 3 parts: a first part from 0 to 20 seconds; a second part from 3 to 23 seconds; a third part from 6 to 26 seconds; and a fourth part from 9 to 28 seconds.

[0042] The neural network model 418 generates a boundary likelihood 432 for each audio portion (e.g., a 20 second audio portion). The boundary likelihood 432 can be a boundary probability curve 438 indicating the probability of crossing a boundary of the audio portion. The neural network model 418 also generates a segment likelihood 434 for each audio portion. The segment likelihood 434 can be a plurality of label probability curves 436 corresponding to a plurality of music theory labels, for example, similar to the music theory label identification 230. The audio processor 111 may also include a merging processor 442 for merging the boundary probability curves 438 of audio portions from a single audio track into a single boundary probability curve 452 for the audio track. Similarly, the audio processor 111 may also include a merging processor 444 for merging the label probability curves 436 of audio portions from a single audio track into a label probability curve 454 for the audio track. Reference is made below to Figure 5 Additionally describe the label probability curve.

[0043] Logic flow 400 includes a boundary processor 460, similar to boundary processor 112, configured to generate segment boundary identifiers within the audio portion. Figure 4In the example shown, the segment boundary identification is shown as a vertical dashed line across the boundary probability curve 452. In some examples, the boundary processor 460 provides the segment boundary identification when the boundary probability curve 452 exceeds a predetermined threshold, such as 60% probability, 85% probability, etc.

[0044] Logic flow 400 also includes a segment processor 470, which is similar to segment processor 114 and is configured to generate music theory label identifications 480 for the segments identified by the segment boundary identifications. Figure 4 In the example shown, the music theory tag identification includes ten tags: Intro, Verse, Chorus, Bridge ("B"), Verse, Chorus, Bridge, Verse, Chorus, and Ending. It is worth noting that, for example, when a boundary identification has a relatively lower probability (e.g., 40%, 60%) than other identified boundaries (e.g., 85%), the segment processor 470 may identify music theory tags for segments covering more than one identification. As an example, a first boundary identification has a relatively low probability compared to a second boundary identification, so the segment processor 470 may consider that the "Intro" segment includes the first boundary identification. Figure 4 In the example shown, the segment processor 470 uses the second, fifth, sixth, seventh, ninth, twelfth, thirteenth, fifteenth, and seventeenth boundary identifiers to identify ten segments.

[0045] Figure 5 A diagram of an example plurality of label probability curves 500 is shown according to an example embodiment. The plurality of label probability curves 500 generally corresponds to the label probability curve 454, and similar to the music theory label identification 230, the plurality of label probability curves 500 includes a probability curve for each music theory label in a plurality of music theory labels that may be generated by the neural network model 418. Figure 5 In the example shown, multiple music theory labels include an intro (probability curve 502), a verse (probability curve 504), a chorus (probability curve 506), a bridge (probability curve 508), and an outro (probability curve 510), with the combined probability curve being shown at element 554 across a horizontal time scale (e.g., the duration of an audio portion or track).

[0046] Figure 6Graph 600 showing example multiple label probability curves, boundary probability curves, and post-processed music theory label identifications according to an example embodiment. For example, graph 600 includes label probability curve 454, boundary probability curve 452, music theory label identification 480, and post-processed music theory label identification 610. Post-processor 116 generates post-processed music theory label identification 610 based on label probability curve 454, boundary probability curve 452, and music theory label identification 480. In general, post-processor 116 is configured to provide a comparison loss function between a ground truth value of an audio track having a labeled segment (not shown) and curves 452 and 454. In other words, post-processor 116 compares generated music theory label identification 480 to the ground truth label from the input audio track and provides an increased level of loss when generated music theory label identification 480 is further away from the ground truth label.

[0047] Fig. 7A A flow chart showing an example method 700 for training a neural network model for identifying music theory labels for an audio track according to an example embodiment. Unless otherwise indicated, the technical processes shown in these figures will be performed automatically. In any given embodiment, some steps of the process may be repeated, possibly using different parameters or data. The steps in the embodiment may also be performed in the same manner as in the example embodiment. Fig. 7A The steps of method 700 may be performed in an order different from the order from top to bottom shown in the flowchart. The steps may be performed serially, in a partially overlapping manner, or completely in parallel. Therefore, the order of performing the steps of method 700 may change from one execution of the process to another execution of the process. Steps may also be omitted, combined, renamed, regrouped, performed on one or more machines, or otherwise departed from the illustrated flow, provided that the performed process is operable and complies with at least one claim. Fig. 7A The steps of may be performed by computing device 110 (e.g., via audio processor 111, neural network model 118) or other suitable computing devices. In some examples, Fig. 7A The steps may be performed using a neural network model 128 .

[0048] Method 700 begins at step 702. At step 702, a first training set of audio portions is generated from a plurality of tracks. Segments within the plurality of tracks are labeled according to a plurality of music theory tags. For example, the plurality of tracks may be provided by source audio 130. In some examples, the audio portions of the first training set are generated using a sliding window spanning the tracks from source audio 130 and correspond to sub-portions 420. In some examples, the audio portions of the first training set have the same fixed duration (e.g., 20 seconds, 24 seconds, 30 seconds, etc.). Further, in some examples, at least some of the audio portions of the first set of audio portions are sub-portions of the same track in the plurality of tracks (e.g., the audio portions are from the same song).

[0049] In some examples, segments within the same track in the multiple tracks do not overlap with each other. In other words, one musical structure element does not overlap with another element.

[0050] At step 704, a deep neural network model is trained using the first training set as input, a first loss function for music theory label identification of the audio portion of the first training set, and a second loss function for segment boundary identification within the audio portion of the first training set. The music theory label identification and the segment boundary identification are generated by the deep neural network model. In some examples, the deep neural network model corresponds to neural network model 118, 128, 200, 300, 418. In some examples, the deep neural network model is a Spectrum Time Transformer (SpecTNT) neural network model.

[0051] In some examples, method 700 continues with steps 706 , 708 , and 710 .

[0052] At step 706 , a first audio track is received. In some examples, the first audio track may correspond to audio track 410 .

[0053] At step 708, segment boundary identifications of segments within the first audio track are generated using the deep neural network model. In various examples, the segment boundary identifications may be identified by the boundary processor 460 and / or the boundary processor 112.

[0054] At step 710, a music theory tag for a segment within the first audio track is generated using a deep neural network model. In some examples, the music theory tag corresponds to music theory tag identification 480.

[0055] In some examples, step 704 also includes generating a music theory label identification of the audio portion of the first training set, selected from a plurality of music theory labels.

[0056] Step 704 may also include: generating the music theory label identification of the first audio portion as a plurality of label probability curves corresponding to the plurality of music theory labels using the SpecTNT neural network model; generating the segment boundary identification of the first audio portion as a boundary probability curve using the SpecTNT neural network model; and merging the generated music theory label identification and segment boundary identification of the first audio portion. For example, the neural network model 418 may generate the segment likelihood 434 and the boundary likelihood 432 merged by the merging processors 442 and 444.

[0057] In some examples, step 704 may also include selecting a single generated music theory label identification for a segment identified by two adjacent segment boundary identifications based on an average probability of music theory label identifications generated during the segment. For example, a segment between two adjacent segment boundary identifications may start with an 80% chance of being a chorus, transition to a 60% chance of being a chorus, and then increase to a 95% chance of being a chorus while also having a constant 70% chance of being a bridge. In this example, the average of the chorus probabilities is 78%, so the chorus label may be selected to have a higher average probability than the bridge label. In some examples, the two adjacent segment boundary identifications may be adjusted using the ground truth boundaries of the segments.

[0058] In some examples, the first loss function is a first sum of first weighted binary cross entropies between generated music theory label identifiers and corresponding music theory labels of the segments, and the second loss function is a second sum of second weighted binary cross entropies between segment boundary identifiers and corresponding boundaries of the segments.

[0059] In some examples, the first loss function also includes a connectionist temporal localization loss configured to model the sequential order of the music theory labels.

[0060] Figure 7B Flowchart showing an example method 750 for identifying music theory tags for an audio track according to an example embodiment. Unless otherwise indicated, the technical processes shown in these figures will be performed automatically. In any given embodiment, some steps of the process may be repeated, possibly using different parameters or data. The steps in the embodiment may also be performed in accordance with Figure 7B The steps of method 750 may be performed in an order different from the order from top to bottom shown in the flowchart. The steps may be performed serially, in a partially overlapping manner, or completely in parallel. Therefore, the order of the steps of method 750 may change from one execution of the process to another execution of the process. Steps may also be omitted, combined, renamed, regrouped, performed on one or more machines, or otherwise departed from the illustrated flow, provided that the performed process is operable and complies with at least one claim. Figure 7B The steps of may be performed by computing device 110 (e.g., via audio processor 111, neural network model 118) or other suitable computing devices. In some examples, Figure 7B The steps may be performed using a neural network model 128 .

[0061] Method 750 begins at step 752. At step 752, an audio track is received. In some examples, the audio track may correspond to audio track 410.

[0062] At step 754, the audio track is divided into a first set of audio portions. In some examples, dividing the audio track includes generating the first audio portions using a sliding window across the audio track. In some examples, the first audio portions have the same fixed duration. In some examples, the first set of audio portions can correspond to audio portion 420.

[0063] At step 756, a music theory tag identification and a segment boundary identification are generated for the first set of audio portions using a Spectrum Time Transformer (SpecTNT) neural network model. In various examples, the SpecTNT model corresponds to neural network models 118, 128, 300, and / or 418. In some examples, the music theory tag identification and the segment boundary identification may correspond to music theory tag identification 438 and segment boundary identification 436.

[0064] At step 758, the generated music theory tag identifications for the first set of audio portions are merged, and the segment boundary identifications for the first set of audio portions are merged (eg, by the merge processor 442 and the merge processor 444).

[0065] At step 760, the merged segment boundary identifications are used to identify segments within the track. In various examples, segments may be identified by boundary processor 460 and / or boundary processor 112.

[0066] At step 762, a corresponding music theory tag is identified for the identified segment. The segment processor 114 may perform step 762. In some examples, identifying the music theory tag includes selecting the music theory tag from a plurality of music theory tags. In some examples, identifying the corresponding music theory tag includes selecting a single music theory tag for the segment based on an average probability of music theory tag identifications generated during the segment. In some examples, the plurality of music tags generally corresponds to the music theory tag identification 230.

[0067] Figure 8 , 9 10 and the associated description provide a discussion of various operating environments in which aspects of the present disclosure may be practiced. Figure 8 , 9 The devices and systems illustrated and discussed in connection with FIG. 10 are for purposes of example and illustration, and are not intended to limit the myriad of computing device configurations that may be used to practice various aspects of the present disclosure as described herein.

[0068] Figure 8A block diagram is provided to illustrate the physical components (e.g., hardware) of a computing device 800 with which aspects of the present disclosure may be practiced. The computing device components described below may have computer-executable instructions for implementing a music theory tag generation application 820 on a computing device (e.g., computing device 110), including computer-executable instructions for the music theory tag generation application 820 that may be executed to implement the methods disclosed herein. In a basic configuration, the computing device 800 may include at least one processing unit 802 and a system memory 804. Depending on the configuration and type of the computing device, the system memory 804 may include, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 804 may include an operating system 805 and one or more program modules 806 suitable for running the music theory tag generation application 820, such as programs related to the music theory tag generation application 820. Figure 1 , 2 , 3 and 4, in particular, a boundary processor 821 (e.g., corresponding to the boundary processor 112), a fragment processor 822 (e.g., corresponding to the fragment processor 114) and a post-processor 823 (e.g., corresponding to the post-processor 116).

[0069] For example, operating system 805 may be suitable for controlling the operation of computing device 800. In addition, embodiments of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application program, and are not limited to any particular application or system. Figure 8 800 is illustrated by those components within dashed line 808. Computing device 800 may have additional features or functionality. For example, computing device 800 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be Figure 8 809 and non-removable storage device 810.

[0070] As described above, a number of program modules and data files may be stored in system memory 804. When executed on processing unit 802, program module 806 (e.g., music theory tag generation application 820) may perform processes including, but not limited to, various aspects as described herein. Other program modules that may be used in accordance with various aspects of the present disclosure, and particularly for generating music theory tags, may include boundary processor 821, segment processor 822, and post-processor 823.

[0071] In addition, embodiments of the present disclosure may be practiced in circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or on a single chip containing electronic components or microprocessors. Figure 8 Each or many of the components described in the embodiment can be integrated into a system on a chip (SOC) on a single integrated circuit for practice. Such SOC devices may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operated via the SOC, the functions described herein regarding the ability of the client to switch protocols can be operated via dedicated logic integrated with other components of the computing device 800 on a single integrated circuit (chip). Embodiments of the present disclosure may also be practiced using other technologies capable of performing logical operations (such as AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present disclosure may be practiced within a general-purpose computer or within any other circuit or system.

[0072] The computing device 800 may also have one or more input devices 812, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. Output devices 814 may also be included, such as a display, a speaker, a printer, etc. The above devices are examples and other devices may be used. The computing device 800 may include one or more communication connections 816 that allow communication with other computing devices 850. Examples of suitable communication connections 816 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuit systems; universal serial bus (USB), parallel and / or serial ports.

[0073] The term computer-readable media as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, or program modules. System memory 804, removable storage device 809, and non-removable storage device 810 are all examples of computer storage media (e.g., memory storage devices). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other article that can be used to store information and can be accessed by computing device 800. Any such computer storage media may be part of computing device 800. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0074] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode the information in the signal. By way of example and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0075] Fig. 9 and Fig.10 A mobile computing device 900 is illustrated, with which embodiments of the present disclosure may be practiced, for example, a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc. In some aspects, the client may be a mobile computing device. Fig. 9 , illustrating one aspect of a mobile computing device 900 for implementing various aspects. In a basic configuration, the mobile computing device 900 is a handheld computer having input elements and output elements. The mobile computing device 900 typically includes a display 905 and one or more input buttons 910 that allow a user to input information to the mobile computing device 900. The display 905 of the mobile computing device 900 can also be used as an input device (e.g., a touch screen display). If included, the optional side input element 915 allows additional user input. The side input element 915 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 900 can incorporate more or fewer input elements. For example, in some embodiments, the display 905 may not be a touch screen. In yet another alternative embodiment, the mobile computing device 900 is a portable telephone system, such as a cellular phone. The mobile computing device 900 can also include an optional keypad 935. The optional keypad 935 can be a physical keypad or a "soft" keypad generated on the touch screen display. In various embodiments, output elements include a display 905 for displaying a graphical user interface (GUI), a visual indicator 920 (e.g., a light emitting diode), and / or an audio transducer 925 (e.g., a speaker). In some aspects, the mobile computing device 900 incorporates a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 900 incorporates input and / or output ports for sending or receiving signals to or from external devices, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port).

[0076] Fig.101 is a block diagram illustrating an architecture of one aspect of a mobile computing device. That is, the mobile computing device 900 can incorporate a system (e.g., architecture) 1002 to implement some aspects. In one embodiment, the system 1002 is implemented as a "smart phone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 1002 is integrated into a computing device, such as an integrated personal digital assistant (PDA) and a wireless phone.

[0077] One or more application programs 1066 may be loaded into the memory 1062 and run on or in association with the operating system 1064. Examples of application programs include a phone dialer program, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 1002 also includes a non-volatile storage area 1068 within the memory 1062. The non-volatile storage area 1068 may be used to store permanent information that should not be lost if the system 1002 loses power. The application programs 1066 may use and store information in the non-volatile storage area 1068, such as emails or other messages used by the email application, and the like. A synchronization application (not shown) also resides on the system 1002 and is programmed to interact with a corresponding synchronization application residing on the host computer to keep the information stored in the non-volatile storage area 1068 synchronized with the corresponding information stored at the host computer.

[0078] The system 1002 has a power source 1070, which can be implemented as one or more batteries. The power source 1070 can also include an external power source, such as an AC adapter or a powered docking station to replenish or recharge the batteries.

[0079] The system 1002 may also include a radio interface layer 1072 that performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 1072 facilitates wireless connections between the system 1002 and the "outside world" via a communications carrier or service provider. Transmissions to and from the radio interface layer 1072 are performed under the control of the operating system 1064. In other words, communications received by the radio interface layer 1072 may be propagated to the application 1066 via the operating system 1064, and vice versa.

[0080] Visual indicator 1020 can be used to provide visual notifications, and / or audio interface 1074 can be used to communicate via audio transducer 925 (eg, Fig. 9The visual indicator 1020 is a light emitting diode (LED) and the audio transducer 925 is a speaker. These devices can be directly coupled to the power supply 1070 so that when activated, they remain on for a duration indicated by the notification mechanism even if the processor 1060 and other components can be turned off to save battery power. The LED can be programmed to remain on indefinitely until the user takes action to indicate the power-on state of the device. The audio interface 1074 is used to provide an auditory signal to the user and receive an auditory signal from the user. For example, in addition to being coupled to the audio transducer 925, the audio interface 1074 can also be coupled to a microphone to receive an auditory input, such as to facilitate a telephone conversation. According to an embodiment of the present disclosure, the microphone can also be used as an audio sensor to facilitate control notifications, as described below. The system 1002 can also include a video interface 1076 that enables the operation of the peripheral device 1030 (e.g., a vehicle-mounted camera) to record still images, video streams, etc.

[0081] The mobile computing device 900 implementing the system 1002 may have additional features or functionality. For example, the mobile computing device 900 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data in a manner similar to that of a computer program product. Fig.10 , illustrated by non-volatile storage area 1068.

[0082] Data / information generated or captured by the mobile computing device 900 and stored via the system 1002 may be stored locally on the mobile computing device 900, as described above, or the data may be stored on any number of storage media that can be accessed by the device via the radio interface layer 1072 or via a wired connection between the mobile computing device 900 and an independent computing device associated with the mobile computing device 900, for example, a server computer in a distributed computing network such as the Internet. As should be appreciated, such data / information may be accessed via the mobile computing device 900, via the radio interface layer 1072, or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use according to well-known data / information transmission and storage means, including email and collaborative data / information sharing systems.

[0083] As should be understood, Fig. 9 and 10 The methods and systems described herein are for purposes of illustrating the present invention and are not intended to limit the present disclosure to a specific sequence of steps or specific combination of hardware or software components.

[0084] The description and explanation of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure claimed in any way. The various aspects, examples and details provided in this application are considered to be sufficient to convey the best mode of possession and enable others to make and use the disclosure claimed. The disclosure claimed should not be interpreted as being limited to any aspect, example or detail provided in this application. Whether combined or shown and described individually, it is intended to selectively include or omit various features (of structure and method) to produce an embodiment with a specific feature set. The description and explanation of this application have been provided, and those skilled in the art can envision changes, modifications and alternative aspects that fall within the spirit of the broader aspects of the overall inventive concept embodied in this application, which do not depart from the broader scope of the disclosure claimed.

Claims

1. A computer-implemented method for training a neural network model for identifying music theory tags for an audio track, the method include: generating a first training set of audio portions from a plurality of audio tracks, wherein segments within the plurality of audio tracks are labeled according to a plurality of music theory labels; training a deep neural network model using the first training set as input, a first loss function for music theory label identifications of audio portions of the first training set, and a second loss function for segment boundary identifications within the audio portions of the first training set, wherein the music theory label identifications and the segment boundary identifications are generated by the deep neural network model; receiving a first audio track; generating segment boundary identifiers for segments within the first audio track using the deep neural network model; and Music theory tags are generated for the segments within the first audio track using the deep neural network model.

2. The method according to claim 1, wherein the deep neural network model is trained include: A music theory label identification selected from the plurality of music theory labels is generated for the audio portion of the first training set. The method of claim 1 , wherein the audio portions of the first training set have the same fixed duration.

4. The method of claim 3, wherein at least some of the audio portions of the first set of audio portions are sub-portions of the same audio track of the plurality of audio tracks. The method according to claim 1 , wherein the segments within the same audio track of the plurality of audio tracks do not overlap with each other.

6. The method of claim 5, wherein generating the first training set comprises generating a first audio portion using a sliding window across the same audio track.

7. The method of claim 6, wherein the deep neural network is a Spectrum-Time Transformer-Transformer (SpecTNT) neural network, and the deep neural network model is trained include: generating the music theory label identification for the first audio portion as a plurality of label probability curves corresponding to the plurality of music theory labels using the SpecTNT neural network model; generating the segment boundary identification for the first audio portion as a boundary probability curve using the SpecTNT neural network model; as well as The music theory tag identification and the segment boundary identification used for the generation of the first audio portion are merged.

8. The method according to claim 7, wherein the method further include: A single generated music theory label identification for a segment identified by two adjacent segment boundary identifications is selected based on an average probability of the generated music theory label identifications during the segment.

9. The method of claim 8, wherein the single generated music theory label identification is selected include: The two adjacent segment boundary identifiers are adjusted using the ground truth boundaries of the segments.

10. The method of claim 1, wherein the first loss function is a first sum of a first weighted binary cross entropy between the generated music theory label identification and the corresponding music theory label of the segment; The second loss function is a second sum of second weighted binary cross entropies between the segment boundary identifiers and corresponding boundaries of the segments.

11. The method of claim 1 , wherein the first loss function further comprises a connectionist temporal localization loss configured to model the ordinal order of music theory labels.

12. A method for identifying music theory tags for an audio track, the method include: Receive audio track; dividing the audio track into a first set of audio parts; generating music theory label identifiers and segment boundary identifiers for the first set of audio portions using a deep neural network model; merging the generated music theory tag identifiers for the first set of audio portions and merging the segment boundary identifiers for the first set of audio portions; Using the merged segment boundary identifiers to identify segments within the audio track; Identifies the corresponding music theory tag for the segment identified.

13. The method of claim 12, wherein the music theory tag is identified include: The music theory tag is selected from a plurality of music theory tags.

14. The method of claim 12, wherein the audio track is divided into the first audio part set include: A first audio portion is generated using a sliding window across the audio track.

15. The method of claim 14, wherein the first audio portions have the same fixed duration.

16. The method according to claim 12, wherein the identification is used to identify the corresponding music theory label of the segment include: A single music theory label for a segment is selected based on an average probability of the music theory label identifications generated during the segment.

17. A non-transitory computer-readable storage medium comprising instructions executable by one or more processors, the instructions, when executed by the one or more processors, causing the one or more processors to: generating a first training set of audio portions from a plurality of audio tracks, wherein segments within the plurality of audio tracks are labeled according to a plurality of music theory labels; and training a deep neural network model using the first training set as input, a first loss function for music theory label identifications of audio portions of the first training set, and a second loss function for segment boundary identifications within the audio portions of the first training set, wherein the music theory label identifications and the segment boundary identifications are generated by the deep neural network model; receiving a first audio track; generating segment boundary identifiers for segments within the first audio track using the deep neural network model; and Music theory tags are generated for the segments within the first audio track using the deep neural network model.

18. The computer-readable storage medium of claim 17, wherein the instructions are executable by the one or more processors to cause the one or more processors to: A music theory label identification selected from the plurality of music theory labels is generated for the audio portion of the first training set.

19. The computer-readable storage medium of claim 17, wherein the segments within a first track of the plurality of tracks do not overlap with each other; Wherein the instructions are executable by the one or more processors to cause the one or more processors to generate a first audio portion using a sliding window across the first audio track.

20. The computer-readable storage medium of claim 17, wherein the instructions are executable by the one or more processors to cause the one or more processors to: generating, using the deep neural network model, the music theory label identification for the first audio portion as a plurality of label probability curves corresponding to the plurality of music theory labels; generating the segment boundary identification for the first audio portion as a boundary probability curve using the deep neural network model; and The music theory tag identification and the segment boundary identification used for the generation of the first audio portion are merged.